[Chen Yufeiyang (feiiiiii5)]

A rollout whose config.k8s.job_template cannot render or parse never reaches a terminal state. _create_job() calls build_job_spec() inside the Kubernetes API try/except, so a deterministic template/manifest error (TemplateSyntaxError, a YAML parse error, metadata: null) is classified as transient unless its message happens to contain 422, unprocessable, or invalid. The rollout stays QUEUING, the controller logs “will retry” on every poll, and AglRolloutManager.enqueue_and_wait_until_completed() waits forever.

Manifest construction happens before any Kubernetes request and is deterministic, so retrying the same rollout cannot succeed. This separates the build step from the API call: a build exception marks the rollout FAILED with the validation error, while K8s API failures keep the existing retry classification.

Test: the new regression covers a Jinja syntax error and a structurally invalid Job (metadata: null); both fail on 05c7fe1 with Job creation failed — will retry and no status patch, and pass here. The controller/server/client/package suites are 75 passed; ruff check/format --check are clean.

Details

The regression mocks _patch_status but exercises the real build_job_spec() failures. The trainer-side hang is not executed end-to-end because it requires the optional verl/torch group; it follows from the manager polling until every rollout is terminal. No live Kubernetes cluster is used.

[Copilot]

🟢 Approval recommended

The focused change addresses the rollout hang, preserves API retry behavior, and includes regression coverage with no identified blocking issues.

No open issues 🎉

What changed in this PR

This PR prevents rollouts with invalid Kubernetes Job templates from remaining in QUEUING indefinitely by treating manifest-build errors as terminal failures.

Changes:

• Separates manifest construction from Kubernetes API error handling while preserving existing API retry behavior.
• Centralizes invalid Job failure reporting.
• Adds regression tests for Jinja syntax errors and null Job metadata.

File	Description
tests/​controller/​test_k8s_reconciler_errors.py	Verifies invalid templates mark rollouts as failed.
agentlightning/​controller/​k8s_reconciler.py	Handles manifest-build failures separately and shares failure reporting.

_{Review effort: Balanced}