Skip to content

fix(cd): Audit & harden deployment pipeline with full health checks, keep-alive, and CRLF hygiene - #7

Merged
ayushcmd merged 2 commits into
mainfrom
fix/cd-audit-remediation
Oct 27, 2026
Merged

ayushcmd merged 2 commits into
mainfrom
fix/cd-audit-remediation

Conversation

@ayushcmd

Copy link
Copy Markdown
Owner

Full CI/CD Audit & Safe Remediation Summary

This Pull Request addresses the audit findings from the GitHub Actions CD deployment pipeline failure following PR #6 merge:

1. Root-Cause Analysis

  • CD Job Failure: Run \40462380305\ halted at \Validate Deployment Secrets\ because required deployment secrets have not yet been configured in GitHub repository/environment secrets.
  • Fail-Safe Operation: The failure was the intended and safe outcome of the secret-validation guard, preventing unauthenticated connection attempts and missing credential crashes.

2. Comprehensive Latent Issues Addressed

  1. GitHub Environment Support:
    • Configured \environment: production\ on the deploy job so GitHub Actions can pull secrets from either Repository secrets or Environment (\production) secrets.
  2. Private Key & Known Hosts Formatting:
    • Converted key/host writing to \printf '%s\n' ... | tr -d '\r'\ to guarantee no Windows CRLF line-ending corruption causes OpenSSH format rejections.
  3. SSH Session Keep-Alive:
    • Added \ServerAliveInterval 15\ and \ServerAliveCountMax 4\ to ~/.ssh/config\ to prevent NAT/firewall idle disconnects during multi-minute container builds.
  4. Git Safe Directory & Remote Update Strategy:
    • Added \git config --global --add safe.directory\ and \git reset --hard origin/main\ on the host to ensure server-side updates never fail due to local modifications, while preserving untracked files like .env.
  5. Comprehensive 5-Point Health Checks:
    • Added bounded probe for the Frontend public SPA endpoint (HTTP 200).
    • Added bounded probe for Backend /health\ verifying HTTP 200 and \status: healthy.
    • Added bounded probe for Prometheus /prom/api/v1/targets, validating JSON and ensuring all 4 required scrape targets (\�lertmanager, \cloudmonitor-backend,
      ode-exporter, \prometheus) are active and UP.
    • Added container state verification on the host via \docker compose ps.
  6. Documentation & Preparation Checklist:
    • Updated \docs/CICD_SETUP_GUIDE.md\ with repository vs environment secret instructions, host key verification steps via EC2 system log, and a step-by-step preparation checklist.

3. Local Verification Results

  • Workflow YAML syntax validated (\yaml.safe_load).
  • Backend test suite passed (30/30 passed).
  • Frontend production bundle built cleanly (
    pm run build\ - 1571 modules).
  • Docker compose configurations validated (\docker compose config).
  • Git diff checked for whitespace and secret hygiene.

@ayushcmd
ayushcmd merged commit aaed24c into main Oct 27, 2026
3 checks passed
@ayushcmd ayushcmd self-assigned this Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant