DriftGuard

Under 2min self-heal

38 tests / 10 properties

7 Terraform modules

12 AWS services

Overview

Self-healing GitOps infrastructure platform on AWS EKS. Terraform provisions the AWS substrate (VPC, EKS, IAM/IRSA, ECR, DNS, KMS), installs ArgoCD once, then steps back. ArgoCD owns all in-cluster state and continuously reconciles from Git. If someone mutates a resource out of band, the platform detects drift within seconds and self-heals within two minutes. The demo workload deploys through Argo Rollouts with a canary strategy: five weight steps (20% through 100%), each with Prometheus-backed analysis. Error rate above 5% or p95 latency above 500ms aborts the rollout automatically. OPA/Gatekeeper admission policies reject privileged containers. Falco monitors runtime behavior. The full LGTM observability stack (Prometheus, Grafana, Loki, Tempo, OpenTelemetry) provides metrics, logs, and traces. CI uses GitHub OIDC federation for short-lived credentials with no static AWS keys anywhere.

Architecture Diagram

Two-Layer Control Model

AWS Infrastructure

GitOps Control Plane

CI/CD Delivery

Runtime Delivery and Observability

Design Decisions

• →Two-layer control model: Terraform manages resources whose lifecycle is external to Kubernetes (VPC, EKS, IAM, ECR, DNS). ArgoCD manages resources that benefit from continuous drift correction (add-ons, policies, workloads). The handoff point is the ArgoCD Helm install.
• →Git as the single source of truth. CI never touches the cluster directly. It builds, scans, pushes to ECR, and commits an image tag to the Config Repo. ArgoCD reconciles. A compromised CI gives ECR push (scoped) and a Git commit (auditable), not kubectl exec on running pods.
• →IRSA over static credentials. Every pod that needs AWS access gets its own OIDC-federated role scoped to enumerated actions and resource ARNs. No role combines wildcard actions with wildcard resources (enforced by OPA policy over Terraform plan JSON).
• →Fail-closed admission. Gatekeeper's validating webhook uses failurePolicy: Fail. If the webhook is unavailable, admission is denied. Tested by scaling the controller to zero and confirming a privileged pod is still rejected.
• →Property-based testing (Hypothesis) for authored logic: tag completeness, IAM wildcard detection, IRSA trust scoping, EKS CIDR allowlist, node-count cap validation, and the no-plaintext-secrets scanner. Each property runs 100+ iterations.
• →Single-command teardown. Drains ALB-backed Services first (to release the load balancer), then runs terraform destroy. Verified zero EKS clusters, zero instances, zero NAT gateways, zero load balancers remaining after completion.

Deployment

Infrastructure provisioned through Terraform with 7 modules (networking, eks, iam, ecr, dns, addons-bootstrap, github-oidc) composed per environment (dev/staging/prod). Each environment uses isolated state backends (S3 + DynamoDB locking). ArgoCD bootstraps from a Root Application that fans out through ApplicationSets to every add-on and workload. CI/CD via GitHub Actions with OIDC federation: Terraform plan on PR (read-only role), apply on merge (scoped write role), scheduled drift-check (read-only role), image build/scan/publish with Trivy blocking HIGH/CRITICAL before push. Scripts for both PowerShell and bash handle validation, integration testing, e2e smoke tests (full provision, deploy, drift, self-heal, canary-abort, teardown), and single-command environment destruction.

Lessons Learned

38 offline tests passed. Then terraform plan against a real account broke 7 things. Terraform's for_each requires string set keys, not numbers. Kustomize's load restrictor rejects cross-directory references. Argo Rollouts refuses an AnalysisTemplate with no bounded metric count. The OIDC trust for PR workflows uses a different subject claim format than branch pushes. A canary Rollout referencing a Service that doesn't exist fails silently until the controller evaluates it. The lesson: a test suite that never runs terraform plan or kustomize build against a real backend is giving you false confidence. Offline validation catches maybe 80% of issues. The remaining 20% only surface when the real tool hits real infrastructure with real controllers evaluating real CRDs. Also: NAT gateway route selection logic must match the creation predicate. Creating per-AZ gateways but routing all traffic through gateway[0] leaves orphaned, billed resources that no monitoring catches until you read the bill.