Terraform State Drift: Detection, Remediation, and the Pull Request Policy That Helps

A routine terraform apply on a Tuesday morning replaced a security group rule that someone had added manually in the AWS console during a Friday incident. The apply succeeded. Traffic from our CI runners to the internal artifact registry dropped. Partial outage, 47 minutes, no customer-facing SLA breach but three teams blocked.

Root cause: state drift. Terraform state said one thing; AWS said another. Apply reconciled toward state and deleted the "extra" rule.

How drift happens in our org

We run Terraform 1.7.x with S3 backend (SSE-KMS, DynamoDB lock table terraform-locks). Roughly 180 modules across three AWS accounts. Drift sources we've cataloged:

1. Console changes during incidents — "just open 443 from this /32 until we fix it"
2. Other IaC — CDK stack touching shared VPC resources
3. AWS service-linked changes — auto-created ENI rules, default SG modifications
4. Manual aws cli from a bastion — rarer but memorable

Before the incident, we ran terraform plan in CI on merge but only on changed modules. Untouched modules could drift silently for months.

Detection tooling we deployed

Scheduled drift detection (not just plan-on-PR)

GitHub Actions workflow, cron every 6 hours:

terraform plan -detailed-exitcode -out=/dev/null
# exit 0 = no changes, 1 = error, 2 = changes present

Exit code 2 opens a Jira ticket tagged drift-detected with plan output attached. Not paging — drift is often intentional-but-unrecorded — but no drift ticket may be older than 7 days without a waiver from infra lead.

We use Spacelift for two production accounts (trial became paid) primarily for drift detection UI and policy-as-code. Native Terraform Cloud would work; we already had Spacelift eval credits.

Resource-level allowlist for known ephemeral drift

Some resources legitimately drift (autoscaling desired capacity, ECS task counts). We use lifecycle { ignore_changes = [...] } sparingly — 23 resources across all modules, each with a comment linking to a ticket. Anything without a ticket gets removed in quarterly audit.

AWS Config as secondary signal

AWS Config rule VPC_SG_OPEN_ONLY_TO_AUTHORIZED_PORTS catches SG drift that Terraform hasn't run against yet. Config → EventBridge → Slack #infra-alerts. Noisy for the first two weeks; tuned authorized port list.

The PR policy that actually helps

Written policy, enforced by CODEOWNERS + CI:

1. No console changes without a Terraform PR within 48 hours — on-call waiver for break-glass, must file ticket before shift ends.
2. Every PR that touches .tf must include terraform plan output for affected workspaces (CI artifact, not pasted in description — too large).
3. Drift remediation PRs get fast-track review — 4-hour SLA from #infra-oncall.
4. Import before apply — if someone created a resource manually, terraform import in the same PR that codifies it. Never let apply "adopt" by accident.

The cultural piece: we stopped shaming console changes. Incidents require speed. Shame drives hiding. The 48-hour codification rule is non-negotiable but break-glass is expected.

Remediation playbook

Drift type	Action
Manual SG rule, still needed	Import + codify in module
Manual SG rule, obsolete	Let apply remove (verify in staging first)
CDK overlap	Single owner per resource tag ManagedBy
State stale, real infra deleted	terraform refresh then plan; may need targeted destroy

After the incident, we ran full plan across all workspaces. 38 had exit code 2. Took two sprints to clear. Several "drifts" were state bugs from module refactoring — not AWS console at all.

Module refactor drift (the silent majority)

Twelve of the 38 drift tickets were not AWS console edits — they were Terraform module output renames without state moves. Example: splitting aws_security_group_rule.main into counted resources changed addresses in state; plan wanted destroy/create on rules that never left AWS.

Fix pattern:

terraform state mv 'module.vpc.aws_security_group_rule.main' \
  'module.vpc.aws_security_group_rule.main[0]'

We added terraform plan gate on module PRs even when workspace isn't the consumer — module maintainers run against one reference workspace.

Break-glass logging

CloudTrail alone wasn't enough — too noisy. We enabled AWS Config recording for AWS::EC2::SecurityGroup and EventBridge rule → Lambda that posts to Slack with IAM user/role ARN. Doesn't prevent drift; creates social pressure when console edits appear without matching Terraform PR within 48h.

IAM policy change: removed AdministratorAccess from on-call roles; replaced with scoped policies + terraform-apply role assumable with MFA for production workspaces. Console edits dropped 70% quarter-over-quarter — not from virtue, from inability.

What I'd do next

• OPA/conftest policy: no 0.0.0.0/0 ingress on port 22/3389 in plan output
• Automated import suggestions — Spacelift's drift view helps; still manual for complex modules
• Tie drift metrics to postmortem culture: repeat drift on same resource type → design review

Cost angle: drift-driven outages are cheaper than cloud egress mistakes but equally preventable with discipline.

Terraform is declarative only if humans stop treating the console as a second UI. Detect drift continuously, not only when someone touches a module.