Terraform Remote State & CI/CD Runbook

This runbook captures the setup I use to make Terraform safe for a team. It pairs with the terraform-aws-reference-architecture and cicd-pipeline-templates repos.

Why remote state with locking

Local state is a loaded gun: it lives on one laptop, isn't shared, and two people applying at once corrupts it. Remote state on S3 with a DynamoDB lock table fixes all three.

1. Bootstrap the backend (once)

Create the state bucket and lock table out-of-band (they can't be in the same state they back). A tiny bootstrap config:

resource "aws_s3_bucket" "state" {
  bucket = "myorg-tf-state"
}

resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    apply_server_side_encryption_by_default { sse_algorithm = "aws:kms" }
  }
}

resource "aws_dynamodb_table" "locks" {
  name         = "terraform-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
}

2. Point each environment at it

Use a distinct state key per environment so they never collide:

terraform {
  backend "s3" {
    bucket         = "myorg-tf-state"
    key            = "environments/prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

The reference-architecture repo deliberately gives every example a unique state key for exactly this reason.

3. The CI/CD flow

The golden rule: plan on pull requests, apply on merge to main. Humans review the plan; the pipeline applies what was reviewed.

name: Terraform
on:
  pull_request:
  push:
    branches: [main]

jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
      - run: terraform fmt -check
      - run: terraform validate
      - run: terraform plan -no-color

  apply:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: production   # require manual approval
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
      - run: terraform apply -auto-approve

4. Guardrails that prevent disasters

• Branch protection: no direct pushes to main; PRs require the plan job to pass.
• Environment approval: the apply job uses a GitHub Environment with required reviewers.
• tfsec / checkov in the pipeline to catch insecure config before apply.
• No secrets in state-readable form: mark sensitive vars sensitive = true, supply via TF_VAR_* or a secrets manager, never a committed .tfvars.
• Plan artifacts: save the plan, apply that exact plan (terraform plan -out → terraform apply tfplan) so what's applied is what was reviewed.

5. Common failure modes

Symptom	Cause	Fix
Error acquiring the state lock	A previous run crashed	terraform force-unlock <ID> (carefully)
Drift between runs	Manual console changes	Reconcile, then enforce no-manual-changes
Two envs clobber each other	Shared state key	Unique key per environment
Secrets in plan output	Unmarked sensitive vars	Add sensitive = true

Set this up once and Terraform stops being scary. The pipeline becomes the only way infra changes, every change is reviewed, and state is safe.