Terraform Ep 10: Remote State Backends and Concurrency Locking

1. The Dangers of Local State

#

In Episode 1, we learned that Terraform uses the .tfstate JSON file to map your HCL code to physical cloud resources. By default, this file is stored locally in the directory where you run terraform apply.

If you work alone, this is perfectly fine. But consider a team of two engineers: Alice and Bob.

1. Alice creates an EC2 instance. The local state file on her laptop records i-12345.
2. Bob wants to add an S3 bucket to the same infrastructure. He pulls the code from GitHub, but GitHub does not contain the state file (because .tfstate is always added to .gitignore to prevent leaking secrets).
3. Bob runs terraform apply. Because Bob’s laptop has no state file, Terraform assumes the EC2 instance doesn’t exist. Terraform attempts to create a brand new EC2 instance and the S3 bucket.
4. Alice and Bob now have entirely disjointed, corrupted infrastructure mappings.

The Concurrency Problem

#

Even worse, what if Alice and Bob share the state file via a network drive, and both run terraform apply at the exact same millisecond? The state file will become corrupted, and Terraform will lose track of your entire production environment.

2. The Solution: AWS S3 & DynamoDB

#

The industry-standard solution for AWS environments is to store the state file remotely in an Amazon S3 Bucket, and use an Amazon DynamoDB Table as a concurrency lock.

• S3 Bucket: Acts as the centralized, highly-available storage for the .tfstate file. When Alice runs terraform apply, Terraform downloads the state from S3 into memory, updates it, and uploads the new version back to S3.
• DynamoDB Table: Acts as a mutex (mutual exclusion) lock. When Alice starts an apply, Terraform writes an item into DynamoDB saying “Alice is holding the lock”. If Bob tries to apply at the same time, Terraform checks DynamoDB, sees Alice’s lock, and throws an error, forcing Bob to wait.

3. Practice: Bootstrapping the Backend Infrastructure

#

Before Terraform can store its state in S3, the S3 bucket and DynamoDB table must actually exist. You face a “Chicken and Egg” problem here: Do you create the S3 bucket manually (ClickOps), or do you use Terraform to create the Terraform backend?

The best practice is to create a dedicated, isolated Terraform directory specifically for bootstrapping the backend.

Step 3.1: Creating the Bootstrap Code

#

Create a folder named terraform-backend-bootstrap:

mkdir terraform-backend-bootstrap
cd terraform-backend-bootstrap

Create main.tf to provision the S3 bucket and DynamoDB table:

# main.tf

provider "aws" {
  region = "us-east-1"
}

# 1. The S3 Bucket for State Storage
resource "aws_s3_bucket" "terraform_state" {
  bucket = "my-company-terraform-state-backend-unique-123" # Must be globally unique!
  
  # Prevent accidental deletion of this bucket
  lifecycle {
    prevent_destroy = true
  }
}

# Enable Versioning (Crucial for rollback if state is corrupted)
resource "aws_s3_bucket_versioning" "enabled" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

# Enable Server-Side Encryption
resource "aws_s3_bucket_server_side_encryption_configuration" "default" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

# 2. The DynamoDB Table for Locking
resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-state-locking"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID" # MUST be exactly "LockID"

  attribute {
    name = "LockID"
    type = "S" # String
  }
}

Run terraform init and terraform apply. You now have a local state file that tracks the remote backend infrastructure.

4. Practice: Migrating to the Remote Backend

#

Now that the backend infrastructure exists, we can instruct our actual application’s Terraform code to use it.

Create a new directory for your application terraform-app:

cd ..
mkdir terraform-app
cd terraform-app

Create a main.tf file. This is where we configure the backend block inside the terraform block.

# terraform-app/main.tf

terraform {
  # Instruct Terraform to use S3 for state management
  backend "s3" {
    # Replace with the exact bucket name you created in Step 3
    bucket         = "my-company-terraform-state-backend-unique-123" 
    
    # The path where the state file will be saved inside the bucket
    key            = "global/s3/terraform.tfstate" 
    
    region         = "us-east-1"
    
    # The exact DynamoDB table name you created in Step 3
    dynamodb_table = "terraform-state-locking"
    
    encrypt        = true
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# A simple resource to test the state file creation
resource "aws_iam_user" "test_user" {
  name = "backend-test-user"
}

Initializing the Remote Backend

#

Whenever you add or change a backend block, you must run terraform init.

terraform init

Expected Terminal Output:

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Now, run terraform apply. Terraform will create the IAM user.

If you look in your local terraform-app directory, you will not find a terraform.tfstate file! It has been safely uploaded to your S3 bucket. Furthermore, during the apply process, Terraform successfully locked the DynamoDB table to ensure no one else could interfere.

Troubleshooting & Common Errors

#

1. Error acquiring the state lock: ConditionalCheckFailedException

   • Root Cause: Terraform attempted to acquire a lock, but DynamoDB reported that a lock already exists. Either a teammate is currently running terraform apply, or a previous Terraform run crashed and left a “stale lock” behind.
   • Solution: Ask your team if anyone is running Terraform. If not, you can manually break the stale lock using the command terraform force-unlock <LOCK_ID>.

2. Error: AccessDenied: Access Denied to S3 bucket

   • Root Cause: Your AWS CLI credentials do not have the required IAM permissions to s3:GetObject and s3:PutObject on the specified bucket.
   • Solution: Ensure your IAM user has the appropriate policies attached to read and write to the state bucket.

3. Error: Backend configuration changed

   • Root Cause: You modified the backend "s3" block (e.g., changed the bucket name or the key path) without running terraform init.
   • Solution: Run terraform init -reconfigure or terraform init -migrate-state to instruct Terraform on how to handle the backend transition.

Conclusion & Next Steps

#

You have successfully elevated your infrastructure to enterprise-grade maturity. Your state is now centrally stored, encrypted at rest, version-controlled, and protected against race conditions.

You have now completed Tier 2: Intermediate Constructs.

In Tier 3: Advanced Platform Engineering, we will explore how to manage completely different environments (like Dev, Staging, and Prod) without duplicating code, starting with Episode 11: Managing Multi-Environment Architecture (Workspaces vs Directories).