Organizing Terraform Projects With Terragrunt

Infrastructure-as-Code (IaC) tools, like Terraform, have become essential for managing infrastructure in a scalable, repeatable, and automated way. Terraform helps engineers define, provision, and manage cloud resources across multiple providers. However, as infrastructure complexity grows, managing Terraform configurations becomes challenging. This is where Terragrunt—a thin wrapper for Terraform—comes into play. It enhances Terraform’s capabilities, making it easier to manage large-scale infrastructures efficiently.

In this blog, we will explore how to organize Terraform projects using Terragrunt, and how it solves common challenges with Terraform’s limitations such as managing backend configurations, large state files, and dependencies between multiple stacks.

Why Terragrunt?

While Terraform is a powerful tool, it has limitations when it comes to managing large-scale, complex infrastructures. These challenges often manifest when trying to manage multiple environments (like dev, staging, and production), ensure consistent backend configurations, and handle dependencies between different Terraform modules or stacks.

Problems with Terraform in Large Projects

1. Hardcoded Backend Configuration: In Terraform, the backend configuration for storing the state (such as an S3 bucket for AWS) is typically hardcoded. This becomes problematic when managing multiple environments in a CI/CD process because each environment may require different backend configurations. This leads to code duplication or complex workarounds that can break maintainability.
2. Big State Files: As infrastructure grows, so does the Terraform state file. When all resources are stored in a single state file, even small updates trigger the entire file to load, which can significantly increase the time required for each apply. Furthermore, a large state file becomes a single point of failure: if the file becomes corrupt or inaccessible, the entire infrastructure management process halts.
3. Handling Dependencies Between Stacks: Terraform doesn’t provide an out-of-the-box solution to handle dependencies between separate infrastructure stacks. For instance, if a Virtual Private Cloud (VPC) module needs to be provisioned before an ECS cluster, you need to manually manage the order, which can lead to errors and misconfigurations.

Terragrunt addresses all of these issues by adding features such as automatic backend configuration management, splitting state files by stack, and defining clear dependencies between stacks.

Parametrizing Terraform State with Terragrunt

One of the core advantages of using Terragrunt is the ability to parametrize and dynamically configure the Terraform backend. With Terragrunt, you no longer need to hardcode the backend configuration for each environment. Instead, you can pass parameters via Terragrunt configuration files, simplifying the management of multiple environments (e.g., dev, staging, prod) while ensuring each environment has its own isolated state.

Example: Dynamic Backend Configuration with Terragrunt

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# terragrunt.hcl

remote_state {
  backend = "s3"
  config = {
    bucket         = "my-terraform-state"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-lock-table"
  }
}

inputs = {
  environment = "prod"
}

In this configuration:

• The backend for state storage is dynamically configured based on the folder structure and environment.
• bucket, key, and other configurations can be shared across environments or customized as needed.

By leveraging dynamic parameters in Terragrunt, you can automatically set the appropriate backend configuration for different environments without duplicating code or manually updating each environment’s backend.

Handling Large State Files with Terragrunt

Instead of storing all resources in a single state file, Terragrunt encourages breaking your infrastructure into smaller, manageable stacks. Each stack will have its own state file, thus reducing the size and complexity of each state.

By splitting infrastructure into multiple stacks (e.g., VPC, ECS, RDS), you:

• Improve Performance: Smaller state files load and update faster.
• Increase Resilience: If one state file becomes corrupted or is locked, it only affects that specific stack, not the entire infrastructure.

Example: Parametrizing Terraform State Files

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# terragrunt.hcl for the VPC stack

remote_state {
  backend = "s3"
  config = {
    bucket         = "my-terraform-state"
    key            = "vpc/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-lock-table"
  }
}

inputs = {
  environment = "prod"
}

In this setup, each stack (such as VPC or ECS) has its own isolated state file (vpc/terraform.tfstate), ensuring quicker and safer operations.

Creating Dependencies Between Stacks with Terragrunt

Terragrunt introduces a powerful feature to manage dependencies between different infrastructure stacks. By defining dependencies, you can enforce the correct order of resource provisioning, ensuring that shared components like a VPC are created before any dependent resources (e.g., ECS or RDS) are applied.

Example: Stack Dependencies in Terragrunt

1
2
3
4
5
6
7
8
9

# terragrunt.hcl for the ECS stack

dependency "vpc" {
  config_path = "../vpc"
}

inputs = {
  vpc_id = dependency.vpc.outputs.vpc_id
}

In this configuration:

• We declare a dependency on the VPC stack, ensuring the VPC is created first.
• We can then reference outputs from the VPC stack, such as vpc_id, and pass them as inputs to the ECS stack.

Terragrunt automatically handles the order in which stacks are applied, reducing the chances of errors due to missing dependencies.

Running Multiple Stacks with One Command

Terragrunt allows you to apply all your infrastructure stacks with a single command. For example, to apply both the VPC and ECS stacks:

1

terragrunt run-all apply

This command will apply all the stacks in the correct order, based on the dependencies defined in the Terragrunt configuration files.

Directory Structure

A well-structured project is crucial for scaling infrastructure management. Terragrunt encourages the use of a modular approach, separating the code that defines resources (modules) from the configuration that applies the modules (live environments).

Example Project Structure

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

infrastructure/
├── live/
│   ├── dev/
│   │   ├── vpc/
│   │   │   └── terragrunt.hcl
│   │   ├── ecs/
│   │   │   └── terragrunt.hcl
│   ├── prod/
│   │   ├── vpc/
│   │   │   └── terragrunt.hcl
│   │   ├── ecs/
│   │   │   └── terragrunt.hcl
└── modules/
    ├── vpc/
    │   └── main.tf
    └── ecs/
        └── main.tf

Explanation:

• Modules Folder: Contains reusable Terraform modules for resources like VPC, ECS, RDS, etc.
• Live Folder: Contains environment-specific configurations for dev, staging, and prod. Each environment has its own Terragrunt configuration (terragrunt.hcl), which references the modules and applies dynamic backend and input configurations.

By separating your code into modules (the infrastructure logic) and live (the environment configurations), you create a clear and scalable structure. This makes it easier to manage multiple environments and isolate infrastructure stacks.

Conclusion

Terragrunt simplifies and enhances Terraform project organization, enabling dynamic configuration, efficient state management, and seamless stack dependencies. By using Terragrunt, you can:

• Dynamically configure backends, avoiding hardcoded configurations.
• Split large state files into smaller, more manageable ones.
• Define and manage dependencies between stacks.
• Organize projects in a modular, scalable way, ensuring maintainability as your infrastructure grows.

By adopting Terragrunt, you can overcome Terraform’s limitations and bring more structure, automation, and efficiency to your infrastructure management processes.

Happy engineering!