Back to Home
March 14, 20266 min read

From ClickOps to Terraform: what we measured

How we cut MTTR and why state files deserve respect.

Problem

Manual cloud changes (ClickOps) caused drift, weak audit trails, and hard-to-reproduce incident fixes.

Solution

We moved core AWS resources to Terraform modules, standardized environments, and enforced PR-based infra changes.

Impact

MTTR dropped, rollbacks became predictable, and infra changes became reviewable and repeatable.

The allure of the AWS console is strong. It's visual, immediate, and perfectly designed to make you feel productive. But as any team scaling past their first few services quickly learns, "ClickOps" is the enemy of reliability.

The Hidden Cost of Manual Provisioning

Before our transition, spinning up a new environment was an oral tradition passed down from senior engineers to new hires. We documented it in Confluence, but the documentation was always a step behind reality. When incidents occurred, reproducing the exact state of a production service in a staging environment to debug it was nearly impossible. We were battling configuration drift on a weekly basis.

The Incident That Changed Everything

A manual update to a security group during a sev-1 incident accidentally exposed an internal service. It took us 45 minutes just to realize what had changed because there was no pull request or audit trail outside of CloudTrail logs.

The Migration Strategy

We didn't boil the ocean. We started with the foundational layers: networking (VPCs, Subnets, Route Tables) and IAM. Once the base was solid, we moved our RDS instances and finally, the compute layer (EKS/ECS).

vpc-module.tf
module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
  version = "5.0.0"

  name = "${var.environment}-main-vpc"
  cidr = var.vpc_cidr

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = var.environment != "prod"
}

What We Measured (The ROI)

  • MTTR Dropped by 60%

    Rollbacks became a simple git revert and terraform apply rather than a frantic click-hunt.

  • Environment Parity

    Staging finally became an exact replica of production, drastically reducing "it works on my machine" bugs.

Takeaway

Terraform isn't just about automation; it's about shifting infrastructure changes into the same review, test, and audit pipelines as your application code. State files require respect and proper remote backends, but the predictability they provide is unmatched.