Hands-on chaos engineering on AWS: 10 deployable Terraform labs using AWS Fault Injection Service (FIS). Each lab is a small, self-contained scenario that breaks a real AWS architecture on purpose, measures what happens, and proves whether the system is as resilient as its diagram claims.
This is the companion code for the "Chaos Engineering on AWS" article series on the Toc Consulting blog.
Ten independent labs, each in its own folder with a complete terraform/ configuration you can deploy, exercise, and destroy.
| # | Lab (code) | Article | What it breaks | Primary FIS action |
|---|---|---|---|---|
| 1 | 01-what-is-chaos-engineering | Read | An EC2 instance behind an ALB and Auto Scaling group | aws:ec2:stop-instances |
| 2 | 02-observability-first | Read | EC2 instances, with CloudWatch alarms wired as a stop condition | aws:ec2:terminate-instances |
| 3 | 03-database-resilience | Read | Aurora writer, with and without RDS Proxy | aws:rds:failover-db-cluster |
| 4 | 04-graceful-degradation | Read | The database backend, behind a circuit breaker, SQS buffer, and DynamoDB cache | RDS Proxy target removal |
| 5 | 05-stress-and-network-faults | Read | CPU, memory, and the network on a whole fleet | aws:ssm:send-command (AWSFIS-Run-*) |
| 6 | 06-availability-zone-failure | Read | An entire Availability Zone | aws:network:disrupt-connectivity |
| 7 | 07-dependency-and-api-faults | Read | Connectivity to DynamoDB and S3 endpoints | aws:network:disrupt-connectivity |
| 8 | 08-containers-and-serverless | Read | An ECS Fargate service and an AWS Lambda function | aws:ecs:stop-task, aws:lambda:invocation-error, aws:lambda:invocation-add-delay |
| 9 | 09-multi-region-dr | Read | Cross-Region DynamoDB global table replication | aws:dynamodb:global-table-pause-replication |
| 10 | 10-chaos-program | Read | An EC2 instance, on a recurring unattended schedule | aws:ec2:stop-instances via EventBridge Scheduler |
Each lab folder contains the Terraform, any application code, the experiment scripts, and a short README explaining the scenario.
Read this before you deploy anything.
- These labs create real, billable AWS resources (EC2, Aurora, NAT gateways, ECS, Lambda, DynamoDB, and more). They are designed to be cheap and short-lived, not free.
- Always run them in a dedicated sandbox account, never in production.
- Always run
terraform destroyas soon as you finish a lab. Several labs leave resources running until you tear them down, and lab 10 starts a new experiment on a schedule until you remove it. - Chaos experiments cause real failures. Only run them against infrastructure you own and can afford to break.
- Terraform 1.5 or newer
- The AWS provider 5.x (pinned per lab in
.terraform.lock.hcl, installed automatically byterraform init) - An AWS account and a named CLI profile with permission to create the resources above and to use AWS FIS
- The AWS CLI v2
- Python 3.12 or newer (used by the traffic and probe scripts; matches the
python3.12Lambda runtime)
Every lab follows the same loop. Replace your-profile with your AWS CLI profile name.
cd 01-what-is-chaos-engineering/terraform
# 1. Initialize providers
terraform init
# 2. Deploy. The provider reads the profile from this variable, so pass it explicitly.
terraform apply -var="aws_profile=your-profile"
# 3. Run the experiment. Each lab documents its own command in the lab README.
# The scripts read your default credentials, so export the profile for them too.
export AWS_PROFILE=your-profile
bash scripts/verify-experiment.sh
# 4. Destroy everything when you are done.
terraform destroy -var="aws_profile=your-profile"The default Region is us-east-1. Override it with -var="region=eu-west-1" (lab 9 is multi-Region and uses its own region_primary and region_secondary variables).
- Every resource name is prefixed
chaos-lab-. - Default tags
Project = chaos-lab,Environment = sandbox,ManagedBy = terraformare applied to all resources, and FIS targets and roles are scoped to them. - Compute runs on Amazon Linux 2023. Database passwords are generated and stored in SSM Parameter Store or Secrets Manager, never hardcoded.
- The Lambda packages in lab 4 and lab 8 are built by
terraform apply(pip install and archive), so no build artifacts are committed.
See LICENSE.
This code is provided as educational material. You are responsible for any AWS charges it incurs and for any impact of running fault-injection experiments. Always use a sandbox account and always destroy resources after use.
