Skip to content

TocConsulting/chaos-on-aws

Repository files navigation

Chaos Engineering on AWS

Chaos Engineering on AWS: deployable AWS FIS and Terraform labs

Hands-on chaos engineering on AWS: 10 deployable Terraform labs using AWS Fault Injection Service (FIS). Each lab is a small, self-contained scenario that breaks a real AWS architecture on purpose, measures what happens, and proves whether the system is as resilient as its diagram claims.

This is the companion code for the "Chaos Engineering on AWS" article series on the Toc Consulting blog.

What is in here

Ten independent labs, each in its own folder with a complete terraform/ configuration you can deploy, exercise, and destroy.

# Lab (code) Article What it breaks Primary FIS action
1 01-what-is-chaos-engineering Read An EC2 instance behind an ALB and Auto Scaling group aws:ec2:stop-instances
2 02-observability-first Read EC2 instances, with CloudWatch alarms wired as a stop condition aws:ec2:terminate-instances
3 03-database-resilience Read Aurora writer, with and without RDS Proxy aws:rds:failover-db-cluster
4 04-graceful-degradation Read The database backend, behind a circuit breaker, SQS buffer, and DynamoDB cache RDS Proxy target removal
5 05-stress-and-network-faults Read CPU, memory, and the network on a whole fleet aws:ssm:send-command (AWSFIS-Run-*)
6 06-availability-zone-failure Read An entire Availability Zone aws:network:disrupt-connectivity
7 07-dependency-and-api-faults Read Connectivity to DynamoDB and S3 endpoints aws:network:disrupt-connectivity
8 08-containers-and-serverless Read An ECS Fargate service and an AWS Lambda function aws:ecs:stop-task, aws:lambda:invocation-error, aws:lambda:invocation-add-delay
9 09-multi-region-dr Read Cross-Region DynamoDB global table replication aws:dynamodb:global-table-pause-replication
10 10-chaos-program Read An EC2 instance, on a recurring unattended schedule aws:ec2:stop-instances via EventBridge Scheduler

Each lab folder contains the Terraform, any application code, the experiment scripts, and a short README explaining the scenario.

Cost and safety

Read this before you deploy anything.

  • These labs create real, billable AWS resources (EC2, Aurora, NAT gateways, ECS, Lambda, DynamoDB, and more). They are designed to be cheap and short-lived, not free.
  • Always run them in a dedicated sandbox account, never in production.
  • Always run terraform destroy as soon as you finish a lab. Several labs leave resources running until you tear them down, and lab 10 starts a new experiment on a schedule until you remove it.
  • Chaos experiments cause real failures. Only run them against infrastructure you own and can afford to break.

Prerequisites

  • Terraform 1.5 or newer
  • The AWS provider 5.x (pinned per lab in .terraform.lock.hcl, installed automatically by terraform init)
  • An AWS account and a named CLI profile with permission to create the resources above and to use AWS FIS
  • The AWS CLI v2
  • Python 3.12 or newer (used by the traffic and probe scripts; matches the python3.12 Lambda runtime)

Running a lab

Every lab follows the same loop. Replace your-profile with your AWS CLI profile name.

cd 01-what-is-chaos-engineering/terraform

# 1. Initialize providers
terraform init

# 2. Deploy. The provider reads the profile from this variable, so pass it explicitly.
terraform apply -var="aws_profile=your-profile"

# 3. Run the experiment. Each lab documents its own command in the lab README.
#    The scripts read your default credentials, so export the profile for them too.
export AWS_PROFILE=your-profile
bash scripts/verify-experiment.sh

# 4. Destroy everything when you are done.
terraform destroy -var="aws_profile=your-profile"

The default Region is us-east-1. Override it with -var="region=eu-west-1" (lab 9 is multi-Region and uses its own region_primary and region_secondary variables).

Conventions

  • Every resource name is prefixed chaos-lab-.
  • Default tags Project = chaos-lab, Environment = sandbox, ManagedBy = terraform are applied to all resources, and FIS targets and roles are scoped to them.
  • Compute runs on Amazon Linux 2023. Database passwords are generated and stored in SSM Parameter Store or Secrets Manager, never hardcoded.
  • The Lambda packages in lab 4 and lab 8 are built by terraform apply (pip install and archive), so no build artifacts are committed.

License

See LICENSE.

Disclaimer

This code is provided as educational material. You are responsible for any AWS charges it incurs and for any impact of running fault-injection experiments. Always use a sandbox account and always destroy resources after use.

About

Hands-on chaos engineering on AWS: 10 deployable Terraform labs using AWS FIS (EC2, Aurora, ECS/Fargate, Lambda, multi-Region DR). Companion code for the Toc Consulting "Chaos Engineering" series

Topics

Resources

License

Stars

2 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors