This projects shows how to create a Kubernetes cluster provided with a set of autoscaling self-hosted runners handling Github Actions and driven by ARC and Karpenter.

Any existing Github action is able to point to this infra just by adding the ARC runner label in the runs-on directing in the GH Actions manifest.

A placeholder application and relative GH workflows are provided to show of this can be seamlessly usded in any dev/production environment.

The whole system while be composed on 3 pilars:

• ARC system: allowing deployment of self-hosted runner on this cluster
• Dagger system: a DaemonSet running the Dagger engine on each allocated ARC runner
• Buildkit: providing consistent caches across multiple runs of building of Docker images
• Karpenter (on AWS EKS cluster): handling scaling, upsizing, downszing of node pools which hanldes the runner according to specific policies

Terraform plan provisioning by spinning:

• one Kube cluster (EKS in this case but can be KinD locally for dev testing)
• generic nodes for Karpenter scheduler

See Provisioning

For local development a KinD cluster can be used is. The config at cluster/kind/kind.yaml pre-labels the control-plane node with dagger.sh/engine=true, which is required by the Dagger engine DaemonSet's node affinity.

kind create cluster --name arc-ci --config=cluster/kind/kind.yaml

Helm and Devspace spin the full deployment of releases, manifests

Karpenter handles autoscaling of self-hosted runners using different classes that are taylored to the type of workload.

As a placeholder 3 classes are provided, they should be tuned but aim to mimic different kind of workloads:

• medium
• large
• xlarge

For specifik cases (e.g. building Docker images) a persistent layer provided with Buildkit can be added. it allows to have consistent Buildkit caches across multiple runs of any workload using any Docker buliding procedure.

• authentication too Github API to target the Github Repository via Token or Github App. See Authenticating ARC to the GitHub API - GitHub Docs
• a target Kubernetes cluster

• Run

cd cluster/devspace
devspace deploy

• Run (including Karpenter node pools)

cd cluster/devspace
devspace run-pipeline deploy-prod

• Run

cd cluster/devspace
devspace purge

• Kubernetes | Dagger
• On-Demand Dagger Engines with Argo CD, EKS, and Karpenter | Dagger
• matipan/dynamic-dagger-engines at 2024-05-20

• Quickstart for Actions Runner Controller - GitHub Docs
• Actions Runner Controller - GitHub Docs
• actions-runner-controller/charts/gha-runner-scale-set/values.yaml at master · actions/actions-runner-controller

• NodePools | Karpenter
• terraform-aws-eks/modules/karpenter at master · terraform-aws-modules/terraform-aws-eks
• Set up the Node Pool | EKS Workshop
• Karpenter Workshop (Immersion Day)
• Running Efficient Kubernetes Clusters on Amazon EC2 with EKS, Karpenter, EC2 Spot, and Graviton
• karpenter-blueprints/blueprints/disruption-budgets at main · aws-samples/karpenter-blueprints