Self-Hosted GitHub Actions Runner Platform

Overview

An autoscaled, ephemeral runner fleet on AWS — a Python autoscaler consuming GitHub webhooks, instance orchestrators managing runner lifecycle, resource monitors, Packer-built AMIs, and Terraform infrastructure supporting 500+ concurrent runners.

Tech Stack

Challenge

GitHub-hosted runners cost $6,400+/month at 10K+ executions/month, growing 15% month-over-month. Queue times exceeded 5 minutes during peak. Builds needed Docker-in-Docker, 32GB+ RAM, and custom kernel configs.

Solution

Three interconnected systems: (1) Python autoscaler (FastAPI) consuming GitHub workflow_job webhooks, driving ASG capacity via Redis state with reactive-plus-predictive sizing, (2) Instance orchestrators with ephemeral detach-on-start, terminate-on-complete pattern, (3) Resource monitors (psutil) for fleet telemetry. Packer-built AMIs with SHA-based version pinning.