CI/CD Observability Stack

Overview

A Python webhook ingestion service that turns GitHub Actions events into Datadog metrics, plus Terraform-managed monitors, dashboards, and SLOs for platform health — reducing MTTR by 40%.

Tech Stack

Challenge

No observability into CI/CD platform. Pipeline failures reported via Slack complaints. 30% of sprint capacity spent on reactive incident response. No SLIs, SLOs, or error budgets — 'Is the platform working?' answered with gut feelings.

Solution

Python webhook ingestion service (FastAPI) processing GitHub workflow_job events into Datadog DogStatsD metrics. Terraform-managed observability layer with monitors, dashboards, and SLOs. Three-tier alerting: P1 (PagerDuty), P2 (Slack), P3 (digest).