2026-03-05 · 8 min read

Building an SRE Reliability Pipeline in a Homelab

Most homelab projects stop at "it works." This one adds SLOs, burn-rate alerting, incident management, and auto-generated postmortems — the same practices used at companies running production infrastructure at scale.

Why reliability matters (even in a homelab)

"It's up" is not a reliability strategy. Services fail. Disks fill. Configs drift. Without measurement, you don't know if your systems are reliable — you just know they haven't broken yet.

SRE practices give you a framework to answer: How reliable is this? Is it getting better or worse? When should I invest in fixing it?

Applying this to a homelab also builds real skills. Employers hiring for SRE, platform, and DevSecOps roles want to see that you understand error budgets, incident response, and operational discipline — not just that you can spin up a VM.

Architecture of the lab

The pipeline runs across a 4-node cluster:

• Jasper — GPU inference node (i9-13900K, RTX 4090, 64 GB). Runs Ollama for local LLM serving and acts as the OpenClaw gateway.
• Nova — Controller node (N305, 32 GB DDR5). Runs Ansible, the SRE pipeline, and storage services.
• Mira — Utility compute (i7-2600K, 16 GB). Agent execution and secondary workloads.
• Orin — Server workloads (Dell R630, Dual Xeon, 16 GB ECC). Heavier compute tasks.

All nodes run on Proxmox (except Jasper on Windows) with OPNsense as the firewall and VLAN segmentation between infrastructure, IoT, and personal devices.

Designing SLOs

The first step is deciding what "reliable" means for each service. I defined 6 SLOs:

Each SLO is evaluated across 5 sliding time windows: 1 hour, 6 hours, 24 hours, 7 days, and 30 days. Short windows catch acute issues; longer windows reveal chronic degradation.

The error budget is the gap between the target and 100%. For a 99.9% SLO over 30 days, that's ~43 minutes of allowed downtime. When the budget is exhausted, the safety gate blocks risky automation.

Burn-rate alerting

Raw uptime percentages tell you what happened. Burn rate tells you what's about to happen.

The burn-rate calculator tracks how fast error budget is being consumed:

• < 1.0× — Normal. Budget is being consumed slower than the window allows.
• 1.0× – 2.0× — Warning. Budget will be exhausted before the window closes at this rate.
• > 2.0× — Critical. SLO breach is likely. Safety gates activate.

This is the same approach Google describes in the SRE Workbook, adapted to a homelab scale. The math is identical — the difference is the number of services, not the methodology.

Incident management workflow

When an SLO breaches or an anomaly is detected, the incident manager automatically:

1. Creates an incident with severity classification based on impact and budget state
2. Tracks the timeline — every state change is logged with timestamps
3. Monitors resolution — TTR (time to resolve) is tracked from detection to close
4. Generates a postmortem — auto-created with timeline, root cause analysis, and action items

Average TTR across all incidents: 13 minutes. Zero false positives since deployment.

Auto-generated postmortems

Every resolved incident produces a postmortem document containing:

• Incident summary and severity
• Chronological timeline of events
• Root cause analysis
• Contributing factors
• Action items with owners

These are stored in the Lab repository and published through the portfolio's Proof Pack. A recruiter can read a real postmortem, not a hypothetical exercise.

Safety gates

The gatekeeper is a simple but critical component: it prevents automated actions when the system is already degraded.

If a service's error budget is exhausted, or if there are active unresolved incidents, the gate blocks any automation that could make things worse. This is the same principle behind deployment freezes in production — don't push changes when you're already on fire.

What I learned

• Start with 2-3 SLOs, not 20. It's tempting to measure everything. Don't. Pick the services that matter most and instrument those first.
• Error budgets change behavior. Once you can see budget consumption, you naturally prioritize reliability work during burn spikes and feature work when budgets are healthy.
• Postmortems are the most valuable artifact. Not for blame — for patterns. After a few incidents, you start seeing systemic issues you'd never catch from monitoring alone.
• The safety gate paid for itself immediately. Within the first week, it blocked an automated config change during a degraded state that would have caused a cascading failure.
• Testing matters more than you think. 38+ acceptance tests catch regressions in the pipeline itself. The pipeline is infrastructure — treat it like infrastructure.

Results

---