Projects
Real infrastructure, real automation, real results. Each project solves a specific problem with measurable outcomes.
Flagship Case Studies
STAR-format deep dives into the engineering behind the platform.
AI Cluster Orchestration Platform (Four-Node Private Cloud)
Situation: Metered cloud inference APIs created cost, rate-limit, and vendor-lock-in risk, and a fleet of heterogeneous nodes had no shared scheduler, state, or recovery path.
Task: Stand up self-hosted GPU inference behind a control plane that schedules, heals, and rebuilds itself.
Action: Built a four-node private cloud — RTX 4090 inference gateway, Proxmox/TrueNAS controller, two compute workers — and an orchestration layer with adaptive routing, shared-nothing state, approval-gated automation, and a monthly sandboxed DR drill.
Result: Three LLMs (16B/32B/70B) served locally at zero per-token cost with sub-second responses, a 99.9% gateway availability SLO, and disaster recovery that is validated every month.
What I Built
- Jasper (i9-13900K, 64GB, RTX 4090) — GPU inference gateway running Ollama
- Nova (N305, 32GB) — Proxmox + TrueNAS + Ansible controller
- Mira (i7-2600K, 16GB) + Orin (Dual Xeon E5-2667v4, 16GB) — compute fleet
- OpenClaw AI agent orchestration with multi-model routing
- 52 cores / 80 threads / 128 GB RAM / 24 GB VRAM total cluster resources
SRE Automation Pipeline
Situation: No reliability instrumentation: failures were found by hand hours later, and there was no objective signal for when automation was safe to run.
Task: Implement SLO-driven reliability with automated incident response and no SaaS tooling.
Action: Built the pipeline from first principles — SLI collection, error budgets, multi-window burn-rate alerting, an Incident Commander, generated postmortems, and a gatekeeper that blocks risky automation during budget burn.
Result: 6 SLOs across 5 rolling windows, incidents opened within 2 minutes of degradation, 13-minute average time to resolve, zero false-positive alerts, 100+ tests.
What I Built
- SLO engine: 6 objectives (gateway 99.9%, inference 99.5%, dashboard 99%)
- Error budget tracking with 5 rolling windows (1h, 6h, 24h, 7d, 28d)
- Multi-window burn rate alerting (fast/medium/slow thresholds)
- Incident Commander: auto-trigger, timeline, evidence packs, postmortem markdown
- Gatekeeper safety gates: deny risky actions when burn rate is high
- 100+ tests across the full pipeline
GitOps Backup & Secret Scanning
Situation: Configuration state lived only on the nodes — no central capture, no change history, no control against a credential reaching version control.
Task: Capture every node's state as code daily, sanitize before commit, enforce in CI, and deliver a tested rebuild path across Windows and Linux.
Action: Platform-native collectors (systemd timers, Task Scheduler), regex sanitization pre-commit, a GitHub Actions gate scanning 11 secret patterns plus a filename blocklist, segregated per-node repo layout, and rebuild runbooks.
Result: 4 nodes committing daily, 11 patterns enforced in CI, zero credential exposure since deployment, 30-minute node rebuild, full Git audit trail.
What I Built
- Per-node backup scripts: systemd timers (Linux) + scheduled tasks (Windows)
- Sanitize pipeline stripping API keys, tokens, emails, phone numbers before commit
- GitHub Actions workflow scanning every push for 11 secret patterns
- Credential policy documentation and incident response runbook
- Segregated repo structure: each node owns its folder
Hypervisor Cluster & Software-Defined Storage
Situation: Services, testing, and development needed isolated environments across heterogeneous hardware with no new capital spend.
Task: Deliver virtualization and centralized storage managed entirely as code.
Action: Three-node Proxmox VE cluster (three of the lab's four nodes), TrueNAS ZFS with NFS/SMB exports for shared VM storage, Ansible-provisioned VMs, and automated health checks every 6 hours.
Result: 10+ VMs and containers on mixed consumer-to-enterprise hardware (N305, i7-2600K, dual Xeon on a Dell PowerEdge R630), all declared in infrastructure as code.
What I Built
- 3-node Proxmox VE cluster (Nova + Mira + Orin)
- TrueNAS on Nova: NFS/SMB shares for centralized VM storage
- Ansible playbooks for VM provisioning and configuration
- Automated health checks every 6 hours
- Mixed hardware fleet: consumer to enterprise (Dell PowerEdge R630)
Zero-Trust Network Segmentation
Situation: A flat network mixing inference, storage, and management traffic gave any compromise an unnecessarily wide blast radius.
Task: Reduce blast radius through segmentation and eliminate password-based access entirely.
Action: Deployed OPNsense on a dedicated appliance, segmented management, storage, inference, and IoT onto separate VLANs with explicit firewall policy, layered an identity-based mesh overlay (Tailscale) for remote access, enforced SSH key-only authentication, and separated 2.5GbE and 10GbE segments for performance-sensitive traffic.
Result: Traffic isolated between planes; no password-based access to any system; remote access requires device identity rather than open ports.
What I Built
- OPNsense firewall on Qotom Q20342G9 dedicated appliance
- VLAN segmentation: management, storage, inference, IoT
- UniFi U7 Pro XG access point (WiFi 7)
- Tailscale mesh VPN for zero-trust remote access
- 2.5GbE + 10GbE network segments for performance-sensitive traffic
- SSH key-only authentication across all nodes
Chaos Testing & Resilience Engineering
Situation: Self-healing and recovery claims were untested; resilience was assumed rather than demonstrated.
Task: Prove recovery under failure continuously, and stop changes that regress reliability.
Action: Built a chaos injection framework (service kills, resource saturation, network partitions) with automatic recovery validation, a resilience score with regression gates, SLO-pipeline integration so chaos failures open incidents, and evidence-pack collection for analysis.
Result: 95% chaos success-rate target, resilience score tracked over time, and deploys blocked automatically when reliability regresses.
What I Built
- Chaos injection framework: service kills, resource saturation, network partitions
- Resilience score engine: quantified reliability with regression gates
- Automatic recovery validation after each chaos experiment
- Integration with SLO pipeline — chaos failures trigger incident management
- Evidence pack collection for post-chaos analysis