Platform Architecture
AI Cluster Orchestration Platform
A self-healing orchestration control plane for a four-node high-availability private cloud. It schedules infrastructure work by node health and role, maintains distributed operational state on snapshot-protected storage, and automates recovery — with human approval gates on anything destructive.
The platform is designed around five architectural planes — control, compute, knowledge, reliability, and observability — each with clear responsibilities and well-defined interfaces. Every subsystem is built for independent operation: nodes can function when the coordinator is unavailable, shared knowledge survives individual node failures, and the entire orchestrator can be rebuilt from a portable recovery bundle.
Platform Architecture
The system is organized into five planes, each responsible for a distinct operational concern. This layered model keeps responsibilities clear and allows each plane to evolve independently.
Control Plane
Coordinates routing decisions, job dispatch, autonomous task generation, and scheduled maintenance across the cluster.
Compute Plane
Distributed worker nodes poll for assigned tasks, execute them within their role specialization, and report results back to shared storage.
Knowledge Plane
Distributed operational state stored as plain-text records on snapshot-protected network storage (immutable, versioned, diffable). Each node maintains a rebuildable local semantic index.
Reliability Plane
Protects system integrity through automated watchdogs, recovery workflows, snapshot-backed storage, portable recovery bundles, and recurring DR validation.
Observability Plane
Surfaces cluster health, queue state, node performance metrics, and weekly operational snapshots through dashboards and structured reports.
Design Principle
Each plane operates independently. The compute plane functions when the coordinator is offline. Knowledge survives node failures. The system can be rebuilt from the recovery bundle alone.
Node Topology
Click any node to explore its role, hardware, and services.
Distributed Orchestration
The orchestrator runs a continuous four-phase loop: check node health, dispatch queued jobs to the best available node, ingest completed results, and update cluster status. This cycle repeats on a fixed schedule, keeping the system responsive without requiring persistent connections between nodes.
Adaptive Routing
Each task is scored against candidate nodes using a blend of static role fitness, historical success rate, average execution time, and recency of last assignment. The routing model improves automatically as execution data accumulates — nodes that consistently complete certain task types faster and more reliably receive higher scores.
Durable Job Queue
Jobs are represented as individual JSON files on shared storage, moving through pending, running, done, and failed directories. This file-based approach eliminates shared database dependencies, prevents lock contention across nodes, and provides natural crash recovery — incomplete jobs are detected by stale-job timeout and returned to the queue.
Shared Knowledge Architecture
The cluster maintains a shared operational knowledge system designed around two principles: durable knowledge belongs in plain text on shared storage, and fast retrieval uses local indexes that can be rebuilt from the source at any time.
Immutable State Corpus
Plain-text operational records on snapshot-protected ZFS storage serve as the cluster's durable memory. Each node contributes daily observations; the control node curates long-term knowledge and cluster-wide context.
Local Semantic Indexes
Each node maintains its own local SQLite search index for fast retrieval. Shared knowledge stays in text files on network storage — never as shared databases — eliminating corruption risk and lock contention.
Nightly Curation
An automated curation process reviews daily notes, classifies content by type, promotes durable facts to curated documents, and archives old notes by week. Knowledge is refined, not accumulated.
Reliability & Recovery
The reliability plane implements defense in depth: multiple independent systems protect against different failure modes, from transient service disruptions to complete coordinator loss.
Self-Healing Watchdog
A runtime watchdog checks service health, mount integrity, and database consistency on a recurring schedule. When degradation is detected, automated recovery workflows quarantine bad state and restore from verified backups.
Recovery Bundle
The entire orchestration system — routing policy, worker configuration, queue structure, scheduling, and architecture documentation — is packaged into a portable recovery bundle on shared storage, ready for rapid rebuilds.
Monthly DR Drills
Automated disaster recovery drills run monthly in a sandboxed environment. Each drill validates the recovery bundle, confirms components would restore correctly, verifies the live system was not modified, then cleans up.
ZFS Snapshots
Storage is backed by ZFS with automated snapshot policies at three tiers: frequent snapshots retained for hours, daily snapshots retained for weeks, and weekly snapshots retained for months.
Conservative Safety Model
All automated actions follow a conservative policy: prefer quarantine over deletion, require preflight checks before service restart, never auto-execute destructive changes, and gate risky operations behind human approval.
Weekly Snapshots
Comprehensive weekly reports capture node health, mount status, queue state, archive growth, and curation metrics — providing a historical record of cluster operational health over time.
Automation & Self-Healing
The platform generates its own maintenance work. Rather than relying on manual intervention to detect and remediate issues, the orchestrator continuously monitors cluster health signals and creates appropriate tasks when action is needed.
Autonomous Task Generation
A rule engine evaluates cluster health signals — node availability, mount integrity, queue health, and performance trends — against a set of detection rules. When a rule triggers, it creates a task with appropriate classification: safe operations go directly to the queue, risky actions are placed in an approval queue for human review, and advisory-only detections are logged without creating tasks.
Approval Gates
Tasks classified as potentially destructive or high-impact are routed to a separate approval queue rather than executing automatically. This creates a clear boundary between what the system can do safely on its own and what requires human judgment. Cooldown tracking prevents duplicate task creation when the same condition is detected repeatedly.
What This Demonstrates
This platform is a working demonstration of the engineering practices and design thinking I bring to infrastructure work.
Systems Thinking
The cluster is designed as a cohesive system — not a collection of scripts. Routing, health, memory, curation, recovery, and orchestration all interact through well-defined interfaces on shared storage.
Reliability Engineering
Multiple layers of defense protect against data loss and service degradation: preflight guards, runtime watchdogs, automated recovery, snapshot-backed storage, and recurring DR validation.
Infrastructure Automation
Every recurring operation is automated and scheduled. Manual intervention is reserved for approval decisions and architecture changes — not for keeping the lights on.
Distributed Systems Design
Shared-nothing memory architecture, file-based coordination, health-aware dispatch, and graceful degradation — built for the real failure modes of multi-node infrastructure.
Operational Documentation
Every subsystem has architecture documentation, scheduling references, and clear rollback procedures. Operational knowledge is curated automatically, not left in chat logs or memory.
Recovery-First Design
The system is designed to be rebuilt, not just maintained. Portable recovery bundles, validated DR drills, and conservative safety policies ensure resilience isn't just a feature — it's the foundation.
Infrastructure Details
Local AI Inference
RTX 4090 running multiple large language models via Ollama — no cloud API dependencies for core inference workloads.
Proxmox Cluster
Three-node Proxmox VE hypervisor cluster (three of the four nodes) with centralized TrueNAS/ZFS software-defined storage and OPNsense network segmentation.
Security Posture
Zero-trust posture: OPNsense perimeter firewall, VLAN micro-segmentation, identity-based mesh access (Tailscale), SSH key-only authentication, CI-enforced secret scanning, and a written credential hygiene policy.
Case Study: Situation, Task, Action, Result
The platform in STAR form — what existed, what had to change, what was built, and what it delivers.
Situation
Four heterogeneous nodes — a GPU inference gateway, a hypervisor and storage controller, and two compute workers — ran independent scripts with no shared scheduler, no shared state, and no automated recovery. Every failure required manual diagnosis over SSH, and losing the coordinator meant rebuilding it from memory with an unbounded recovery time.
Task
Design a control plane that schedules work across the fleet by health and role, keeps operational state durable and shared without a database single point of failure, detects and repairs its own degradation, bounds the blast radius of any automated action, and can be rebuilt from scratch on demand — then prove that recovery on a schedule instead of assuming it.
Action
- Separated concerns into five planes (control, compute, knowledge, reliability, observability) with well-defined interfaces on shared storage, so each plane degrades and evolves independently.
- Built an adaptive scheduler that scores each job against node role fitness, historical success rate, execution time, and recency, so routing policy self-tunes from execution data.
- Implemented a database-free durable work queue — jobs as individual records moving through pending, running, done, and failed states on shared storage, with stale-lease detection returning crashed work to the queue.
- Designed shared-nothing state management: plain-text operational records on ZFS with three-tier snapshot retention, rebuildable per-node semantic indexes, and nightly curation that promotes durable facts and archives noise.
- Added self-healing: a runtime watchdog checks service health, mount integrity, and index consistency, quarantines corrupt state, and restores from verified backups; preflight guards refuse to start on inconsistent state.
- Bounded the blast radius: a rule engine turns health signals into tasks, safe actions run automatically, destructive or high-impact actions wait in an approval queue, and cooldowns prevent task storms.
- Made recovery a tested property: packaged the entire control plane into a portable recovery bundle and run monthly sandboxed DR drills that validate the rebuild without touching production state.
- Instrumented everything: weekly operational snapshots of node health, mount status, queue state, archive growth, and curation metrics feed the status dashboard and the SRE pipeline.
Result
- Graceful degradation: workers keep executing when the coordinator is offline, and shared state survives any single node failure.
- Rehearsed recovery: the full control plane is rebuilt from the recovery bundle in a sandbox every month, so recovery is measured rather than assumed.
- Contained automation: no destructive action has ever executed without human approval; safe remediation runs without an operator in the loop.
- Self-tuning operations: routing quality improves as execution history grows, with no manual rebalancing.
- Operator time reclaimed: manual effort is reserved for approval decisions and architecture changes, not for keeping services alive.
- No external inference dependency: self-hosted LLM inference on the RTX 4090 gateway removes per-token cost and third-party availability risk from core workloads.
All infrastructure code, automation, and documentation are open source.