GitOps State Capture & CI-Enforced Secrets Control

Situation

Configuration state for a four-node fleet lived only on the nodes themselves: no central capture, no change history, and no control preventing a credential from landing in version control. Losing a node meant rebuilding it from memory — an unbounded recovery time and an unauditable one.

Constraints: mixed operating systems (Windows and Linux), no budget for an enterprise backup product, GitHub free tier only, and a hard requirement that a credential can never be exposed even when the operator makes a mistake.

Task

• Capture every node's configuration and service state as code on a daily cadence, with a full audit trail.
• Sanitize secrets before commit and enforce the same rule in CI so nothing that slips the local filter can merge.
• Apply one policy across Windows and Linux with platform-native tooling.
• Deliver a tested rebuild path with a bounded recovery time objective.

Action

Defense in depth: the sanitizer strips secrets before commit, and the CI gate independently rejects anything that slips through.

• Linux collection (Nova, Mira, Orin). systemd user timers run a full state capture daily at 02:30 plus an hourly delta sync; Bash collectors gather system and service state.
• Windows collection (Jasper). PowerShell collectors capture OS, GPU, and service state via Task Scheduler on the same daily cadence — different tooling, identical policy.
• Pre-commit sanitization. Regex-based stripping of API keys, tokens, private keys, emails, and phone numbers on both platforms.
• CI secret-scanning gate. A GitHub Actions workflow scans every push against 11 secret patterns and a filename blocklist (.env, auth-profiles.json) and blocks the merge on any match.
• Segregated repository layout. Each node owns its directory, so a compromised or misbehaving node cannot overwrite another's state.
• Runbooks. Credential policy documentation, an incident-response runbook for a suspected leak, and a tested node-rebuild procedure.

Implementation

Result

• Recovery: clone the repository, follow the runbook, and a node is rebuilt in 30 minutes — a bounded, tested RTO instead of an unknown.
• Security: zero credential exposures since deployment, with two independent controls between an operator mistake and the remote.
• Auditability: every configuration change across 4 nodes is tracked in Git with timestamps and diffs.
• Consistency: one policy enforced identically on Windows and Linux.
• Self-documenting: each capture includes system specs, services, and health data, so the repository doubles as live infrastructure documentation.

Lessons & Next Steps

• Defense in depth for secrets. One layer is not enough. The sanitizer catches known patterns locally; the CI gate catches anything that evolves past the local filter.
• Different tooling, one policy. PowerShell and Bash implementations look different but enforce identical capture and sanitization rules.
• Untested backups are not backups. The restore runbook is validated by actually rebuilding nodes from the repository.
• Git is a strong state store. Versioned, diffable, hosted off-site, and with CI hooks built in.

Next Steps

• Add automated restore testing (monthly dry-run rebuild)
• Expand to backup VM snapshots metadata from Proxmox
• Add Slack/Telegram notification on backup failure