Infrastructure · Automation · Reliability Engineering — Fairfield, CA. Get in touch →
Back to Projects
Flagship Case Study

GitOps Backup & Secret Scanning

Automated compliance-grade backups with CI-enforced secrets hygiene — because one leaked API key can ruin your week.

Problem

Infrastructure configs were scattered across 4 nodes with no central backup, no change history, and no protection against accidentally committing secrets. If a node died, recreation would require remembering what was installed and configured — a disaster recovery nightmare.

Real constraint: Mixed OS fleet (Windows + Linux). No enterprise backup solution budget. Must work with GitHub free tier. Must never leak credentials even if someone (me) makes a mistake.

Architecture

Each Node (daily timer) → Collect State → Sanitize Secrets → Git Commit
                                                       ↓
                              Git Push → GitHub Actions CI Gate
                                           ↓           ↓
                                 ✅ Merge    🚫 Block + Alert

Two layers of defense: the sanitize script strips secrets before commit, and the CI gate catches anything that slips through.

Implementation

Linux Nodes (Nova/Mira/Orin)
  • lab-backup.sh — Collect system + OpenClaw state
  • lab-sanitize.sh — Regex-based secret stripping
  • lab-git-push.sh — Lightweight hourly sync
  • install_timer.sh — systemd user timer setup
  • Daily full backup at 02:30 + hourly delta sync
Windows Node (Jasper)
  • LabBackup.ps1 — Collect OS, GPU, OpenClaw state
  • Sanitize.ps1 — PowerShell secret stripping
  • SetupScheduledTask.ps1 — Task Scheduler setup
  • Daily backup at 02:30 AM
CI Gate
  • verify-no-secrets.yml — GitHub Actions workflow
  • 11 regex patterns (API keys, tokens, private keys, etc.)
  • Filename blocklist (.env, auth-profiles.json)

Results

4
Nodes backed up
11
Secret patterns blocked
0
Credential leaks
24h
Max data loss window
  • Full disaster recovery: clone repo → follow runbook → 30 minutes to rebuild a node
  • Audit trail: every config change tracked in Git with timestamps
  • Cross-platform: works identically on Windows and Linux
  • Self-documenting: each backup includes system specs, services, and health data

Lessons Learned

  • Defense in depth for secrets. One layer isn't enough. The sanitize script catches known patterns locally; the CI gate catches anything that evolves past the local filter.
  • Windows and Linux need different tooling, same policy. PowerShell and Bash scripts look different but enforce identical backup and sanitization rules.
  • Backups you don't test aren't backups. The restore runbook is tested by actually rebuilding nodes from the repo.
  • Git as a backup medium is underrated. Free, versioned, diffable, hosted offsite, with built-in CI hooks.
Next Steps
  • Add automated restore testing (monthly dry-run rebuild)
  • Expand to backup VM snapshots metadata from Proxmox
  • Add Slack/Telegram notification on backup failure
GitHub Actions CI run showing secret scanning results — 11 patterns checked, zero secrets detected across 4 nodes

Redacted sample (structure accurate; sensitive values removed).

What to look at: Secret-scan CI gate with 11 pattern checks, per-node backup status with timestamps, and restore readiness indicator.

Evidence: Secret Scanning CI Gate (sanitized)

# .github/workflows/verify-no-secrets.yml (excerpt)
- name: Scan for secrets
  run: |
    PATTERNS=(
      'AKIA[0-9A-Z]{16}'           # AWS Access Key
      'ghp_[a-zA-Z0-9]{36}'        # GitHub PAT
      '-----BEGIN.*PRIVATE KEY-----' # Private keys
      'sk-[a-zA-Z0-9]{48}'         # OpenAI API key
      'token.*=.*[a-f0-9]{32,}'    # Generic tokens
    )
    # 11 patterns total — blocks push on match
4
Nodes backed up
11
Patterns blocked
0
Credential leaks
24h
Max data loss
30m
Node rebuild time

Related artifacts: Proof Pack →

View Code on GitHub View Proof Pack SRE Pipeline Case Study →