An autonomous monitoring loop for the VilaVPN WHMCS stack, cloned from the battle-proven infra-sentinel skeleton. It watches business invariants — customers can log in, invoices exist, money settles exactly once, VPN auth answers — applies one bounded hot-fix only when provably safe, and turns everything else into /build-loop-ready plan docs. Full plan: ~/ssh/docs/runbooks/whmcs-sentinel-monitoring-loop-plan.md
ultracode workflow (7 agents incl. live read-only prod inventory) + /codex SRE catalog + /recall session history. Three discoveries reshaped the design:
ocserv VPN auth terminates on a dedicated radius box (172.104.80.179) — a fleet-wide SPOF with no host firewall, pointing at the master DB. Verified live: its freeradius container reads 139.162.106.129/radius. Master down ⇒ billing AND VPN auth down.
Every expensive past incident was a silent money bug: Payssion FX leaving paid invoices Unpaid, Stripe webhooks silently missing capture events, node traffic double-billing (4.2TB phantom). 3 of 4 panel report endpoints are still non-idempotent today.
grafana-agent + node/mysqld exporters already run on both main boxes. What's missing: synthetics, php-fpm status (250-worker pool, zero visibility), backup success, cert aggregation, radius canary, and all billing invariants.
Deterministic Python core owns every probe and every mutation. The LLM appears in exactly two caged roles — diagnosis (tools-stripped) and plan-writing (batch) — never in the fix path.
Highlights of the money tier: duplicate-transaction detector · Paid-but-underfunded / Unpaid-with-money invoices · the exact Payssion Convert-To fingerprint · Stripe webhook event-list drift (catches the silent revert risk) · invoice-generation weekday bands · node traffic double-billing detector · radius accounting freshness · signup/revenue dead-man.
A bug fixed once and for all in code can't silently recur — a probe for it is pure noise. Checks are spent only on regressable failures: config a WHMCS/module update can revert, a provider changing under us, or data-shape drift. Permanently-closed incidents (PassWall UA sync-invariant, VilaX2 UTF-8 byte-safety, xrayr rollback guard, no_drop QUIC, ApiPath bypass) are deliberately unmonitored.
| Incident (real, from history) | Cost then | Catcher now |
|---|---|---|
| Payssion Convert-To FX — paid invoices stuck Unpaid | revenue leak, manual fixes | B2 coherence + B3 fingerprint PAGE |
| Payssion secret-key leak (3 years, key still unrotated) | forged-pingback exposure | B3 replay/source telemetry + rotation-age (partial — reconciliation, not checksums) |
| Stripe uncaptured payments after WHMCS 8.13.3 | auto-voided renewals | B4 capture_method + aged requires_capture reconcile PAGE |
| php-fpm retry-storm (load 248, daily 02:00 recurrence) | site down nightly | H1 saturation + D5 backup correlation HOT-FIX |
| Node traffic double-billing (4.2TB phantom, 1,461 users) | customer over-billing | B7 duplicate-tuple detector PAGE |
| Stale redis node cache (5-min stale configs) | wrong node configs | B7 degraded-mode grep + H3 HOT-FIX |
| UA-validator too narrow ("no active packages" fleet-wide) | all Go clients dead | A6 UA regression matrix |
| Cert drift — vilavpn.net EXPIRED today, .jp at 30d | trust breakage | S4 all-domain expiry + A3 dry-run |
| DR replication break (DROP USER 1396 — live, 2026-07-15) | 7-min replica stall | D2 immediate SQL-error page |
Recon itself surfaced these. They seed findings.jsonl so the plan-writer lane produces its first /build-loop docs immediately:
Revised after codex round 1 (it correctly downgraded several v1 framings). Reply like: DEC1-A DEC2-A DEC3-A DEC4-A DEC5-A or just "all recommended".
A $5–10/mo box outside all four failure domains. Cleanest single-writer/fencing story, no coverage gap when your Mac sleeps or travels. Cost: one more box to keep patched.
Proven 24/7 ops model, ssh keys + claude CLI already here. But monitoring pauses when the Mac does — the external dead-man catches loop death, not the coverage gap while asleep.
24/7 and in-fleet — but the monitor then lives inside a fleet it monitors (a DR-box problem blinds the loop AND the DR) and adds load to the replica the money probes read.
Every primitive ships in shadow mode (journals "would have acted"), gets its own soak review, and is armed by an individual owner go. Order: nginx/logrotate reloads → fpm reload → radius container restart → DR dump rerun. Nothing arms before the M5 soak gate.
Never auto-fix; every incident pages you. Zero action risk — and 3AM php-fpm wedges wait for you to wake up, forever.
The v1 framing: arm reload-class fixes as one batch post-soak. Codex round 1 correctly flagged the bundling — one bad actor in the batch taints the go for all.
Server-local truth (connections, processlist, binlogs) runs MASTER-only; heavy money invariants run on the replica gated on fresh replication health — if D2 isn't fresh-PASS, the result is UNKNOWN_REPLICA_STALE, never a false certification. Compound checks name a target per sub-probe.
Freshest data — bought with load on a box already at 272MB free RAM and history of swap-thrash.
The v1 wording. Cheap, but a stalled replica could silently certify stale money invariants — exactly the class of silent failure this loop exists to kill.
Now: fpm status_path + backup exit-metric + external dead-man. Next: radius canary account once its NAS-source restriction is reviewed. Optional later: non-ASCII auth test account (until then the UTF-8 incident class stays visibly unmonitored, marked ☐).
Fastest to full coverage, but bundles unrelated prod changes into one change window — harder to attribute if anything regresses.
No prod changes; H1 and the radius canary degrade to inference (process counts, port checks) — monitoring by inference is the anti-pattern this design bans.
T0 synthetics 60s · host/safety 120s · expensive DB 300s · billing 15–30min · assurance on calendar. Monotonic per-check timers, non-overlap locks, jitter — and the loop pages on its own overdue work.
One clock for everything. Fiction in practice — billing sweeps and restore drills can never fit a 60s frame, so the "base cycle" becomes decorative.
Light, but ~15min to CONFIRMED is too slow for checkout-path breakage during the CN evening peak.