Twenty CRM + PraeSyn: Disaster Recovery & Infrastructure Hardening Case Study

Client: PraeSyn Internal / LiftedOps
Service: Incident Response & Infrastructure Resilience
Recovery Time: ~30 minutes (June 11, 2026)
Key Result: CRM fully recovered with zero data loss after 4 days of silent outage; infrastructure hardened with hostname-over-IP policy to prevent recurrence

The Challenge

On June 11, 2026, the Twenty CRM instance at crm.praesyn.com was discovered down. The last successful deployment had been June 7 — 4 days of silent outage. No alerts fired because monitoring was still being built out.

Twenty CRM is PraeSyn's self-hosted customer database — the central system for LiftedOps lead capture. Leads flow from a Cloudflare Pages Function into Twenty's GraphQL API, creating Companies and People records linked together. When the CRM goes down, lead capture stops. Period.

Investigation revealed a single root cause with three manifestations: a Tailscale node re-registration on approximately June 9 had assigned new IP addresses to infrastructure nodes, but the Nomad job specs and Consul KV store still referenced the old IPs.

Root Cause Chain

Registry URL: The Nomad job spec tried to pull Docker images from 100.64.0.1:5000 — an IP that had moved to 100.64.0.5 after re-registration. The registry was actually at 100.64.0.4 (vps-1).
Health check path: The Nomad health check hit /health — returning 404. The actual NestJS endpoint was /healthz. Every allocation had been failing its health check.
Database and cache URLs: Consul KV values for PG_DATABASE_URL and REDIS_URL still pointed to 100.64.0.2 after vps-2 re-registered to 100.64.0.3. Redis was ENETUNREACH; PostgreSQL eventually connected after retries, masking the partial failure.

All three failures were silent. The Nomad job showed 15 failed allocations, 12 complete, and 1 lost — but no one was watching.

Our Approach

Immediate Recovery (30 minutes)

1. Diagnosed the full stack. Identified all three failure points by tracing Nomad allocation logs, container exit codes, and Consul KV values against current Tailscale IPs.

2. Patched the Nomad spec. Replaced all hardcoded IPs with Tailscale hostnames (gbyte, vps-1, vps-2) and fixed the health check path to /healthz.

3. Updated Consul KV. Corrected PG_DATABASE_URL and REDIS_URL to current vps-2 IP.

4. Pre-pulled the image from vps-1:5000 to vps-2 to avoid slow Tailscale registry pull during deployment.

5. Deployed. nomad job run — 1/1 healthy on both server and worker allocations.

Infrastructure Hardening

Post-recovery, PraeSyn implemented systematic hardening to prevent recurrence:

Hostname-over-IP policy: All Nomad job specs, Consul KV values, and Woodpecker pipeline configs now use Tailscale hostnames (gbyte, vps-1, vps-2) instead of raw 100.64.x.x IPs. Hostnames survive Tailscale re-registration; raw IPs do not.
Post-re-registration audit checklist: After any Tailscale node re-registration, run a systematic audit — grep all Nomad specs for raw IPs, scan all Consul KV values for stale addresses.
CI/CD pipeline commitment: Adopted the canonical 3-step Woodpecker pipeline pattern — build, pre-pull, deploy — matching the standard established by the RELM project.

The Results

Rapid recovery: CRM fully operational in ~30 minutes after 4 days of silent outage. Lead capture restored immediately.

Zero data loss: The PostgreSQL volume at /opt/nomad/volumes/crm_db/ (96MB) was intact across the entire outage — no backup restore needed. No service recreation required: crm-db and crm-redis Nomad jobs continued running throughout.

All users preserved: Workspaces, data, and OAuth logins intact — no user-visible data loss or configuration drift.

Systemic prevention: No hardcoded IPs remain anywhere in the CRM stack. The vps-1:5000 hostname pattern is durable against future Tailscale re-registrations.

Lessons Learned

Three principles emerged from this incident that now guide all PraeSyn infrastructure:

Stale IPs are silent killers. Tailscale re-registration makes any IP-based config drift. Hostnames are the fix — always.
Health check paths matter. One character difference (/health vs /healthz) caused every allocation to fail its health check, silently blocking deployment rollouts.
Consul KV is more fragile than Nomad specs. Nomad specs are git-tracked and re-evaluated on job run. Consul values are persistent and drift forever if not audited.

"This incident taught us that infrastructure resilience isn't about never failing — it's about designing systems where failures are visible, diagnosable, and recoverable in minutes instead of days. Hostname-over-IP, health-check verification, and Consul auditing are now table stakes for every PraeSyn deployment." — PraeSyn Infrastructure Team

Twenty CRM + PraeSyn