# Reliability

Design for retries, failover, backups, health-gated changes, and recovery you can verify.

Ferrite provides durable primitives and fenced failover, but your application
still decides when an effect is safe to retry and what constitutes recovery.

## Build for retry

- Attach an idempotency key to every retried mutation.
- Persist intent before publishing background work.
- Acknowledge queue leases only after the durable effect.
- Verify provider webhooks over exact bytes and deduplicate event IDs.
- Treat stale scheduled jobs as no-ops after reading current domain state.

Run a local fault exercise:

```bash
ferrite dev --instances 3 --chaos 5
```

## Health and readiness

Expose a cheap liveness route and a readiness route that proves the app can
serve its required dependencies. Do not report ready while a required resource,
policy generation, or key handle is unavailable.

## Backup and restore

Replication keeps service available; backup protects against deletion,
corruption, or a bad authoritative decision. Ferrite backup exports and restores
all five durable storage planes with content verification. Test restoration
into an isolated target and verify application invariants before cutover.

## Roll forward or back

Use immutable releases and generation fencing. After any recovery action,
verify control-plane status, the public route, and the most important
authenticated product journey separately.

See [Scaling & Releases](/developers/scaling-releases/) and [Logs](/developers/logs/).