How is idempotency actually enforced end-to-end?

Every settlement intent carries an Idempotency-Key computed as SHA-256(tenant, intent_id, version). We persist that key together with a hash of the canonical request body in a strongly consistent store with a 30-day, per-tenant window. A duplicate submit with the same key and same body returns the original response byte-for-byte; a duplicate with the same key and a different body returns 409 Conflict. The ledger effect is written in the same transaction as the key, so there is no window where a duplicate can produce a second posting.

What exactly do you retry, and how do you avoid amplifying an outage?

Retries are first-class objects: a policy declares max attempts, base delay, cap and jitter (decorrelated, AWS-pattern) per counterparty. A per-tenant retry budget bounds the total in-flight retries so a rail outage cannot fan out into a thundering herd. When the budget is exhausted, new work is shed rather than queued. Non-retryable errors (schema, auth, poison) skip retries entirely and route straight to a DLQ.

What is the audit trail and how can we independently verify it?

The ledger is append-only. Every posting is content-addressed with blake3 and linked to the previous one via prev_hash — a hash chain. Once per epoch we compute a Merkle root over all postings in that epoch and publish it to an internal transparency log. Given the journal and any snapshot, an auditor can recompute the Merkle root and match it against the published value; any tampering with a historical posting breaks the chain.

How does the multi-region topology work, and what is the RPO across regions?

Within a region the engine runs active-active per tenant. Across regions we run active-standby: the journal streams continuously to the standby region, where a mirror engine folds the same events into the same state. Failover is a control-plane action that promotes the standby only once its offset matches the last durable offset of the primary. RPO within a region is 0; cross-region RPO is under 5 seconds and cross-region RTO is under 15 minutes.

How do you prevent split-brain during a network partition?

Each settlement partition is owned by exactly one worker via a fenced lease with a monotonic fencing token. TTL is 8s and heartbeats are 2s. Writes to the journal and downstream stores validate the token; a stale token is rejected even if the previous owner still believes it holds the lease. This means the classic partition-plus-GC-pause scenario cannot produce a second committer.

What happens if a downstream rail double-processes despite our idempotency?

Idempotency at our layer only guarantees at most one ledger effect per intent. If a counterparty rail acknowledges twice for the same submission, reconciliation compares the rail's own settlement report against our journal by external reference and quarantines any surplus credit. Corrections are always compensating entries — we never mutate history — and the resulting exception is surfaced in the operations console with a linked audit path.

How is state recovered after a worker or database failure?

State is a pure fold: state = fold(reduce, snapshot, journal[snapshot.offset:]). Snapshots are emitted every 10^6 events. On a worker crash a new owner acquires the lease and replays the journal from the last committed offset — p99 recovery is under 90 seconds on a hot node. On primary DB failover we reconnect, verify the last durable LSN, and resume; RPO is 0 because the journal is the source of truth, not the database.

How do we roll back a bad release without corrupting the ledger?

Reducers are versioned and pure. A bad release can only produce forward events; it cannot rewrite history. Rolling back the binary and replaying from the last known-good offset reproduces state deterministically. If a specific reducer version emitted incorrect postings, we issue compensating entries against those specific intents — the ledger still tells the full truth of what happened, including the correction.

For engineering leaders

Settlement Engine

The core that moves regulated money at Hostra. Designed around five properties that make it safe to build a bank on top of: exactly-once settlement, deterministic recovery, append-only ledger, lease ownership and retry orchestration.

Book technical demo Architecture PDF

Durability

11×9

Commit p99

38 ms

Recovery p99

< 90 s

RPO

Exactly-once settlement

Every settlement intent carries a client-side idempotency key. Duplicate submits collapse to a single ledger effect — safe to retry from any layer of your stack.

Idempotency key = SHA-256(tenant, intent_id, version)
Deduplication window: 30 days, per tenant, strongly consistent
Replay-safe: identical payload → identical response, byte-for-byte

Deterministic recovery

The engine reconstructs state by replaying the append-only journal against pure reducers. No manual reconciliation, no drift between primary and replica.

State = fold(reduce, snapshot, journal[snapshot.offset:])
Snapshots every 10⁶ events; recovery p99 < 90s on a hot node
Bit-for-bit reproducible: two nodes at the same offset compute the same hash

Append-only ledger

Double-entry, hash-chained, immutable. Every posting is signed and linked to the previous, so the ledger is auditable end-to-end without trust in the operator.

Postings are content-addressed (blake3) and chained by prev_hash
Merkle root emitted per epoch — verifiable by external auditors
Corrections are compensating entries, never in-place edits

Lease ownership

Each settlement partition is owned by exactly one worker via a fenced lease. Network partitions can't produce split-brain — a stale lease can't commit.

Lease TTL 8s, heartbeat every 2s, fencing token monotonic
Writes rejected on stale token — even if the old owner is still alive
Ownership handoff observed p99 < 4s across AZ failure

Retry orchestration

Retries are first-class objects, not a for-loop. Backoff, jitter, budget and dead-letter routing are declared per counterparty and inspected in the control plane.

Exponential backoff with decorrelated jitter (AWS pattern)
Per-tenant retry budget prevents thundering-herd on rail outage
DLQ replay is deterministic: same input → same downstream effect

Commit protocol

One intent, one effect, forever

settlement.commit()

POST /v1/settlement/intents
Idempotency-Key: 8f2a…c41d
{
  "tenant":   "acme",
  "amount":   { "value": "12500.00", "ccy": "EUR" },
  "debit":    "acct_9c…",
  "credit":   "acct_2a…",
  "rail":     "sepa_instant",
  "version":  1
}

→ 201 Created                    // first submit
→ 200 OK   (replayed=true)       // any subsequent submit, same key
→ 409 Conflict                   // same key, different payload

Ledger effect (append-only):
  posting#382914  debit  acct_9c…  -12500.00 EUR
  posting#382915  credit acct_2a…  +12500.00 EUR
  prev_hash = 6b1e…    blake3(entry) = a70c…

Recovery matrix

Every failure mode has an owner

We enumerate what can go wrong, how we detect it, what the engine does automatically, and the RTO / RPO we hold ourselves to.

Failure	Detected by	Automatic action	RTO	RPO
Worker crash mid-commit	Lease heartbeat lost (≤ 2s)	New owner replays journal from last committed offset	< 6s	0
Primary DB failover	Write error / topology change	Reconnect, verify last LSN, resume from journal	< 20s	0
Downstream rail timeout	No ACK within counterparty SLA	Retry with backoff, then quarantine to DLQ	N/A	0
AZ outage	Quorum loss on lease store	Ownership migrates to surviving AZ, journal reconciled	< 90s	0
Poisoned message	Reducer error, non-retryable	Isolate intent, route to DLQ, alert on-call	immediate	0
Full region loss	Cross-region health probes	Promote standby region from streaming journal	< 15m	< 5s

Worker crash mid-commit

Detected

Lease heartbeat lost (≤ 2s)

Action

New owner replays journal from last committed offset

RTO

< 6s

RPO

Primary DB failover

Detected

Write error / topology change

Action

Reconnect, verify last LSN, resume from journal

RTO

< 20s

RPO

Downstream rail timeout

Detected

No ACK within counterparty SLA

Action

Retry with backoff, then quarantine to DLQ

RTO

N/A

RPO

AZ outage

Detected

Quorum loss on lease store

Action

Ownership migrates to surviving AZ, journal reconciled

RTO

< 90s

RPO

Poisoned message

Detected

Reducer error, non-retryable

Action

Isolate intent, route to DLQ, alert on-call

RTO

immediate

RPO

Full region loss

Detected

Cross-region health probes

Action

Promote standby region from streaming journal

RTO

< 15m

RPO

< 5s

FAQ · for CTOs and staff engineers

The questions we get on the first call

How is idempotency actually enforced end-to-end?: Every settlement intent carries an Idempotency-Key computed as SHA-256(tenant, intent_id, version). We persist that key together with a hash of the canonical request body in a strongly consistent store with a 30-day, per-tenant window. A duplicate submit with the same key and same body returns the original response byte-for-byte; a duplicate with the same key and a different body returns 409 Conflict. The ledger effect is written in the same transaction as the key, so there is no window where a duplicate can produce a second posting.
What exactly do you retry, and how do you avoid amplifying an outage?: Retries are first-class objects: a policy declares max attempts, base delay, cap and jitter (decorrelated, AWS-pattern) per counterparty. A per-tenant retry budget bounds the total in-flight retries so a rail outage cannot fan out into a thundering herd. When the budget is exhausted, new work is shed rather than queued. Non-retryable errors (schema, auth, poison) skip retries entirely and route straight to a DLQ.
What is the audit trail and how can we independently verify it?: The ledger is append-only. Every posting is content-addressed with blake3 and linked to the previous one via prev_hash — a hash chain. Once per epoch we compute a Merkle root over all postings in that epoch and publish it to an internal transparency log. Given the journal and any snapshot, an auditor can recompute the Merkle root and match it against the published value; any tampering with a historical posting breaks the chain.
How does the multi-region topology work, and what is the RPO across regions?: Within a region the engine runs active-active per tenant. Across regions we run active-standby: the journal streams continuously to the standby region, where a mirror engine folds the same events into the same state. Failover is a control-plane action that promotes the standby only once its offset matches the last durable offset of the primary. RPO within a region is 0; cross-region RPO is under 5 seconds and cross-region RTO is under 15 minutes.
How do you prevent split-brain during a network partition?: Each settlement partition is owned by exactly one worker via a fenced lease with a monotonic fencing token. TTL is 8s and heartbeats are 2s. Writes to the journal and downstream stores validate the token; a stale token is rejected even if the previous owner still believes it holds the lease. This means the classic partition-plus-GC-pause scenario cannot produce a second committer.
What happens if a downstream rail double-processes despite our idempotency?: Idempotency at our layer only guarantees at most one ledger effect per intent. If a counterparty rail acknowledges twice for the same submission, reconciliation compares the rail's own settlement report against our journal by external reference and quarantines any surplus credit. Corrections are always compensating entries — we never mutate history — and the resulting exception is surfaced in the operations console with a linked audit path.
How is state recovered after a worker or database failure?: State is a pure fold: state = fold(reduce, snapshot, journal[snapshot.offset:]). Snapshots are emitted every 10^6 events. On a worker crash a new owner acquires the lease and replays the journal from the last committed offset — p99 recovery is under 90 seconds on a hot node. On primary DB failover we reconnect, verify the last durable LSN, and resume; RPO is 0 because the journal is the source of truth, not the database.
How do we roll back a bad release without corrupting the ledger?: Reducers are versioned and pure. A bad release can only produce forward events; it cannot rewrite history. Rolling back the binary and replaying from the last known-good offset reproduces state deterministically. If a specific reducer version emitted incorrect postings, we issue compensating entries against those specific intents — the ledger still tells the full truth of what happened, including the correction.

Architecture PDF

Technical whitepaper: data model, journal format, lease protocol, failure semantics, benchmark methodology, threat model. Written for engineering review, not sales — no email gate.

Open the PDF

Book a technical demo

60 minutes with a Hostra staff engineer. We walk your team through the journal, a live failover, and answer whatever you'd ask a vendor before betting production money on them.

Contact engineering