Transactional outbox pattern

The outbox pattern solves the dual-write problem: you need to update your DB and publish an event, but you cannot atomically do both because they are separate systems.

The problem

def create_order(data):
    db.insert("orders", data)        # step 1
    kafka.publish("order.created")    # step 2

If step 2 fails after step 1 succeeds, the order exists but downstream services never hear about it. The customer is charged but no email is sent.

If you swap the steps, step 1 might fail after step 2 succeeded. Now Kafka knows about an order that doesn't exist.

You cannot solve this with a try/catch. The process can die between the steps.

The fix

Write the event into an outbox table in the same transaction as the business data. A separate worker reads the outbox and publishes to Kafka.

BEGIN;
INSERT INTO orders (id, total, ...) VALUES (...);
INSERT INTO outbox (event_type, payload, created_at) VALUES ('order.created', '{...}', now());
COMMIT;

The orders row and the outbox row commit together. They are now atomically consistent. The worker can publish at its own pace.

The worker

Two flavors:

Polling. Worker periodically queries the outbox for unpublished rows, publishes them, marks done. Simple. Adds polling latency (typically 100ms-1s).

Change data capture (Debezium). Debezium reads the Postgres WAL, publishes outbox inserts directly to Kafka. Sub-second latency. More infrastructure.

For most teams: start with polling. Move to CDC if you need lower latency or higher throughput.

At-least-once

The worker might crash after publishing but before marking the outbox row done. The next run republishes. Consumers must be idempotent. Use an event_id in the payload for dedup.

Why not just CDC the orders table?

Debezium can stream the orders table directly to Kafka. Why use outbox?

• You control the event shape. Outbox lets you publish a clean event ("order.created with customer + total"), not the raw row diff.
• You control which events fire. Some DB writes don't need an event.
• You can include derived data. The outbox row can have joined data from other tables.

For internal CDC where consumers want raw rows, just use Debezium directly. For domain events, use outbox.

My rule

Any business operation that needs to atomically update DB and emit an event uses the outbox. Cost: one extra insert per event, plus a worker process. Saves: an entire class of dual-write bugs that are unreproducible until they hit production.