Pub-sub patterns

Pub-sub is the pattern where producers publish messages to a topic and any number of consumers subscribe to that topic. It is the opposite of a queue, where each message is consumed exactly once.

The mental model

• Topic = broadcast channel. Every subscriber gets a copy.
• Queue = work distribution. One consumer processes each message.
• Consumer group (Kafka, Pub/Sub). Hybrid. Within a group it acts like a queue. Across groups it acts like pub-sub.

When to use pub-sub

• Fan-out. One event needs multiple reactions. Order created -> email, analytics, inventory.
• Event-driven architecture. Services emit events; other services react. No direct calls.
• Audit and replay. Stream of events is the source of truth. New consumers can backfill.

When to use a queue instead

• Work distribution. N items to process, M workers. Each item processed once.
• Tasks with no logical "subscribers." Send these 1000 emails. Resize these 500 images.

The three decouplings

1. Space. Publisher doesn't know who is listening. Add a new subscriber without changing the publisher.
2. Time. Subscriber can be down when message is published; message waits.
3. Identity. Publisher doesn't authenticate to every subscriber individually.

This is why event-driven architectures scale organizationally: teams ship subscribers without coordinating with publishers.

Push vs pull

• Push (Webhooks, Pub/Sub push subscriptions). Broker delivers to subscriber. Lower latency. Subscriber must be available.
• Pull (Kafka, Pub/Sub pull, SQS). Subscriber polls. Higher latency. Subscriber controls pace.

Pull is more common in production. Subscribers can autoscale, handle their own backpressure, and survive broker issues.

Schemas

Pub-sub couples publisher and subscriber through the message format. Without a schema registry, a publisher change silently breaks every subscriber.

Use Protobuf or Avro with a schema registry (Confluent's is the standard). Enforce backward compatibility: new fields are optional, never remove fields, never change types.

Tools

• Kafka. Ordered per partition, retains messages for days, consumer groups for fan-out + work distribution. Industry default for high-throughput event streaming.
• Google Pub/Sub, AWS SNS+SQS. Managed. Easier ops. Less throughput than Kafka.
• NATS, Redis Streams. Lightweight. Use for low-latency in-cluster messaging.

My defaults

Use Kafka if you need replay, multiple consumer groups, or >100k msg/sec. Use Pub/Sub or SNS+SQS otherwise. Always define a schema. Always have a DLQ. Treat consumers as untrusted - they will lag, crash, and get deployed at the wrong time.