Skip to content

Messaging Patterns: Log-stream vs Queue vs Pub/Sub

Summary

How the five messaging systems in this vault (Kafka, NATS, RabbitMQ, Redpanda and Pulsar) map to the three fundamental messaging patterns, and what delivery guarantees each one gives. Every system can serve more than one pattern, but each has a native primitive that fits one pattern best.

Pattern Definitions

Pattern Semantics Replay? Consumer model
Log-stream Append-only ordered log. Consumers track offsets Yes Pull (consumer controls position)
Queue Competing consumers. Message removed (or marked done) on ack No Push or pull. Each message is processed by one consumer
Pub/Sub Fan-out to all subscribers Depends on the system (none in Core NATS or a non-durable fanout) Push (broker delivers to every subscriber)

System × Pattern Matrix

System Log-stream Queue Pub/Sub Notes
Kafka ✅ Primary ✅ Consumer groups (one consumer per partition). Share groups (KIP-932, production-ready 4.2) add per-record acks and more consumers than partitions ✅ Each consumer group gets every record Everything is a log. Queue and pub/sub are consumption patterns over the log.
NATS ✅ JetStream streams ✅ Queue groups (Core) + JetStream WorkQueue streams ✅ Core NATS (at-most-once, no acks, no storage) Core NATS is pure pub/sub and request-reply. JetStream adds persistence, acks, replay and queue semantics.
RabbitMQ ✅ Streams and super streams (retain messages after reads) ✅ Primary (quorum queues) ✅ Fanout and topic exchanges Queues are the native primitive. Streams add log replay with retention by age or size.
Redpanda ✅ Primary ✅ Consumer groups ✅ Each consumer group gets every record Kafka-API compatible, so the same patterns as Kafka's consumer groups.
Pulsar ✅ Primary ✅ Shared and Key_Shared subscriptions ✅ Multiple subscriptions on one topic, each gets every message Exclusive and Failover subscriptions give ordered streaming. All subscription types work on the same topic.

Delivery Guarantees

System At-most-once At-least-once Exactly-once
Kafka Producer acks=0 or commit before processing Default with acks=all and commit after processing Idempotent producer + transactions for read-process-write inside Kafka
NATS Core NATS JetStream with acks JetStream: Nats-Msg-Id publish dedup (within the dedup window) + double ack on consume
RabbitMQ Auto-ack consumers Publisher confirms + manual consumer acks Not provided by the broker. Use idempotent consumers
Redpanda As Kafka As Kafka Kafka idempotence and transactions API
Pulsar Non-persistent topics Persistent topics with individual or cumulative acks Transactions (read-committed) across topics inside Pulsar

For all five, exactly-once ends at the broker boundary. External sinks still need idempotent writes or a transactional connector.

Choosing a Pattern

Start from what consumers need to do with a message. Each branch lists the native primitive in each system.

flowchart TD
    Start{"What do consumers need?"}
    Start -->|"Replay old events"| LogStream["Log-stream"]
    Start -->|"Distribute work across workers"| Queue["Queue"]
    Start -->|"Notify all subscribers now"| PubSub["Pub/Sub"]
    LogStream --> KafkaRedpanda["Kafka / Redpanda / Pulsar topics"]
    LogStream --> NatsJS["NATS JetStream streams"]
    LogStream --> RabbitStream["RabbitMQ Streams"]
    Queue --> RabbitQQ["RabbitMQ quorum queues"]
    Queue --> NatsQG["NATS queue groups /<br/>JetStream WorkQueue"]
    Queue --> KafkaSG["Kafka share groups (4.2+)<br/>or consumer groups"]
    Queue --> PulsarShared["Pulsar Shared /<br/>Key_Shared subscription"]
    PubSub --> NatsCore["Core NATS<br/>(at-most-once)"]
    PubSub --> RabbitFanout["RabbitMQ fanout exchange"]
    PubSub --> KafkaCGAll["Kafka / Redpanda<br/>(one consumer group per subscriber)"]
    PubSub --> PulsarSubs["Pulsar<br/>(one subscription per subscriber)"]

To pick a system rather than a pattern: if you need several patterns and the largest ecosystem, use Kafka. If you need rich routing, per-message TTL, priorities and dead-lettering, use RabbitMQ. If you need a light, multi-tenant fabric for edge and request-reply, use NATS. The Streaming Brokers Comparison covers Kafka vs Redpanda vs Pulsar in depth.

Trade-off Summary

The ratings are qualitative. No cross-system benchmark is recorded in this vault.

Dimension Log-stream Queue Pub/Sub
Ordering Per partition / per subject FIFO per queue (weakened by redelivery and multiple consumers) Usually per publisher only
Replay Yes (by offset or time) No (consumed = gone) No (fire-and-forget)
Throughput High (batching, sequential I/O) Medium (per-message ack overhead) Highest when non-persistent
Latency Medium (batching + fsync) Low–medium Lowest when non-persistent
Durability Strong (replicated log) Strong (replicated queue) None unless layered on a stream
Use case Event sourcing, CDC, analytics Work distribution, RPC Notifications, real-time updates

Sources