Skip to content

Apache Pulsar

Summary

Apache Pulsar is a cloud-native messaging and streaming platform. It separates compute and storage: stateless brokers serve traffic, and data lives in Apache BookKeeper. It has multi-tenancy (tenants, namespaces, topics), built-in geo-replication, tiered storage, and transactions. The production line is the 4.0 LTS (4.0.13, 2026-08-03). The newest feature release is 4.2.4 (2026-08-03). 5.0 is in milestone preview (5.0.0-M2, 2026-09) and brings Scalable Topics, a new V5 client API, and Oxia as the recommended metadata store in place of ZooKeeper.

Overview

Pulsar is a distributed pub-sub and streaming platform with a different architecture from Kafka. Brokers are stateless, and storage lives in Apache BookKeeper bookies. Compute and storage scale independently. A failed broker's topics move to another broker without copying data, and the same brokers serve hot reads, catch-up reads, and reads tiered to object storage.

Yahoo open-sourced Pulsar in 2016 and donated it to the Apache Software Foundation. It became a top-level ASF project in 2018. It is developed by many vendors, including StreamNative (founded by Pulsar's original creators), DataStax (now IBM), Tencent, and others.

Key Facts

Attribute Detail
Website pulsar.apache.org
GitHub apache/pulsar
Stars ~14k+
Latest Version 4.2.4 (2026-08-03), newest feature line
Current LTS 4.0.13 (2026-08-03). Active support to 2026-10-21, security support to 2027-10-21
Next LTS 5.0.0-M2 milestone (tagged 2026-09-12). GA "expected later in 2026". Not for production
Release model LTS.feature.patch: LTS every 18 months, feature release every ~3 months
Language Java (broker needs Java 21 since 3.3). Clients in Java, C++, Python, Go, Node.js, C#, and more
License Apache-2.0
Steward Apache Software Foundation. Commercial support from StreamNative, DataStax (IBM), and others
Metadata store ZooKeeper (default through 4.x). Oxia is supported since 3.3 and recommended from 5.0. etcd is removed in 5.0
Wire protocols Native Pulsar binary protocol. Protocol-handler plugins: MoP (MQTT), AoP (AMQP 0-9-1). KoP (Kafka) is archived

Full version, support, port, and configuration tables are in Reference.

Evaluation

Pros Cons
Compute/storage separation. No partition rebalancing when brokers fail More moving parts: brokers, BookKeeper, and a metadata store
Native multi-tenancy (tenants, namespaces, topics) with quotas and policies Heavier to operate than NATS or Redpanda
Built-in, per-namespace asynchronous geo-replication Geo-replication gives per-producer ordering only. Active-active needs careful design
Tiered storage to S3 / GCS / Azure Blob JVM tuning (heap plus direct memory, GC)
Four subscription types serve queueing and streaming on one topic Smaller connector and tooling ecosystem than Kafka
Built-in schema registry and transactions (read-committed) Schema registry is not Confluent-API compatible. KoP is archived
Oxia and scalable topics (5.0) address metadata scale and partition sizing 5.0 changes are still in milestone preview
BookKeeper quorum writes (E/Qw/Qa) give strong durability Short feature-line support (6 months) pushes production onto LTS lines

Architecture

This compact view shows the three tiers. The component-level diagrams, the write path, and the transaction flow are in Explanation.

flowchart TB
    Client["Producer / Consumer<br/>(Pulsar client)"]
    subgraph BrokerLayer["Pulsar brokers (stateless)"]
        Broker1["Broker 1"]
        Broker2["Broker 2"]
        Broker3["Broker 3"]
    end
    subgraph BookKeeper["Apache BookKeeper (storage)"]
        Bookie1["Bookie 1"]
        Bookie2["Bookie 2"]
        Bookie3["Bookie 3"]
    end
    subgraph Metadata["Metadata layer"]
        MS["ZooKeeper or Oxia<br/>(local metadata store)"]
        ConfigStore["Configuration store<br/>(optional, multi-cluster)"]
    end
    subgraph TieredStorage["Tiered storage"]
        S3["S3 / GCS / Azure Blob"]
    end
    Client --> Broker1
    Client --> Broker2
    Client --> Broker3
    Broker1 -->|"ledger writes Qw of E"| Bookie1
    Broker1 --> Bookie2
    Broker2 --> Bookie2
    Broker2 --> Bookie3
    Broker3 --> Bookie1
    Broker3 --> Bookie3
    BrokerLayer -.->|"ownership, ledger metadata"| MS
    BookKeeper -.->|"bookie registration"| MS
    BrokerLayer -.-> ConfigStore
    BrokerLayer -->|"offload closed ledgers"| S3

Use Cases

  • Multi-tenant SaaS messaging. Tenants map to namespaces and topics, with per-namespace quotas, retention, and auth.
  • Geo-distributed streaming. Built-in cross-region replication, with no external MirrorMaker-style tool.
  • Queue and stream on one platform. Shared and Key_Shared subscriptions give work queues. Exclusive and Failover give ordered streams.
  • Independent compute/storage scaling. High-fanout consumers do not strain storage, and long retention does not strain brokers.
  • Long retention at low cost. Closed ledgers are offloaded to object storage.
  • Lightweight stream processing with Pulsar Functions (ETL, enrichment, routing).
  • IoT ingest with the MQTT-on-Pulsar (MoP) plugin.

Licensing & Pricing

  • Apache Pulsar: Apache-2.0, free for any use.
  • Oxia: Apache-2.0, a CNCF Sandbox project.
  • StreamNative Cloud: managed Pulsar and Kafka-compatible service (Serverless, Dedicated, and BYOC). StreamNative's Ursa engine is proprietary. See Explanation.
  • DataStax Astra Streaming: managed Pulsar. DataStax was acquired by IBM in 2025. The long-term roadmap is an open question.
  • Tencent Cloud TDMQ for Pulsar: managed Pulsar on Tencent Cloud.

Ecosystem

  • Pulsar Functions: lightweight stream-processing runtime, in the broker or on separate workers.
  • Pulsar IO connectors: sources and sinks (JDBC, Elasticsearch, Kafka, and more). They move to a separate repository in 5.0 (PIP-465).
  • Schema registry: built in. Avro, JSON, Protobuf, and KeyValue.
  • Transactions: atomic produce and ack across topics, with read-committed isolation.
  • Protocol handlers: MoP (MQTT) and AoP (AMQP). KoP (Kafka) is archived by StreamNative.
  • Pulsar SQL: removed from the core in December 2023 (last shipped in 3.0.x). The code now lives in apache/pulsar-sql.
  • Pulsar Manager: web admin UI (apache/pulsar-manager).
  • Helm chart: apachepulsar/pulsar (Kubernetes 1.25+), with an option to deploy Oxia.
  • Clients: Java (reference, plus the new V5 API in 5.0), C++, Python, Go, Node.js, C#/.NET, and community clients.

Compatibility & Requirements

Requirement Detail
Java (4.1+ brokers) Java 21 for broker and Functions. Java 17 or 21 for CLI and Java client
Java (4.0 LTS) Broker Java 21. Java client 8, 11, 17, or 21
Bookies SSD or NVMe. Separate journal and ledger devices recommended
Metadata store ZooKeeper (bundled 3.9.x) or Oxia. RocksDB or memory only for standalone
Configuration store Optional. Needed only for a shared multi-cluster configuration
Tiered storage S3 (and S3-compatible), GCS, Azure Blob, filesystem
Network 6650 binary, 8080 HTTP. TLS on 6651 and 8443 by convention. Bookie 3181
Container apachepulsar/pulsar:<version> (Alpine-based, Java 21 since 4.0)

Latest Versions

  • 5.0.x (milestones): 5.0.0-M1 (2026-06-23) and M2 (2026-09). Scalable Topics, V5 client, Oxia recommended, ZooKeeper-to-Oxia live migration, etcd removed, jakarta.* APIs, Gradle build, structured logging. Preview only.
  • 4.2.x: 4.2.4 (2026-08-03). Topic-list memory limits, Java-client OpenTelemetry tracing, Jetty 12, BookKeeper 4.17.3. It is the newest feature line, but its computed 6-month support window ended 2026-09-24.
  • 4.1.x: 4.1.3 (2026-02-19). Out of support since 2026-03.
  • 4.0.x (LTS): 4.0.13 (2026-08-03). Key_Shared rework (PIP-379), Java 21 Alpine images, rate-limiting and QoS improvements. Recommended for production.
  • 3.0.x (LTS): 3.0.17 (2026-04-23). Security support ended 2026-05-02. Upgrade to 4.0.

Track new releases at pulsar.apache.org/release-notes. The full matrix is in Reference.

Recent Developments

  • Oxia becomes the recommended metadata store (5.0). ZooKeeper stays supported. PIP-454 adds a zero-downtime ZooKeeper-to-Oxia migration (announcement).
  • Scalable Topics (PIP-460). Segments split and merge automatically while keeping per-key order. They need the V5 client.
  • etcd backend removed (PIP-462) in 5.0. Migrate to ZooKeeper or Oxia first.
  • KoP archived. StreamNative now directs Kafka users to its commercial Kafka-compatible service and to Ursa, a leaderless, lakehouse-native engine (VLDB 2025 Best Industry Paper).
  • IO connectors leave the core repo (PIP-465) and get their own release cadence.

Alternatives

  • Apache Kafka: single-tier compute and storage (KRaft). Largest ecosystem, and coupled scaling.
  • Redpanda: Kafka API in C++ with per-partition Raft. Single binary.
  • NATS: lighter and lower latency, with JetStream persistence. Isolation through accounts rather than tenants and namespaces.
  • RabbitMQ: AMQP routing primitives, plus Streams for log-style replay.
  • AWS Kinesis / Azure Event Hubs / GCP Pub/Sub: managed cloud-native equivalents.

Migration & Lock-in

  • Kafka clients: KoP is archived, so Kafka-protocol compatibility on open-source Pulsar is no longer maintained upstream. Plan on Pulsar clients, or on a vendor's Kafka-compatible service.
  • Tiered-storage offload format is Pulsar-specific (ledger blocks plus index objects written by the offloader). Other systems cannot read it directly.
  • Pulsar Functions are Pulsar-specific. Rewriting them for Flink or Kafka Streams is non-trivial.
  • Schema registry is Pulsar-native, not Confluent-API compatible. PIP-420 (4.1) allows third-party registries on the client side.
  • Subscription semantics: Key_Shared and individual acks with redelivery have no exact equivalent in Kafka consumer groups.
  • 5.0 V5 client: scalable topics need the new API, so plan client migration alongside the 5.0 upgrade.

Community Health

  • ASF top-level project with contributors from many vendors (StreamNative, Tencent, DataStax/IBM, and others).
  • Regular cadence: 4.0.x and 4.2.x patch releases shipped roughly monthly through 2026, plus 5.0 milestones.
  • Documented release policy with LTS lines, and a public security page.
  • Production users on the project's "Powered by" list include Yahoo! JAPAN, Tencent, Splunk and Verizon Media, among about 70 organisations (data/powered-by.ts, checked 2026-09-28). The list is self-reported and not dated per entry.

Topic Map

  • How-to Guides: deployment, sizing, security setup, geo-replication, tiered storage, metadata-store migration, troubleshooting, everyday pulsar-admin recipes.
  • Reference: release lines and support windows, Java requirements, ports, metadata store URLs, key broker.conf and bookkeeper.conf settings, subscription types.
  • Explanation: brokers, BookKeeper and the metadata store, the write path, transactions, protocol handlers and StreamNative Ursa.

Sources

Questions

  • When does 5.0.0 reach GA, and will there be a 4.3 feature release before it? The 4.2 support window formally ended 2026-09-24.
  • How does Oxia behave operationally in production (failure modes, backup and restore, sizing) compared with ZooKeeper, for existing clusters that migrate with PIP-454?
  • For multi-region active-active, is Pulsar's geo-replication operationally simpler than Kafka MirrorMaker 2 or Cluster Linking, and at what RPO?
  • What is the practical BookKeeper sizing rule for sustained 1 GB/s ingest with 30-day retention?
  • For tiered storage, what is the time-to-first-byte for reading offloaded data compared with hot storage?
  • After IBM's DataStax acquisition, what is the long-term roadmap for Astra Streaming?
  • Will StreamNative's Ursa components be open-sourced as announced, and will any of them land in Apache Pulsar?