Apache Pulsar How-to Guides¶

Task recipes for running Apache Pulsar in production: deployment, sizing, security setup, geo-replication, tiered storage, metadata-store migration, troubleshooting, and everyday pulsar-admin / pulsar-client commands. Commands target the 4.0 LTS and 4.2 lines unless marked otherwise. Look up defaults and ports in Reference. The reasons behind each step are in Explanation.

Which version to deploy

For production, run the 4.0 LTS line (latest 4.0.13, 2026-08-03). It has active support until 2026-10-21 and security support until 2027-10-21. The 5.0.0 milestones (M1, M2) are previews and not for production. See support windows.

Deployment Patterns¶

Standalone (dev only)¶

Standalone runs the broker, bookie, and metadata store in one JVM. Pin an explicit image tag instead of latest.

docker run -it -p 6650:6650 -p 8080:8080 \
  --name pulsar-standalone \
  apachepulsar/pulsar:4.0.13 \
  bin/pulsar standalone

Production cluster¶

Three tiers: at least 3 brokers, 3 or more bookies (4 to 6 is common), and 3 metadata-store nodes (ZooKeeper, or Oxia for new clusters on 5.0).

Tier	Sizing (starting point, validate with load tests)
Brokers	3 nodes × 4 vCPU, 4 to 8 GB heap plus direct memory
Bookies	3 to 6 nodes × 4 to 8 vCPU, 16 GB or more RAM. NVMe journal and SSD ledger disks
ZooKeeper or Oxia	3 nodes × 2 vCPU + 4 GB. Small SSD
Configuration store	Optional. Only for a shared multi-cluster configuration store
Pulsar Functions worker	2 or 3 nodes if you run Functions outside the brokers

Kubernetes (Helm chart)¶

The Apache pulsar-helm-chart is the reference path. It needs Kubernetes 1.25 or later. StreamNative also publishes operators for its distribution.

helm repo add apachepulsar https://pulsar.apache.org/charts
helm repo update
helm show values apachepulsar/pulsar > defaults.yaml   # inspect options
helm install pulsar apachepulsar/pulsar \
  --namespace pulsar --create-namespace \
  --values prod-values.yaml

Key values.yaml choices:

Separate StatefulSets for brokers and bookies, with distinct storage classes (NVMe for bookie journals, SSD for ZooKeeper or Oxia).
components.oxia: true with components.zookeeper: false deploys Oxia instead of ZooKeeper. With Oxia, Pulsar Functions need FileSystemPackagesStorage enabled, or the chart refuses to render.
On clusters with fewer than 3 nodes, set affinity.anti_affinity: false.

Sizing Guidance¶

Resource	Guidance
Broker JVM heap	4 to 8 GB. Most broker memory is direct memory (entry cache, Netty buffers)
Broker direct memory	Set `-XX:MaxDirectMemorySize` through `PULSAR_MEM` in `conf/pulsar_env.sh`. The default is `-Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g`. A common starting point is direct memory of at least 2× heap
Broker entry cache	`managedLedgerCacheSizeMB`. The default is 1/5 of direct memory
Bookie journal disk	Dedicated NVMe. Journal latency sets publish latency
Bookie ledger disk	SSD sized for the hot working set plus retention not offloaded to tiered storage
Bookie JVM	G1 or ZGC. Give DbLedgerStorage write and read-ahead caches enough direct memory
ZooKeeper heap	2 to 4 GB. Keep snapshots and transaction logs on SSD
Network	10 GbE or more between brokers and bookies. Replication multiplies the write bandwidth by Qw

Best Practices¶

Place brokers and bookies on different hosts. Otherwise they compete for memory, page cache, and CPU.
Use E=3, Qw=3, Qa=2 for general workloads (the shipped default is 2/2/2). Raise Qa to 3 for stricter durability.
Keep journal sync on (journalSyncData=true, the default). Never turn it off in production.
Set namespace retention and backlog quotas before you create topics.
Lock down schemas in production: set-is-allow-auto-update-schema --disable and set-schema-validation-enforce --enable.
Use partitioned topics when one topic's throughput exceeds a single broker. On 5.0, evaluate scalable topics for new applications.
Offload old ledgers to object storage instead of over-provisioning bookie disks.
Run Functions in a separate worker cluster at scale, not inside the broker.
Stay on a supported line and upgrade LTS to LTS (3.0 -> 4.0 -> 5.0). Do not skip LTS lines.
Back up metadata. Take ZooKeeper snapshots, or follow the Oxia backup guidance.

Performance Tuning¶

Match the symptom to the setting. Exact defaults are in Reference.

Goal	Setting
More replay reads served from memory	Raise `managedLedgerCacheSizeMB` (and direct memory)
Smoother Shared-subscription dispatch	`dispatcherMaxRoundRobinBatchSize`, consumer `receiverQueueSize`
Trade durability for latency	`managedLedgerDefaultEnsembleSize` / `WriteQuorum` / `AckQuorum`, or per namespace with `set-persistence`
Lower bookie publish latency	`journalMaxGroupWaitMSec` (default 1 ms) and a faster journal device
Keep topics that are idle for a while	`brokerDeleteInactiveTopicsEnabled=false` for stable applications
Better load distribution	`loadManagerClassName` (Modular or Extensible) and `loadBalancerLoadSheddingStrategy`
Automatic offload	`managedLedgerOffloadThresholdInSeconds` or `managedLedgerOffloadAutoTriggerSizeThresholdBytes`, or namespace offload policies

Configure JWT authentication¶

Create a signing key and tokens, then turn on the token provider on brokers. Brokers also need a client token for internal (broker-to-broker) calls.

bin/pulsar tokens create-secret-key --output /etc/pulsar/jwt/secret.key
bin/pulsar tokens create --secret-key file:///etc/pulsar/jwt/secret.key \
  --subject broker > /etc/pulsar/jwt/broker.token
bin/pulsar tokens create --secret-key file:///etc/pulsar/jwt/secret.key \
  --subject orders-svc --expiry-time 7d

# broker.conf
authenticationEnabled=true
authorizationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
tokenSecretKey=file:///etc/pulsar/jwt/secret.key
superUserRoles=broker,admin
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters=file:///etc/pulsar/jwt/broker.token

For asymmetric keys, use create-key-pair and tokenPublicKey. For an external IdP, use AuthenticationProviderOpenID on the broker and the OAuth 2.0 client-credentials plugin on clients.

Configure TLS (broker)¶

tlsEnabled is deprecated. Setting the TLS ports is what enables TLS.

# broker.conf
brokerServicePortTls=6651
webServicePortTls=8443
tlsCertificateFilePath=/etc/pulsar/certs/broker.cert.pem
tlsKeyFilePath=/etc/pulsar/certs/broker.key-pk8.pem
tlsTrustCertsFilePath=/etc/pulsar/certs/ca.cert.pem
tlsRequireTrustedClientCertOnConnect=true   # mTLS
tlsProtocols=TLSv1.3,TLSv1.2
# broker-to-broker (replication, lookups) over TLS
brokerClientTlsEnabled=true
brokerClientTrustCertsFilePath=/etc/pulsar/certs/ca.cert.pem

Clients then use pulsar+ssl://broker:6651 and https://broker:8443. After clients have moved, disable the plaintext ports by leaving brokerServicePort and webServicePort empty.

Grant Permissions¶

Per-namespace permissions¶

pulsar-admin namespaces grant-permission my-tenant/ns-prod \
  --role orders-svc \
  --actions produce,consume

pulsar-admin namespaces revoke-permission my-tenant/ns-prod \
  --role orders-svc

Per-topic permissions¶

pulsar-admin topics grant-permission persistent://my-tenant/ns-prod/orders \
  --role downstream-svc \
  --actions consume

Tenant administrators¶

pulsar-admin tenants create my-tenant \
  --admin-roles tenant-admin \
  --allowed-clusters pulsar-cluster-1

A tenant-admin role can create namespaces and grant permissions inside its own tenant. Keep superUserRoles in broker.conf to a minimum.

Subscription-auth modes¶

pulsar-admin namespaces set-subscription-auth-mode my-tenant/ns-prod \
  --subscription-auth-mode Prefix

Encrypt Messages End to End¶

Producers encrypt with consumer public keys. Consumers decrypt with the matching private key (Java client).

Producer<byte[]> producer = client.newProducer()
    .topic("persistent://my-tenant/ns-prod/orders")
    .addEncryptionKey("orders-key")
    .defaultCryptoKeyReader("file:///etc/pulsar/keys/orders-public.pem")
    .create();

Consumer<byte[]> consumer = client.newConsumer()
    .topic("persistent://my-tenant/ns-prod/orders")
    .subscriptionName("orders-sub")
    .defaultCryptoKeyReader("file:///etc/pulsar/keys/orders-private.pem")
    .subscribe();

Configure Tiered Storage Offload¶

Set a broker-wide driver in broker.conf, or a policy per namespace:

# broker.conf (broker-wide default)
managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=pulsar-cold
s3ManagedLedgerOffloadRegion=us-east-1
managedLedgerOffloadThresholdInSeconds=86400

pulsar-admin namespaces set-offload-policies my-tenant/ns-prod \
  --driver aws-s3 \
  --bucket pulsar-cold \
  --region us-east-1 \
  --offloadAfterThreshold 10G \
  --offloadAfterElapsed 24h

# or trigger manually for one topic
pulsar-admin topics offload --size-threshold 10M persistent://my-tenant/ns-prod/orders
pulsar-admin topics offload-status persistent://my-tenant/ns-prod/orders

Encryption at rest for offloaded data

Enable default server-side encryption (SSE-S3 or SSE-KMS) on the bucket itself. Earlier versions of this note listed s3ManagedLedgerOffloadServerSideEncryption* broker keys. Those keys do not appear in the 4.2 broker.conf, so do not rely on them.

Enable Transactions¶

# broker.conf on every broker
transactionCoordinatorEnabled=true
systemTopicEnabled=true

# once per cluster, before first use
bin/pulsar initialize-transaction-coordinator-metadata \
  -cs zk1:2181 -c pulsar-cluster-1 --initial-num-transaction-coordinators 16

On clients, build the PulsarClient with .enableTransaction(true), then client.newTransaction().withTransactionTimeout(...). Pass the transaction to newMessage(txn) and acknowledgeAsync(msgId, txn). For high-throughput transactional workloads, enable batched transaction-log writes (settings).

Migrate the Metadata Store from ZooKeeper to Oxia¶

Pulsar 5.0 only

The live migration framework (PIP-454) ships with Pulsar 5.0 and is currently available only in milestone builds. Test it on non-production clusters.

Deploy Oxia. Create the namespaces you need (for example broker and bookkeeper) in the Oxia coordinator config.
Make sure bookies use the Pulsar metadata driver (metadataServiceUri=metadata-store:zk:... in bookkeeper.conf). Plain zk+hierarchical:// bookies do not take part in the migration.
Start the migration and watch it:

bin/pulsar-admin metadata-migration status
bin/pulsar-admin metadata-migration start --target oxia://oxia-1.example.com:6648/broker
bin/pulsar-admin metadata-migration status   # PREPARATION -> COPYING -> COMPLETED

After COMPLETED, set metadataStoreUrl=oxia://... on brokers and the bookie metadata URI to metadata-store:oxia://.../bookkeeper, then do a rolling restart. If the status shows FAILED, the cluster has already reverted to ZooKeeper. Check the broker logs and retry.

Troubleshooting¶

Slow consumer / dispatcher backlog¶

Symptom: pulsar-admin topics stats <topic> shows msgRateOut below msgRateIn and a growing msgBacklog.

Causes: too few consumers on a Shared subscription, or slow downstream processing.

Fix: scale consumers, raise receiverQueueSize, or add partitions. For growth, set a backlog quota so producers get back-pressure instead of disks filling.

Bookie auto-recovery stuck¶

Symptom: under-replicated ledgers do not drain.

Cause: AutoRecovery is disabled, the auditor cannot elect a leader, or there are not enough bookies for the ensemble size.

Fix:

bin/bookkeeper shell autorecovery -status
bin/bookkeeper shell autorecovery -enable
bin/bookkeeper shell listunderreplicated
bin/bookkeeper shell decommissionbookie   # only if a bookie is permanently lost

Metadata store quorum loss¶

Symptom: brokers log MetadataStoreException / ZooKeeper session errors, and topic ownership churns.

Fix: restore the ZooKeeper (or Oxia) quorum first. Brokers then reconcile. Do not restart all brokers at once, because each one re-acquires its bundles through the metadata store.

Redelivered messages (Shared subscription)¶

Symptom: consumers sometimes receive a message again.

Cause: ack timeout or negative acks, a consumer disconnecting with unacked messages, or acknowledgements batched on the client and lost on crash.

Fix: check pulsar-admin topics stats (unackedMessages, msgRedeliver) and tune ackTimeout and acknowledgmentGroupTime. Make consumers idempotent. At-least-once delivery means redelivery is expected.

Geo-replication lag¶

pulsar-admin topics stats persistent://my-tenant/ns-prod/orders
# check replication.<cluster>.replicationBacklog and connected

Lag usually tracks WAN RTT plus remote-cluster broker load. Monitor the replication backlog metrics and raise set-replicator-dispatch-rate if replication is throttled.

Pulsar Functions failing¶

pulsar-admin functions status \
  --tenant my-tenant --namespace ns-prod --name enrichment
pulsar-admin functions stats \
  --tenant my-tenant --namespace ns-prod --name enrichment

Commands & Recipes¶

Cluster bootstrap¶

# Initialize cluster metadata (ZooKeeper example; use oxia://host:6648/broker for Oxia)
bin/pulsar initialize-cluster-metadata \
  --cluster pulsar-cluster-1 \
  --metadata-store zk:zk1:2181,zk2:2181,zk3:2181 \
  --configuration-metadata-store zk:zk1:2181,zk2:2181,zk3:2181 \
  --web-service-url http://broker1:8080 \
  --broker-service-url pulsar://broker1:6650

# Start a bookie
bin/pulsar bookie

# Start a broker
bin/pulsar broker

Tenant + namespace management¶

pulsar-admin tenants create my-tenant --allowed-clusters pulsar-cluster-1
pulsar-admin namespaces create my-tenant/ns-prod \
  --bundles 16 \
  --clusters pulsar-cluster-1

# Retention (size, time)
pulsar-admin namespaces set-retention my-tenant/ns-prod \
  --size 100G --time 30d

# Backlog quota
pulsar-admin namespaces set-backlog-quota my-tenant/ns-prod \
  --limit 50G --policy producer_request_hold

# Schema validation
pulsar-admin namespaces set-schema-validation-enforce --enable my-tenant/ns-prod
pulsar-admin namespaces set-is-allow-auto-update-schema --disable my-tenant/ns-prod

Topic management¶

pulsar-admin topics create-partitioned-topic \
  persistent://my-tenant/ns-prod/orders --partitions 12
pulsar-admin topics list my-tenant/ns-prod
pulsar-admin topics stats persistent://my-tenant/ns-prod/orders
pulsar-admin topics stats-internal persistent://my-tenant/ns-prod/orders

Producing / consuming¶

pulsar-client produce persistent://my-tenant/ns-prod/orders \
  --num-produce 1000 --messages "hello"

pulsar-client consume persistent://my-tenant/ns-prod/orders \
  --subscription-name orders-sub \
  --subscription-type Shared \
  --num-messages 0

Geo-replication¶

# Register the remote cluster on each side (repeat per cluster)
pulsar-admin clusters create eu-west \
  --url http://eu-west-broker:8080 \
  --broker-url pulsar://eu-west-broker:6650

# Allow and enable replication for the namespace
pulsar-admin tenants update my-tenant --allowed-clusters us-east,eu-west,ap-southeast
pulsar-admin namespaces set-clusters my-tenant/ns-prod \
  --clusters us-east,eu-west,ap-southeast

# Throttle replication
pulsar-admin namespaces set-replicator-dispatch-rate my-tenant/ns-prod \
  --msg-dispatch-rate 10000 --byte-dispatch-rate 10485760 --dispatch-rate-period 1

Consumers that must fail over between regions should use replicateSubscriptionState(true) in the Java consumer builder (replicated subscriptions).

Functions¶

pulsar-admin functions create \
  --tenant my-tenant --namespace ns-prod --name enrichment \
  --inputs persistent://my-tenant/ns-prod/orders \
  --output persistent://my-tenant/ns-prod/orders-enriched \
  --jar ./enrichment-1.0.jar \
  --classname com.example.Enrich \
  --parallelism 3

BookKeeper diagnostics¶

bin/bookkeeper shell bookieinfo
bin/bookkeeper shell bookiesanity
bin/bookkeeper shell listbookies -rw

Prometheus¶

Brokers expose Prometheus metrics on http://<broker>:8080/metrics. Bookies and Functions workers also expose /metrics on their HTTP ports. The Helm chart can deploy kube-prometheus-stack with Grafana dashboards. 4.x adds OpenTelemetry metrics, and 4.2 adds OpenTelemetry tracing in the Java client (PIP-446).

Cross-references¶

Explanation — the broker, bookie, and metadata model you operate, plus the security model.
Reference — defaults, ports, support windows, and the hardening checklist.
Messaging domain — cross-broker comparison.