Skip to content

Redpanda How-to Guides

What this page covers

Tasks for running Redpanda (self-managed "Redpanda Streaming", 26.x): install, form a cluster, deploy on Kubernetes, secure it, enable Tiered Storage, Iceberg Topics and data transforms, manage the Enterprise license, do rolling upgrades, troubleshoot, plus a Commands & Recipes section. Sizing, tuning tables, property defaults and the license matrix are in Reference. Internals are in Explanation.

Version and license caveats

Commands were checked against the rpk source on the dev branch (2026-09). Since 24.1, ACL and user commands live under rpk security. Tiered Storage, Cloud Topics, Iceberg Topics, Shadowing, RBAC and OIDC need an Enterprise license (a 30-day trial is built in). See the license matrix.

Deployment Patterns

Single-binary install

For development or small production on Debian/Ubuntu. The repository setup URL comes from the upstream README:

curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.deb.sh' | sudo -E bash
sudo apt-get install redpanda
sudo rpk redpanda mode production     # production defaults and tuners on
sudo rpk redpanda tune all            # CPU governor, IRQ affinity, I/O scheduler, and so on
sudo rpk iotune                       # measure the disk for the Seastar I/O scheduler
sudo systemctl enable --now redpanda

On RHEL, Fedora or Amazon Linux use setup-redpanda.rpm.sh and yum install redpanda. On macOS, brew install redpanda-data/tap/redpanda && rpk container start runs Redpanda in Docker.

Use three brokers across three AZs with NVMe local storage. Node settings go in /etc/redpanda/redpanda.yaml. Cluster settings (SASL, Tiered Storage, and so on) are stored in the controller and set with rpk cluster config. They are ignored in redpanda.yaml after bootstrap.

# /etc/redpanda/redpanda.yaml on redpanda-0 (node config only)
redpanda:
  data_directory: /var/lib/redpanda/data
  empty_seed_starts_cluster: false   # recommended for production
  seed_servers:
    - host: { address: redpanda-0, port: 33145 }
    - host: { address: redpanda-1, port: 33145 }
    - host: { address: redpanda-2, port: 33145 }
  rpc_server: { address: 0.0.0.0, port: 33145 }
  advertised_rpc_api: { address: redpanda-0, port: 33145 }
  kafka_api:
    - address: 0.0.0.0
      port: 9092
  advertised_kafka_api:
    - address: redpanda-0
      port: 9092
  admin:
    - address: 0.0.0.0
      port: 9644
  rack: us-east-1a                   # enables rack-aware replica placement

node_id can be omitted. Brokers get IDs automatically. After the cluster forms, apply cluster-wide settings:

rpk cluster config set cloud_storage_enabled true          # Enterprise; needs restart
rpk cluster config set cloud_storage_bucket redpanda-tiered-prod
rpk cluster config set cloud_storage_region us-east-1
rpk cluster config set cloud_storage_credentials_source aws_instance_metadata
rpk cluster config status                                   # shows brokers needing restart

Kubernetes (Operator + Helm)

The redpanda Helm chart now lives in the redpanda-operator repo, and chart versions track Redpanda versions (chart 26.2.x deploys Redpanda v26.2.x):

helm repo add redpanda https://charts.redpanda.com
helm repo update
helm install redpanda redpanda/redpanda \
  --namespace redpanda --create-namespace \
  --set tls.enabled=true \
  --set storage.persistentVolume.size=200Gi \
  --set storage.persistentVolume.storageClass=ssd \
  --set resources.cpu.cores=4 \
  --set statefulset.replicas=3

For declarative management install the Redpanda Operator (redpanda/operator chart). It adds Redpanda, Topic, User and Schema CRDs, NodePool (26.1, for blue/green node-pool migrations), stretch clusters, and Pipeline (26.2, runs Redpanda Connect pipelines). Pass an Enterprise license with enterprise.licenseSecretRef in the chart values.

Best Practices

  • Run rpk redpanda mode production and the tuners on bare metal or VMs.
  • Give Redpanda whole cores. One core is one shard. Increasing cores is always possible. Decreasing is supported from 24.3.
  • Disable swap. Seastar pre-allocates memory per core.
  • Use rpk cluster config edit/set for cluster properties rather than editing YAML.
  • Set empty_seed_starts_cluster: false in production so a mis-configured broker cannot start a second cluster.
  • Choose the storage mode per topic. Use local for the lowest latency, tiered for long retention, and cloud (Cloud Topics, 26.1+) for latency-tolerant, high-volume streams where cross-AZ traffic dominates cost.
  • Stay under topic_partitions_per_shard (default 5000 partition replicas per core).
  • Use cert-manager with the Operator for TLS rather than hand-rolled certificates.
  • Enable Continuous Data Balancing (Enterprise, partition_autobalancing_mode=continuous) for clusters with skewed disk usage or frequent broker loss.
  • Stay within a supported feature release (about 12 months each) and upgrade one feature release at a time.

Security Setup

Enable SASL/SCRAM and create users

# Create the first superuser before turning on SASL
rpk security user create admin -p '<strong-password>' --mechanism scram-sha-512
rpk cluster config set superusers '["admin"]'
rpk cluster config set enable_sasl true
# rpk profile / env vars: RPK_USER, RPK_PASS, RPK_SASL_MECHANISM
rpk security user create app -p '<app-password>' --mechanism scram-sha-512
rpk security user list

TLS on the Kafka listener

The listener TLS block is node config (redpanda.yaml):

redpanda:
  kafka_api:
    - name: external
      address: 0.0.0.0
      port: 9092
      authentication_method: sasl
  kafka_api_tls:
    - name: external
      enabled: true
      key_file: /etc/redpanda/certs/server.key
      cert_file: /etc/redpanda/certs/server.crt
      truststore_file: /etc/redpanda/certs/ca.crt
      require_client_auth: true      # mTLS

Repeat for admin_api_tls, rpc_server_tls, schema_registry_api_tls and pandaproxy_api_tls. Force TLS 1.3 with rpk cluster config set tls_min_version v1.3 (needs restart).

Grant ACLs

# Prefixed pattern replaces the 'orders.*' wildcard idea
rpk security acl create --allow-principal 'User:orders-svc' \
  --operation read,describe \
  --topic orders. --resource-pattern-type prefixed

rpk security acl create --allow-principal 'User:orders-svc' \
  --operation read --group orders-consumer

rpk security acl list

Roles (RBAC, Enterprise)

rpk security role create orders-readers
rpk security acl create --allow-role orders-readers --operation read,describe --topic orders
rpk security role assign orders-readers --principal orders-svc

Enable Tiered Storage on a Topic

Tiered Storage needs cloud_storage_enabled=true and the bucket settings from Three-node cluster:

rpk topic create orders -p 12 -r 3 \
  -c redpanda.remote.write=true \
  -c redpanda.remote.read=true \
  -c retention.ms=2592000000 \
  -c retention.local.target.ms=86400000     # keep ~1 day on local disk
rpk topic describe orders

To make it the default for new topics, set cloud_storage_enable_remote_write and cloud_storage_enable_remote_read to true, or on newer releases use default_redpanda_storage_mode=tiered.

Create an Iceberg Topic

rpk cluster config set iceberg_enabled true            # Enterprise; restart required
rpk cluster config set iceberg_catalog_type rest       # or object_storage (default)
# point iceberg_rest_catalog_endpoint (and auth settings) at Glue / Unity / Polaris etc.
rpk topic create clicks -p 6 -r 3 -c redpanda.iceberg.mode=value_schema_id_prefix
rpk registry schema create clicks-value --schema ./clicks.avsc

Producers must use the Schema Registry wire format for value_schema_id_prefix. Use key_value mode when records have no schema.

Deploy a Data Transform

rpk cluster config set data_transforms_enabled true    # restart required
rpk transform init --language=tinygo-no-goroutines redact
cd redact && rpk transform build
rpk transform deploy --input-topic=raw-events --output-topic=clean-events
rpk transform list
rpk transform logs redact

Manage the Enterprise License

rpk cluster license info                               # status, expiry, features in use
rpk cluster license set --path ./redpanda.license      # apply a purchased key
rpk generate license --apply                           # request a 30-day trial extension

Rolling Upgrade

Put one broker at a time into maintenance mode. This drains Raft leadership. Then upgrade and restart it:

rpk cluster health                         # must report Healthy: true
rpk cluster maintenance enable <broker-id> --wait
rpk cluster maintenance status
sudo apt-get install --only-upgrade redpanda && sudo systemctl restart redpanda
rpk cluster maintenance disable <broker-id>
rpk cluster health --watch --exit-when-healthy

On Kubernetes, bump the chart or Redpanda resource version. The Operator and chart perform the same maintenance-mode roll.

Troubleshooting

Raft leadership thrash

Symptom: leaders change rapidly. Clients log NOT_LEADER_OR_FOLLOWER or LEADER_NOT_AVAILABLE retries.

Causes: slow disks (fsync latency), oversubscribed CPU, network latency or packet loss.

Fixes:

  • Watch the public metric redpanda_raft_leadership_changes (per topic).
  • Check per-disk write latency with iostat -xz 1, and re-run rpk iotune after hardware changes.
  • Collect rpk debug bundle for Redpanda support.

Slow archival uploads

Symptom: local disk keeps growing on tiered topics and uploads lag behind.

Causes: object-store throttling, too few connections, slow network.

Fixes:

  • Raise cloud_storage_max_connections (default 20 per shard).
  • Check bucket request-rate limits and throttling responses (for example, S3 503 SlowDown).
  • Inspect the cloud-storage metrics on :9644/public_metrics. Metric names vary by version, so check the metrics reference for yours.

Broker degraded state

Symptom: rpk cluster health reports an unhealthy broker or leaderless partitions.

Causes: disk full, OOM, network partition, certificate expiry.

Fixes:

  • rpk cluster info and rpk cluster health for broker state.
  • Check TLS certificate expiry if mTLS is enabled.
  • journalctl -u redpanda for broker logs.

Controller leader overloaded

Symptom: topic creates and config changes are slow, and the controller leader's broker is also busy with partition leadership.

Fix: move controller leadership to a quieter broker. rpk accepts internal namespaces with a {namespace}/ prefix. The controller NTP is redpanda/controller/0, so check this on your version before relying on it:

rpk cluster partitions transfer-leadership --partition redpanda/controller/0:<target-broker-id>

Kafka client compatibility issue

Symptom: a specific Kafka client fails on a feature.

Fix: Redpanda implements most KIPs but can lag on the newest ones. Check the Kafka client compatibility docs and pin the client's API versions if needed.

Commands & Recipes

Local cluster

rpk container start -n 3       # 3-broker Docker cluster plus an rpk profile
rpk cluster info
rpk cluster health
rpk container purge            # tear it down

Topic management

rpk topic create orders --partitions 12 --replicas 3 \
  --topic-config retention.ms=604800000 \
  --topic-config cleanup.policy=delete
rpk topic list
rpk topic describe orders
rpk topic produce orders < data.txt
rpk topic consume orders -o start -n 5
rpk topic delete orders

The replication factor must be odd because of Raft majorities.

Consumer groups

rpk group list
rpk group describe orders-consumer
rpk group seek orders-consumer --to start --topics orders

Cluster config

rpk cluster config get log_segment_size
rpk cluster config edit
rpk cluster config set cloud_storage_max_connections 50
rpk cluster config status

Schema Registry

rpk registry subject list
rpk registry schema create orders-value --schema ./schema.avsc --type avro
rpk registry compatibility-level set orders-value --level FORWARD_TRANSITIVE

Redpanda Connect (Benthos-based)

rpk connect install            # downloads the Connect plugin
rpk connect run ./pipeline.yaml
rpk connect lint ./pipeline.yaml

Benchmark

rpk benchmark --topic rpk-benchmark-topic --partitions 18 --replicas 3 --duration 60

Debug bundle

rpk debug bundle -o /tmp/bundle.zip

Prometheus and Grafana

Redpanda exposes :9644/public_metrics (recommended) and :9644/metrics (internal, verbose). Generate dashboards with rpk generate grafana-dashboard --dashboard operations (26.2 adds operations-stretch and load-factor), or use redpanda-data/observability.

Helm upgrade

helm repo update
helm upgrade redpanda redpanda/redpanda \
  --namespace redpanda \
  --reuse-values \
  --version 26.2.1

Cross-references