Skip to content

How-to Guides

Task recipes for deploying, configuring, securing, upgrading and troubleshooting Apache SkyWalking 11.x with BanyanDB and Horizon UI. Version matrices, ports and config keys are in Reference; background is in Explanation.

Version pairing

OAP 11.0.0 needs BanyanDB 0.11.x and a horizon-* UI image. OAP refuses to start against any other BanyanDB API version. Move oap.image.tag and banyandb.image.tag together in every upgrade.

Choose a Storage Backend

Use this decision flow before the first install. Changing backends later means starting with empty history.

flowchart TD
    Q1{"New install on OAP 11?"} -->|Yes| Q2{"Need GraalVM native OAP<br/>or lowest memory?"}
    Q1 -->|"No, existing ES cluster"| ES["Elasticsearch / OpenSearch<br/>(SW_STORAGE=elasticsearch)"]
    Q2 -->|Yes| BDB["BanyanDB 0.11.x<br/>(only option for GraalVM Distro)"]
    Q2 -->|No| Q3{"Team already runs<br/>Elasticsearch at scale?"}
    Q3 -->|Yes| ES
    Q3 -->|No| Q4{"Small, low trace/log<br/>sampling, SQL skills only?"}
    Q4 -->|Yes| SQL["PostgreSQL / MySQL<br/>(medium scale only)"]
    Q4 -->|No| BDB

Deploy on Kubernetes with Helm

Chart 5.0.0 is published only as an OCI artifact (Helm 3.8+). No registry login is needed for public pulls. oap.image.tag, oap.storageType and ui.image.tag have no defaults.

helm install skywalking oci://docker.io/apache/skywalking-helm \
  --version 5.0.0 -n skywalking --create-namespace \
  --set oap.image.tag=11.0.0 \
  --set oap.storageType=banyandb \
  --set ui.image.tag=horizon-1.0.0 \
  --set elasticsearch.enabled=false \
  --set banyandb.enabled=true \
  --set banyandb.image.tag=0.11.0

With Elasticsearch instead (the chart's default backend is ECK-managed Elasticsearch 8.18.8, which needs the ECK CRDs first; see the chart's Quick Start):

helm install skywalking oci://docker.io/apache/skywalking-helm \
  --version 5.0.0 -n skywalking --create-namespace \
  --set oap.image.tag=11.0.0 \
  --set oap.storageType=elasticsearch \
  --set ui.image.tag=horizon-1.0.0

Then configure a login (next section). A fresh install has none.

OAP init job

The chart runs an OAP init Job (OAP in init mode) that installs the storage schema, while the serving OAP pods start in no-init mode. Use --wait --wait-for-jobs so Helm reports an init failure directly.

Configure Horizon UI Logins

Horizon has no default credentials and still reports Ready when no users exist. Check the state:

kubectl port-forward -n skywalking svc/skywalking-skywalking-helm-ui 8080:80
curl -s http://127.0.0.1:8080/api/auth/health
# {"backend":"local","configured":false, ...} means no users yet

Provide users through HORIZON_AUTH_LOCAL_USERS, a JSON array of {username, passwordHash, roles} entries with argon2id hashes. Keep the value in a Kubernetes Secret and reference it from ui.envFromSecret or ui.extraEnv, as in the chart's docs/ui/logins.md. The chart's demo admin/admin hashes are public, so use them only on a trusted network. LDAP is also supported through horizon.yaml.

Run a Local Quickstart with Docker Compose

The OAP repo ships a compose file with banyandb and elasticsearch profiles. It defaults to development images, so point it at released images:

git clone --depth 1 --branch v11.0.0 https://github.com/apache/skywalking.git
cd skywalking/docker
export OAP_IMAGE=apache/skywalking-oap-server:11.0.0
export UI_IMAGE=apache/skywalking-ui:horizon-1.0.0
export BANYANDB_IMAGE=apache/skywalking-banyandb:0.11.0
docker compose --profile banyandb up -d
# UI: http://localhost:8080 (container port 8081), OAP gRPC 11800, HTTP 12800, admin 17128

The compose file mounts ./horizon.yaml, which holds the OAP URLs and demo logins. Edit it before sharing the host.

Run BanyanDB Standalone

docker run -d --name banyandb -p 17912:17912 -p 17913:17913 \
  apache/skywalking-banyandb:0.11.0 standalone

# Point OAP at it
export SW_STORAGE=banyandb
export SW_STORAGE_BANYANDB_TARGETS=banyandb:17912

BanyanDB flags map to BYDB_-prefixed environment variables (for example --grpc-port becomes BYDB_GRPC_PORT, default 17912; --http-port becomes BYDB_HTTP_PORT, default 17913). The embedded web UI is at http://<host>:17913/.

Tune BanyanDB Retention and Tiering

With BanyanDB, SW_CORE_RECORD_DATA_TTL and SW_CORE_METRICS_DATA_TTL are ignored. Retention comes from the groups in bydb.yml:

# Helm values: oap.env
oap:
  env:
    SW_STORAGE_BANYANDB_TRACE_TTL_DAYS: "5"
    SW_STORAGE_BANYANDB_LOG_TTL_DAYS: "5"
    SW_STORAGE_BANYANDB_METRICS_MINUTE_TTL_DAYS: "7"
    # Enable a warm tier for minute metrics (needs data nodes labelled type=warm)
    SW_STORAGE_BANYANDB_METRICS_MINUTE_ENABLE_WARM_STAGE: "true"
    SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_TTL_DAYS: "15"

Warm/cold stages need BanyanDB data nodes that match each stage's nodeSelector (type=warm, type=cold). Configure those node pools in the skywalking-banyandb-helm subchart values.

Enable Trace Tail Sampling in BanyanDB

OAP 11 with BanyanDB 0.11 can drop healthy traces during merges while keeping errors and slow traces:

oap:
  env:
    SW_STORAGE_BANYANDB_TRACE_PIPELINE_ENABLED: "true"
    SW_STORAGE_BANYANDB_TRACE_SAMPLER_DURATION_THRESHOLD_MS: "500"
    SW_STORAGE_BANYANDB_TRACE_SAMPLER_HEALTHY_SAMPLE_RATE: "0.1"

The sampler plugin (sw-trace-sampler.so) must be present on the BanyanDB nodes. See the upstream "BanyanDB trace tail sampling" guide before enabling it in production.

Secure Agent-to-OAP Traffic

Enable gRPC TLS on OAP

# application.yml (or the matching SW_CORE_GRPC_SSL_* env vars)
core:
  default:
    gRPCSslEnabled: ${SW_CORE_GRPC_SSL_ENABLED:true}
    gRPCSslKeyPath: ${SW_CORE_GRPC_SSL_KEY_PATH:"/skywalking/certs/server.key"}
    gRPCSslCertChainPath: ${SW_CORE_GRPC_SSL_CERT_CHAIN_PATH:"/skywalking/certs/server.crt"}
    # Set to require client certificates (mTLS)
    gRPCSslTrustedCAPath: ${SW_CORE_GRPC_SSL_TRUSTED_CA_PATH:"/skywalking/certs/ca.crt"}

To keep agent traffic off the cluster port, enable the receiver sharing server (SW_RECEIVER_GRPC_PORT, TLS via SW_RECEIVER_GRPC_SSL_*, CA key gRPCSslTrustedCAsPath). Certificates are hot-reloaded.

Add an Agent Token

# OAP: token checked on the receiver sharing server
export SW_AUTHENTICATION="$(cat /run/secrets/sw-agent-token)"

# Java agent: agent.config (or SW_AGENT_AUTHENTICATION)
agent.authentication=${SW_AGENT_AUTHENTICATION:}
agent.force_tls=${SW_AGENT_FORCE_TLS:false}
agent.ssl_trusted_ca_path=${SW_AGENT_SSL_TRUSTED_CA_PATH:/ca/ca.crt}
# mTLS is enabled when both of these point at files
agent.ssl_key_path=${SW_AGENT_SSL_KEY_PATH:}
agent.ssl_cert_chain_path=${SW_AGENT_SSL_CERT_CHAIN_PATH:}

Store the token in a Kubernetes Secret or Vault and inject it as an environment variable. Never commit it.

Enable TLS on OAP HTTP Servers (11.0+)

export SW_CORE_REST_SSL_ENABLED=true
export SW_CORE_REST_SSL_KEY_PATH=/skywalking/certs/server.key
export SW_CORE_REST_SSL_CERT_CHAIN_PATH=/skywalking/certs/server.crt
# Same pattern: SW_ADMIN_SERVER_REST_SSL_*, SW_PROMQL_REST_SSL_*, SW_LOGQL_REST_SSL_*, SW_TRACEQL_REST_SSL_*

HTTP TLS is server-side only. Add authentication with a reverse proxy or mesh policy in front of 12800 and 17128.

Secure OAP-to-Storage Traffic

BanyanDB Auth and TLS

# BanyanDB liaison (users defined in a 0600 YAML file)
banyand liaison --auth-config-file=/etc/banyandb/auth.yaml \
  --tls=true --key-file=server.key --cert-file=server.crt

# OAP side
export SW_STORAGE_BANYANDB_USER=skywalking
export SW_STORAGE_BANYANDB_PASSWORD="$(cat /run/secrets/bydb-password)"
export SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH=/skywalking/bydb-tls/ca.crt

bydbctl then needs -u / -p (or a ~/.bydbctl.yaml with username and password).

Elasticsearch Auth and TLS

storage:
  selector: ${SW_STORAGE:elasticsearch}
  elasticsearch:
    namespace: ${SW_NAMESPACE:""}
    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:es-cluster:9200}
    protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"https"}
    user: ${SW_ES_USER:"skywalking"}
    password: ${SW_ES_PASSWORD:""}
    trustStorePath: ${SW_STORAGE_ES_SSL_JKS_PATH:""}
    trustStorePass: ${SW_STORAGE_ES_SSL_JKS_PASS:""}
    # properties file with user/password, rotated without restart
    secretsManagementFile: ${SW_ES_SECRETS_MANAGEMENT_FILE:""}

Create a dedicated Elasticsearch user limited to the SkyWalking index prefix, and restrict the ES network to OAP pods.

Instrument Applications

Java Agent

java -javaagent:/opt/skywalking-agent/skywalking-agent.jar \
  -Dskywalking.agent.service_name=checkout \
  -Dskywalking.collector.backend_service=oap.skywalking:11800 \
  -jar checkout.jar

On Kubernetes, SWCK can inject the agent instead:

kubectl label namespace shop swck-injection=enabled
kubectl patch deployment checkout -n shop --type merge \
  -p '{"spec":{"template":{"metadata":{"labels":{"swck-java-agent-injected":"true"}}}}}'

Python and NodeJS Agents

pip install "apache-skywalking==1.3.0"
SW_AGENT_NAME=orders SW_AGENT_COLLECTOR_BACKEND_SERVICES=oap:11800 sw-python run python3 app.py

npm install skywalking-backend-js@0.9.0

Send OpenTelemetry Data

OAP 11 enables otlp-traces,otlp-metrics,otlp-logs by default. Point an OTel Collector at it:

exporters:
  otlp/skywalking:
    endpoint: oap.skywalking:11800   # OTLP/gRPC
    tls:
      insecure: true                 # use TLS settings in production
  otlphttp/skywalking:
    endpoint: http://oap.skywalking:12800   # OTLP/HTTP, OAP 11.0+
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlp/skywalking]

Metrics only become SkyWalking metrics when a MAL rule under otel-rules/ matches them (select rules with SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES). OTLP traces are stored as Zipkin-format traces, so enable the Zipkin query module (SW_QUERY_ZIPKIN=default) or TraceQL (SW_TRACEQL=default) to read them.

Upgrade from 10.x to 11.0

  1. Upgrade BanyanDB to 0.11.x first (liaison nodes before data nodes, because of the vectorized frame format). OAP 11 will not start against 0.10.
  2. Replace the UI: switch skywalking/ui:<oap-version> (booster) to apache/skywalking-ui:horizon-1.0.0. The container port changes from 8080 to 8081, /graphql is no longer proxied by the UI, and logins must be configured.
  3. Expose the admin port 17128 to Horizon only (Helm: oap.ports.admin: 17128).
  4. Migrate scripts that used the removed UIConfigurationManagement GraphQL mutations to /ui-management/templates on the admin host.
  5. Move status/debug callers: /status/* and /debugging/* now live on the admin host, not 12800.
  6. Helm 4.9.0 to 5.0.0: rename oap.ports.zipkinreceiver / zipkinquery to zipkin-receiver / zipkin-query, and replace ui.env with ui.extraEnv / ui.envFromSecret.
helm upgrade skywalking oci://docker.io/apache/skywalking-helm \
  --version 5.0.0 -n skywalking -f values.yaml \
  --set oap.image.tag=11.0.0 --set ui.image.tag=horizon-1.0.0 \
  --set banyandb.image.tag=0.11.0 \
  --wait --wait-for-jobs

v10.4.0 Migration Notes

  • LAL breaking change: slowSql {} and sampledTrace {} sub-DSLs were removed. Set outputType: SlowSQL (or SampledTrace) on the rule, assign the fields (id, statement, latency, ...) directly in extractor {}, and add an explicit sink {} block. Without sink {} nothing is persisted. The bundled mysql-slowsql.yaml, pgsql-slowsql.yaml and redis-slowsql.yaml show the new form.
  • LALOutputBuilder.init() signature changed to init(LogData, Optional<Object> extraLog, NamingControl). Custom builders must be updated.
  • Default JDK: the Docker base image is JDK 25; -java11, -java17 and -java21 variants exist.
  • Virtual threads are on by default on JDK 25+; set SW_VIRTUAL_THREADS_ENABLED=false to fall back to platform threads.

Operate BanyanDB with bydbctl

bydbctl talks to the liaison HTTP port (default http://127.0.0.1:17913) and stores its target in ~/.bydbctl.yaml.

# List groups created by OAP (sw_records, sw_trace, sw_metricsMinute, ...)
bydbctl --addr http://banyandb:17913 group list

# Inspect a measure schema, then query the last 30 minutes
bydbctl measure get -g sw_metricsMinute -n service_cpm_minute
bydbctl measure query --start -30m -f - <<EOF
name: "service_cpm_minute"
groups: ["sw_metricsMinute"]
tagProjection:
  tagFamilies:
    - name: "storage-only"
      tags: ["entity_id"]
fieldProjection:
  names: ["total", "value"]
EOF

# Interactive BydbQL agent (TUI)
bydbctl agent

Measure and tag family names

The query above follows the upstream bydbctl example. Run measure get first to confirm the tag family and field names in your OAP version.

Check Health and Self-Observability

# OAP health (health-checker module is on by default in 11.0)
curl -s -o /dev/null -w '%{http_code}\n' http://oap:12800/healthcheck   # 200 healthy, 503 unhealthy

# OAP self-metrics for Prometheus
curl -s http://oap:1234/metrics | head

# BanyanDB metrics listener
curl -s http://banyandb:2121/metrics | head

kubectl get pods -n skywalking

OAP 11 also ships BanyanDB self-observability rules (otel-rules/banyandb/*, rebuilt in 11.0.0 around the cluster/container/group model, requires BanyanDB 0.11+).

Troubleshooting

OAP Crash-Loops After an Upgrade

kubectl logs -n skywalking -l component=oap --tail=500 | grep -i "Incompatible BanyanDB server API version"

If it matches, the BanyanDB image does not match the OAP line. Set banyandb.image.tag to 0.11.x for OAP 11. Do not override SW_STORAGE_BANYANDB_COMPATIBLE_SERVER_API_VERSIONS; unlisted pairings are unsupported.

OAP Not Receiving Agent Data

kubectl get svc -n skywalking | grep oap
kubectl logs -n skywalking -l component=oap --tail=200 | grep -iE "error|unauthenticated|ssl"
kubectl exec -it deploy/checkout -n shop -- sh -c 'nc -zv oap.skywalking 11800'

Common causes: the token is set on OAP but not on the agent (or the reverse), TLS is enabled on only one side, or agents point at 12800 instead of 11800.

UI Shows a Login Page Nobody Can Pass

/api/auth/health returns "configured": false. Add users as in Configure Horizon UI Logins.

BanyanDB Disk Filling Up

kubectl exec -n skywalking sts/skywalking-banyandb-data -- df -h /data   # pod/volume names depend on chart values

Lower the per-group TTLs (SW_STORAGE_BANYANDB_*_TTL_DAYS), enable warm/cold stages on cheaper volumes, or enable trace tail sampling. Reducing SW_CORE_*_TTL has no effect on BanyanDB. Expired segments are removed by a write-triggered retention task, so a completely idle group keeps its last data.

Sources