How-to Guides¶

What this page covers

Task recipes for the Victoria Stack: deployment (binaries, Docker, Helm, operator), vmauth routing and security setup, upgrades, backups, troubleshooting, and the Commands & Recipes library. Commands target VictoriaMetrics v1.152.x, VictoriaLogs v1.52.x, VictoriaTraces v0.11.x and operator v0.74.x (2026-09). Facts and flag defaults: Reference; background: Explanation; hub: Victoria Stack.

Deployment & Typical Setup¶

Single-Node (Simplest Production Path)¶

# VictoriaMetrics — single binary, metrics (bare number = months)
./victoria-metrics -storageDataPath=/data/vm -retentionPeriod=12

# VictoriaLogs — single binary, logs (default retention is 7d)
./victoria-logs -storageDataPath=/data/vl -retentionPeriod=30d

# VictoriaTraces — single binary, traces; OTLP/HTTP on :10428
# add -otlpGRPCListenAddr=:4317 to accept OTLP/gRPC (TLS on by default)
./victoria-traces -storageDataPath=/data/vt \
  -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false

Each binary starts an HTTP server and is ready to receive data; basic use needs no config file. Use -otlpGRPC.tls=false only on a trusted network, or pass -otlpGRPC.tlsCertFile/-otlpGRPC.tlsKeyFile.

Kubernetes (vmoperator)¶

The recommended Kubernetes path is the operator with CRDs:

# Install the operator (chart 0.67.x ships operator v0.74.1)
helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
helm install vmoperator vm/victoria-metrics-operator -n monitoring --create-namespace

# Deploy a VictoriaMetrics cluster via CRD
kubectl apply -n monitoring -f - <<EOF
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
  name: vm-cluster
spec:
  retentionPeriod: "12"
  replicationFactor: 2
  vminsert:
    replicaCount: 2
    resources:
      requests: { cpu: "500m", memory: "512Mi" }
  vmselect:
    replicaCount: 2
    resources:
      requests: { cpu: "500m", memory: "1Gi" }
  vmstorage:
    replicaCount: 3
    storageDataPath: /vm-data
    resources:
      requests: { cpu: "1", memory: "4Gi" }
    storage:
      volumeClaimTemplate:
        spec:
          resources:
            requests: { storage: 100Gi }
          storageClassName: fast-ssd
EOF

With replicationFactor: 2, keep at least 2*RF-1 = 3 vmstorage replicas so writes keep full replication while one node is down.

Kubernetes: VictoriaLogs and VictoriaTraces via the Operator¶

VictoriaLogs and VictoriaTraces CRDs use API version operator.victoriametrics.com/v1 (the VM* CRDs are v1beta1):

apiVersion: operator.victoriametrics.com/v1
kind: VLSingle
metadata:
  name: logs
  namespace: monitoring
spec:
  retentionPeriod: "30d"
  storage:
    resources:
      requests:
        storage: 50Gi
  resources:
    requests: { cpu: 500m, memory: 500Mi }
    limits: { memory: 4Gi }

VTSingle, VLCluster and VTCluster follow the same pattern; check field names against the operator API reference for your operator version. The operator tip also adds a VTAgent CRD and grpcSpec for OTLP/gRPC on VictoriaTraces.

Production Readiness Checklist¶

Configuration & Optimal Tuning¶

vmauth Routing Configuration¶

The central config file that routes traffic across all three databases by path (single-node backends shown for logs and traces; cluster tenant 0 for metrics):

# vmauth-config.yaml
unauthorized_user:
  url_map:
    # === METRICS (cluster, tenant 0) ===
    - src_paths:
        - "/api/v1/write"
        - "/api/v1/import.*"
      url_prefix: "http://vminsert:8480/insert/0/prometheus"
    - src_paths:
        - "/api/v1/query.*"
        - "/api/v1/series.*"
        - "/api/v1/label.*"
      url_prefix: "http://vmselect:8481/select/0/prometheus"

    # === LOGS (VictoriaLogs keeps its own /insert and /select prefixes) ===
    - src_paths:
        - "/insert/jsonline.*"
        - "/insert/elasticsearch/.*"
        - "/insert/loki/.*"
        - "/insert/opentelemetry/v1/logs"
        - "/select/logsql/.*"
      url_prefix: "http://victorialogs:9428"

    # === TRACES ===
    - src_paths:
        - "/insert/opentelemetry/v1/traces"
        - "/select/jaeger/.*"
        - "/select/tempo/.*"
      url_prefix: "http://victoriatraces:10428"

Do not leave unauthorized_user open in production

The example above has no authentication. In production move each url_map under a users: entry (Basic, Bearer or JWT), and override AccountID/ProjectID headers (see below) because backends accept tenant headers by default since v1.150.0.

For the flag reference, see Critical Tuning Flags.

Security Setup¶

vmauth — Authentication Proxy¶

vmauth is the primary authentication and routing component. It sits in front of vminsert/vmselect (and the logs/traces backends), authenticates requests and routes them to the correct tenant.

Token-Based Authentication¶

Each user gets a bearer_token or a username/password pair; vmauth maps users to backend URLs and tenants:

users:
  - username: "team-alpha"
    password: "secure-token-alpha"
    url_map:
      - src_paths: ["/api/v1/write"]
        url_prefix: "http://vminsert:8480/insert/1/prometheus/"
      - src_paths: ["/api/v1/query.*", "/api/v1/series", "/api/v1/labels"]
        url_prefix: "http://vmselect:8481/select/1/prometheus/"
    headers:
      - "AccountID: 1"
      - "ProjectID: 0"

  - username: "team-beta"
    password: "secure-token-beta"
    url_map:
      - src_paths: ["/api/v1/write"]
        url_prefix: "http://vminsert:8480/insert/2/prometheus/"
      - src_paths: ["/api/v1/query.*", "/api/v1/series", "/api/v1/labels"]
        url_prefix: "http://vmselect:8481/select/2/prometheus/"
    headers:
      - "AccountID: 2"
      - "ProjectID: 0"

Token isolation and tenant headers

Use distinct credentials for every tenant so a leaked token only exposes one tenant. Pinning AccountID/ProjectID in headers stops clients from switching tenant through headers when backends run with the default -enableMultitenancyViaHeaders=true (v1.150.0+).

URL Map Routing¶

Use url_map to route different API paths to different backends, for example a write-only service account:

users:
  - username: "writer-only"
    password: "write-token"
    url_map:
      - src_paths: ["/api/v1/write"]
        url_prefix: "http://vminsert:8480/insert/1/prometheus/"

This pattern allows write-only service accounts, read-only Grafana users, and different tenants per endpoint. Paths not matched by any src_paths are rejected.

JWT Authentication (v1.137.0+)¶

vmauth verifies JWTs with RSA/ECDSA public keys (or keys discovered via OIDC since v1.138.0) and can template tenant values from the vm_access claim:

users:
  - jwt:
      oidc:
        issuer: "https://idp.example.com/realms/observability"
      match_claims:
        aud: "vmauth"
    url_map:
      - src_paths: ["/api/v1/write"]
        url_prefix: "http://vminsert:8480/insert/{{.MetricsTenant}}/prometheus/"
      - src_paths: ["/api/v1/query", "/api/v1/query_range"]
        url_prefix: "http://vmselect:8481/select/{{.MetricsTenant}}/prometheus/"

Supported placeholders include {{.MetricsTenant}}, {{.MetricsAccountID}}, {{.MetricsProjectID}}, {{.MetricsExtraLabels}}, {{.MetricsExtraFilters}}, {{.LogsAccountID}}, {{.LogsProjectID}}, {{.LogsExtraFilters}} and {{.LogsExtraStreamFilters}}. They are only valid for JWT users; vm_access is optional since v1.147.0 (tenant 0:0 is assumed, or default_vm_access_claim is used).

Upgrade before relying on match_claims

GHSA-f99m-22fh-qw96 is an authorization bypass in JWT routing with match_claims, fixed in v1.152.0. Also always check aud via match_claims for OIDC tokens, and never use skip_verify: true outside tests.

mTLS-Based Routing (Enterprise)¶

Enterprise vmauth routes requests by client certificate subject fields; mTLS protection (-tls -mtls) must be enabled:

users:
  - mtls:
      organizational_unit: finance
    url_prefix: "http://victoriametrics-finance:8428"
  - mtls:
      organizational_unit: devops
    url_prefix: "http://victoriametrics-devops:8428"

Routing fields: organizational_unit, organization, common_name.

vmgateway — Enterprise Authentication¶

vmgateway (Enterprise) validates JWTs from an OIDC provider (Keycloak, Okta and similar), extracts the vm_access claim for tenant routing, and can enforce rate limits:

./vmgateway \
    -licenseFile=./vm-license.key \
    -enable.auth=true \
    -clusterMode=true \
    -write.url=http://localhost:8480 \
    -read.url=http://localhost:8481

The vm_access payload routes to the tenant, appends extra_labels on writes, applies extra_filters on reads, and uses mode as a bitfield for read (1) and write (2):

{
  "exp": 1617304574,
  "vm_access": {
    "tenant_id": {
      "account_id": 1,
      "project_id": 5
    },
    "extra_labels": {
      "team": "dev",
      "project": "mobile"
    },
    "extra_filters": ["{env=~\"prod|dev\",team!=\"test\"}"],
    "mode": 1
  }
}

Note that the vmgateway claim layout differs from the vmauth vm_access layout (metrics_account_id, logs_account_id, and so on).

TLS Configuration¶

vmauth Automatic TLS (Enterprise)¶

vmauth Enterprise can issue certificates from Let's Encrypt:

./vmauth \
    -httpListenAddr=:443 \
    -tls=true \
    -tlsAutocertHosts=metrics.example.com \
    -tlsAutocertEmail=admin@example.com \
    -tlsAutocertCacheDir=/var/cache/vmauth/tls

Manual TLS¶

For community builds or custom certificates:

./vmauth \
    -httpListenAddr=:443 \
    -tls=true \
    -tlsCertFile=/etc/vmauth/certs/server.crt \
    -tlsKeyFile=/etc/vmauth/certs/server.key \
    -auth.config=/etc/vmauth/config.yml

To keep internal endpoints (metrics, pprof, reload) off the public TLS listener, add a second listener: -httpInternalListenAddr=,localhost:8426 -httpListenAddr=0.0.0.0:443, -tls=true,false.

Backend TLS¶

Connections from vmauth to backends can also use TLS:

users:
  - username: "secure-client"
    password: "token"
    url_prefix:
      - "https://vmselect-1.internal:8481/select/1/prometheus/"
      - "https://vmselect-2.internal:8481/select/1/prometheus/"
    tls_insecure_skip_verify: false

Multiple url_prefix entries are load-balanced; they must be equivalent backends (not a mix of vminsert and vmselect).

Kubernetes Operator Security¶

The operator's strict-security mode applies a hardened pod and container security context (non-root user 65534, runAsNonRoot, seccomp profile, dropped capabilities, read-only root filesystem):

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: example
spec:
  useStrictSecurity: true

Enable it globally with the operator env var VM_ENABLESTRICTSECURITY=true, or fine-tune with spec.securityContext.

Internal Metrics Protection¶

Protect diagnostic endpoints with dedicated auth keys:

./vminsert \
  -metricsAuthKey="secret-for-metrics" \
  -pprofAuthKey="secret-for-pprof" \
  -flagsAuthKey="secret-for-flags"

Clients pass the key as the authKey query arg (for example /metrics?authKey=...). Alternatively serve these endpoints only on -httpInternalListenAddr.

VictoriaLogs Tenant Isolation¶

Configure vmauth to force the AccountID and ProjectID headers per team:

# vmauth for VictoriaLogs cluster
users:
  - username: "team-a"
    password: "token-a"
    headers:
      - "AccountID: 1"
      - "ProjectID: 0"
    url_map:
      - src_paths: ["/select/.*"]
        url_prefix: ["http://vlselect:9428/"]
      - src_paths: ["/insert/.*"]
        url_prefix: ["http://vlinsert:9428/"]
  - username: "team-b"
    password: "token-b"
    headers:
      - "AccountID: 2"
      - "ProjectID: 0"
    url_map:
      - src_paths: ["/select/.*"]
        url_prefix: ["http://vlselect:9428/"]

For JWT users, use "AccountID: {{.LogsAccountID}}" and "ProjectID: {{.LogsProjectID}}" instead of fixed values.

Network Security Best Practices¶

Never expose ingestion or storage nodes to the internet — put vmauth (or a gateway) in front.
Use Kubernetes NetworkPolicies to restrict pod-to-pod traffic (operator v0.74.0+ looks up NetworkPolicy objects; its ClusterRole needs the extra RBAC).
Expose only vmauth externally.
Use mTLS between components where required (Enterprise), otherwise TLS + network isolation.
Isolate tenants by path or pinned headers; never expose /select/multitenant/* without pinned extra_label and blank extra_filters.

For the security model, see Security Architecture Overview and the Hardening Checklist.

Upgrading¶

Upgrade VictoriaMetrics¶

Pick a target: latest release (community) or the newest patch of your LTS line (Enterprise: v1.148.x or v1.136.x as of 2026-09).
Read every Update Note in the CHANGELOG between your version and the target. Recent ones: per-partition IndexDB (v1.133.0), -disableRerouting default (v1.149.0), POST-only delete endpoints (v1.149.0), tenant headers on by default (v1.150.0).
Skip known-bad versions (v1.140.0, v1.136.4, v1.122.19 — MetricsQL operator bug).
Cluster: do a rolling upgrade node by node (upstream's order is vmstorage, then vminsert, then vmselect; at least two nodes of each type). This is safe only if the new version is compatible with the remaining old components, so check the CHANGELOG compatibility notes; otherwise upgrade all nodes of a role at once and accept a short interruption.
Single-node: stop, replace binary, start. In-memory data from the last seconds is flushed on graceful shutdown (SIGINT/SIGTERM).

Upgrade a VictoriaLogs Cluster Across v1.51¶

Coming from v1.38.0-v1.50.0 to v1.52.0+, avoid query downtime from the internal protocol change:

Upgrade all vlstorage nodes to v1.51.1.
Upgrade all vlselect nodes to v1.51.1.
Upgrade the whole cluster to v1.52.0 (storage first, then select and insert).
Review LogsQL queries and vmalert vlogs rules for the v1.51.0 bare-word pipe change (foo | bar must become foo bar or foo | filter bar).

Best Practices¶

Metrics¶

Global relabeling: add datacenter/environment labels at vmagent (-remoteWrite.label) before data reaches storage.
Drop high-cardinality labels: use vmagent relabeling to drop labels like pod_ip or request_id before ingestion.
Recording rules: precompute expensive MetricsQL via vmalert.
Deduplication: with replication set -dedup.minScrapeInterval=1ms on vmselect; with HA vmagent pairs set it to the scrape interval.

Logs¶

Do not translate: use native APIs — point Fluent Bit at /insert/jsonline rather than through an intermediary.
Structured logging: send JSON so fields are stored as columns.
Stream fields: set _stream_fields to a few low-cardinality fields (app, namespace, host); never put trace_id in stream fields.
Retention per signal: for example logs 30d, metrics 12 months, traces 14d; add -retention.maxDiskUsagePercent as a disk guard.

Operations¶

Monitor with itself: scrape every component's /metrics and import the official Grafana dashboards.
Back up regularly: daily full plus hourly incremental vmbackup to object storage (the "smart backup" pattern).
Pin versions: for production pick an LTS line (Enterprise) or a tested latest release; VictoriaTraces is 0.x, so expect API changes.

Troubleshooting¶

Common Issues & Playbook¶

Symptom	Likely Cause	Fix
High CPU on vmstorage during queries	Queries over long ranges or many series	Lower `-search.maxQueryDuration` / `-search.maxUniqueTimeseries` on vmselect; add recording rules; scale vmselect
OOM or high RAM on vmstorage	High active series or churn	Drop unused labels at vmagent; set `-storage.maxHourlySeries`/`-storage.maxDailySeries`; add memory. `-memory.allowedPercent` only sizes caches
"too many unique timeseries" / query rejected	Query touches more series than the limit	Refine the query or raise `-search.maxUniqueTimeseries` on vmselect
Slow VictoriaLogs queries	Large time range, unselective word filters	Add `_time:` filter and stream filters (`{app="x"}`) first
vmagent not discovering targets	`VMServiceScrape`/`VMPodScrape` not selected	Check operator logs and the `VMAgent` selectors / labels
VictoriaTraces rejects OTLP/gRPC	gRPC listener disabled by default, or TLS mismatch	Start with `-otlpGRPCListenAddr=:4317`; match client TLS to `-otlpGRPC.tls`
Seconds of data missing after vmstorage crash	No WAL: unflushed in-memory data lost on unclean shutdown	Expected; senders (vmagent persistent queue) resend. Use graceful shutdowns
Client can read another tenant	Backends accept `AccountID` headers (v1.150.0+ default)	Override headers in vmauth or set `-enableMultitenancyViaHeaders=false`
`GET .../delete_series` returns 405	POST required since v1.149.0	Use `curl -X POST`
vmalert `vlogs` rules fail validation after upgrade	LogsQL v1.51.0 syntax change embedded in vmalert v1.147.0+	Rewrite bare-word pipes (`\| filter ...`)

Commands & Recipes¶

Installation¶

Docker (Quick Start — All Components)¶

# VictoriaMetrics (metrics)
docker run -d --name vm \
  -p 8428:8428 \
  -v vm-data:/storage \
  victoriametrics/victoria-metrics:v1.152.0 \
  -storageDataPath=/storage -retentionPeriod=12

# VictoriaLogs (logs)
docker run -d --name vl \
  -p 9428:9428 \
  -v vl-data:/vlogs \
  victoriametrics/victoria-logs:v1.52.0 \
  -storageDataPath=/vlogs -retentionPeriod=30d

# VictoriaTraces (traces): OTLP/HTTP on 10428, OTLP/gRPC on 4317 (plaintext, dev only)
docker run -d --name vt \
  -p 10428:10428 \
  -p 4317:4317 \
  -v vt-data:/vtraces \
  victoriametrics/victoria-traces:v0.11.1 \
  -storageDataPath=/vtraces -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false

VictoriaLogs v1.52.0+ and VictoriaTraces v0.10.0+ images are distroless (no shell); debug with kubectl debug or an ephemeral container.

Docker Compose (Full Stack)¶

# compose.yaml — Victoria stack for development (no auth)
services:
  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.152.0
    ports: ["8428:8428"]
    volumes: ["vm-data:/storage"]
    command:
      - "-storageDataPath=/storage"
      - "-retentionPeriod=12"

  victorialogs:
    image: victoriametrics/victoria-logs:v1.52.0
    ports: ["9428:9428"]
    volumes: ["vl-data:/vlogs"]
    command:
      - "-storageDataPath=/vlogs"
      - "-retentionPeriod=30d"

  victoriatraces:
    image: victoriametrics/victoria-traces:v0.11.1
    ports:
      - "10428:10428"  # HTTP: OTLP/HTTP ingest, Jaeger and Tempo query APIs
      - "4317:4317"    # OTLP gRPC (only with -otlpGRPCListenAddr)
    volumes: ["vt-data:/vtraces"]
    command:
      - "-storageDataPath=/vtraces"
      - "-otlpGRPCListenAddr=:4317"
      - "-otlpGRPC.tls=false"

  vmagent:
    image: victoriametrics/vmagent:v1.152.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - "-promscrape.config=/etc/prometheus/prometheus.yml"
      - "-remoteWrite.url=http://victoriametrics:8428/api/v1/write"

  vmauth:
    image: victoriametrics/vmauth:v1.152.0
    ports: ["8427:8427"]
    volumes:
      - ./vmauth-config.yml:/etc/vmauth/config.yml
    command:
      - "-auth.config=/etc/vmauth/config.yml"

  vmalert:
    image: victoriametrics/vmalert:v1.152.0
    volumes:
      - ./alert-rules.yml:/etc/rules/rules.yml
    command:
      - "-rule=/etc/rules/*.yml"
      - "-datasource.url=http://victoriametrics:8428"
      - "-remoteWrite.url=http://victoriametrics:8428"
      - "-remoteRead.url=http://victoriametrics:8428"
      - "-notifier.blackhole"

  grafana:
    image: grafana/grafana-oss:latest
    ports: ["3000:3000"]
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_PLUGINS_PREINSTALL=victoriametrics-logs-datasource

volumes:
  vm-data:
  vl-data:
  vt-data:

-notifier.blackhole lets vmalert run without Alertmanager in development; replace it with -notifier.url=http://alertmanager:9093 in real setups. GF_PLUGINS_PREINSTALL is the plugin-install variable in current Grafana releases; older releases use GF_INSTALL_PLUGINS.

Helm (Kubernetes)¶

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update

# Full k8s monitoring stack (operator + VMSingle/VMCluster + vmagent + vmalert + dashboards)
helm install vmks vm/victoria-metrics-k8s-stack -n monitoring --create-namespace

# Single-node VictoriaMetrics
helm install vm vm/victoria-metrics-single -n monitoring

# Cluster VictoriaMetrics
helm install vm-cluster vm/victoria-metrics-cluster -n monitoring -f vm-values.yaml

# vmoperator only (manages components via CRDs)
helm install vmoperator vm/victoria-metrics-operator -n monitoring

# vmagent / vmalert / vmauth
helm install vmagent vm/victoria-metrics-agent -n monitoring
helm install vmalert vm/victoria-metrics-alert -n monitoring
helm install vmauth vm/victoria-metrics-auth -n monitoring

# VictoriaLogs (single or cluster) and a log collector DaemonSet
helm install vl vm/victoria-logs-single -n monitoring
helm install vlc vm/victoria-logs-cluster -n monitoring
helm install vlcollector vm/victoria-logs-collector -n monitoring

# VictoriaTraces (single or cluster)
helm install vt vm/victoria-traces-single -n monitoring
helm install vtc vm/victoria-traces-cluster -n monitoring

vmagent Recipes¶

# Start vmagent as a drop-in Prometheus scraper
./vmagent \
  -promscrape.config=/path/to/prometheus.yml \
  -remoteWrite.url=http://victoriametrics:8428/api/v1/write

# Add global labels to all scraped metrics, write to cluster tenant 0
./vmagent \
  -remoteWrite.label=datacenter=us-east-1 \
  -remoteWrite.label=env=production \
  -promscrape.config=prometheus.yml \
  -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write

# Multi-destination remote write (each URL gets its own persistent queue)
./vmagent \
  -remoteWrite.url=http://vm-primary:8428/api/v1/write \
  -remoteWrite.url=http://vm-secondary:8428/api/v1/write

Data Ingestion Recipes¶

Fluent Bit → VictoriaLogs¶

# fluent-bit.conf — push logs directly to VictoriaLogs
[OUTPUT]
    Name  http
    Match *
    Host  victorialogs
    Port  9428
    URI   /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
    Format json_lines
    Json_date_format iso8601
    Compress gzip

OpenTelemetry Collector → VictoriaTraces and VictoriaMetrics¶

# otel-collector-config.yaml
exporters:
  otlp/victoriatraces:
    endpoint: "victoriatraces:4317"   # requires -otlpGRPCListenAddr=:4317
    tls:
      insecure: true                  # only if -otlpGRPC.tls=false

  otlphttp/victoriatraces:
    traces_endpoint: "http://victoriatraces:10428/insert/opentelemetry/v1/traces"

  otlphttp/victoriametrics:
    metrics_endpoint: "http://victoriametrics:8428/opentelemetry/v1/metrics"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/victoriatraces]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/victoriametrics]

Use either the gRPC or the HTTP traces exporter, not both. prometheusremotewrite to /api/v1/write also works for metrics.

Promtail / Loki Push → VictoriaLogs¶

# promtail-config.yaml — VictoriaLogs accepts the Loki push API
clients:
  - url: http://victorialogs:9428/insert/loki/api/v1/push

Direct OTLP → VictoriaTraces¶

HTTP: http://victoriatraces:10428/insert/opentelemetry/v1/traces
gRPC: victoriatraces:4317 (after enabling -otlpGRPCListenAddr)

vmalert with VictoriaLogs¶

Evaluate LogsQL alerting rules (type: vlogs) against VictoriaLogs and write results to VictoriaMetrics:

# vlogs-rules.yml
groups:
  - name: ServiceLog
    type: vlogs
    interval: 5m
    rules:
      - alert: HasMoreThan10ErrorLogs
        expr: '{env=prod} status:in(error,warn) | stats by (k8s.pod.name) count() as error_logs | filter error_logs:>10'

./vmalert -rule=vlogs-rules.yml \
  -datasource.url=http://victorialogs:9428 \
  -remoteWrite.url=http://victoriametrics:8428 \
  -notifier.url=http://alertmanager:9093

vmauth Routing Config¶

For the full url_map with commentary, see vmauth Routing Configuration in the Configuration section.

Backup & Restore¶

# Create an instant snapshot (single-node) — vmbackup can do this for you
curl http://victoriametrics:8428/snapshot/create
# Returns: {"status":"ok","snapshot":"<snapshot-name>"}

# Backup to S3 (creates a snapshot via -snapshot.createURL)
./vmbackup \
  -storageDataPath=/data/vm \
  -snapshot.createURL=http://localhost:8428/snapshot/create \
  -dst=s3://my-bucket/vm-backups/latest

# Incremental: re-run against the SAME -dst; only new/changed parts are uploaded
./vmbackup \
  -storageDataPath=/data/vm \
  -snapshot.createURL=http://localhost:8428/snapshot/create \
  -dst=s3://my-bucket/vm-backups/latest

# Fast daily full copy via server-side copy from an existing backup
./vmbackup -origin=s3://my-bucket/vm-backups/latest -dst=s3://my-bucket/vm-backups/20260925

# Restore (VictoriaMetrics must be stopped)
./vmrestore \
  -src=s3://my-bucket/vm-backups/latest \
  -storageDataPath=/data/vm-restored

Cluster: run vmbackup on every vmstorage node with -snapshot.createURL=http://vmstorage-N:8482/snapshot/create and a per-node -dst. Enterprise vmbackupmanager automates the hourly/daily/weekly/monthly schedule.

Back Up a VictoriaLogs Partition¶

# 1. Snapshot one per-day partition (POST)
curl -X POST 'http://victorialogs:9428/internal/partition/snapshot/create?partition_prefix=20260924'
# returns a JSON array of snapshot paths

# 2. Copy it (or rclone to S3/GCS)
rsync -avh --delete <snapshot-path>/ backup@host:/backups/vl/20260924

# 3. Delete the snapshot
curl -X POST 'http://victorialogs:9428/internal/partition/snapshot/delete?path=<snapshot-path>'

The same partition API applies to vlstorage nodes and to VictoriaTraces. The tip after VictoriaLogs v1.52.0 makes /internal/partition/* POST-only; using POST works on both old and new versions.

API Recipes¶

# Query VictoriaMetrics (PromQL/MetricsQL)
curl -s "http://vm:8428/api/v1/query?query=up" | jq .

# Range query
curl -s "http://vm:8428/api/v1/query_range?query=rate(http_requests_total[5m])&start=-1h&step=60s" | jq .

# Import data via JSON line format
curl -d '{"metric":{"__name__":"test","job":"api"},"values":[1,2,3],"timestamps":[1617000000000,1617000001000,1617000002000]}' \
  http://vm:8428/api/v1/import

# Delete series (POST only since v1.149.0)
curl -X POST 'http://vm:8428/api/v1/admin/tsdb/delete_series' -d 'match[]={job="test"}'

# Query VictoriaLogs (LogsQL)
curl -s http://vl:9428/select/logsql/query -d 'query=_time:5m error' | head

# Push a test log
curl -X POST "http://vl:9428/insert/jsonline?_stream_fields=app&_msg_field=msg" \
  -d '{"app":"test","msg":"hello from curl","level":"info"}'

# Look up a trace by ID (Jaeger API on VictoriaTraces)
curl -s "http://vt:10428/select/jaeger/api/traces/<trace_id>" | jq .

# Check health
curl -s "http://vm:8428/-/healthy" && echo "OK"

Grafana Data Source Config¶

# Grafana provisioning for the Victoria stack (behind vmauth)
apiVersion: 1
datasources:
  - name: VictoriaMetrics
    type: prometheus
    url: http://vmauth:8427
    isDefault: true
    jsonData:
      httpMethod: POST

  - name: VictoriaLogs
    type: victoriametrics-logs-datasource   # plugin must be installed
    url: http://vmauth:8427

  - name: VictoriaTraces (Jaeger)
    type: jaeger
    url: http://vmauth:8427/select/jaeger

  - name: VictoriaTraces (Tempo, experimental)
    type: tempo
    url: http://vmauth:8427/select/tempo

The vmauth routes must forward /select/jaeger/.* and /select/tempo/.* to VictoriaTraces (see the routing config above). VictoriaMetrics also offers a victoriametrics-metrics-datasource plugin with MetricsQL-specific features; the built-in Prometheus type works for most dashboards.