LGTM Stack Reference
Look-up facts for the Grafana LGTM stack (Loki, Grafana, Tempo, Mimir, plus Pyroscope and Alloy): versions, licenses, Helm charts, deployment modes, components, ports, configuration keys with defaults, limits, scale figures and hardening checklists. For how the pieces fit together see Explanation; for tasks and recipes see How-to Guides.
Verification scope
Versions and dates were checked on 2026-09-25 against the GitHub release pages and the CHANGELOG.md files of each project. Configuration keys and defaults were checked against the generated config references (cmd/mimir/config-descriptor.json, Loki release-3.7.x docs/sources/shared/configuration.md, Tempo docs/sources/tempo/configuration/manifest.md, Pyroscope configuration reference). Re-check them before you upgrade.
Version and Support Matrix
| Component |
Signal |
Latest stable (date) |
Current major line |
Also patched |
License |
Release cadence |
| Grafana Mimir |
Metrics |
3.2.1 (2026-09-10) |
3.x (3.0 GA 2025-11) |
3.1.6, 3.0.9 |
AGPL-3.0 |
Minor release about once per quarter (early Mar/Jun/Sep/Dec) |
| Grafana Loki |
Logs |
3.7.8 (2026-09-17) |
3.x (3.0 in 2024) |
3.6.17 |
AGPL-3.0 |
Major about yearly, minor about quarterly, patch 1-2 per month; weekly k<N> builds from main |
| Grafana Tempo |
Traces |
3.0.3 (2026-08-13) |
3.x (3.0.0 on 2026-05-28) |
2.10.8, 2.9.5 |
AGPL-3.0 |
3.1.0-rc.1 published 2026-09-17 |
| Grafana Pyroscope |
Profiles |
2.3.1 (2026-09-08) |
2.x (2.0.0 on 2026-04-20) |
2.2.2 |
AGPL-3.0 |
Minor about every 4-6 weeks in 2026 |
| Grafana Alloy |
Collection |
1.20.0 (2026-09-25) |
1.x |
1.19.2 |
Apache-2.0 |
Minor about monthly |
| Grafana |
Visualization |
13.2.2 (2026-09-15) |
13.x |
13.1.6, 13.0.9, 12.4.11 |
AGPL-3.0 |
See Grafana |
Loki 3.8 is not released yet
The CHANGELOG.md on Loki's main branch carries a 3.8.0 (2026-06-08) entry (removing BoltDB, Cassandra, DynamoDB, BigTable and gRPC store backends), but as of 2026-09-25 there is no v3.8.0 tag, no release-3.8.x branch and no v3.8 release notes. The GitHub "Latest" release is 3.7.8. Treat the 3.8 removals as upcoming.
Helm Charts
The Grafana Labs Helm repository was split in 2026. Several charts moved to a community-maintained repository, and the old copies stop receiving updates.
| Chart |
Helm repository |
Source repo |
Latest chart (app) |
Status |
mimir-distributed |
https://grafana.github.io/helm-charts |
grafana/mimir (operations/helm/charts/) |
6.2.0 |
Maintained by Grafana Labs. Since 6.0.0 deploys the ingest storage architecture by default and bundles a single-node Kafka for demos; requires Kubernetes 1.29+ |
loki |
https://grafana-community.github.io/helm-charts |
grafana-community/helm-charts |
18.13.5 (3.7.8) |
Forked to the community repo effective 2026-03-16; default deploymentMode: Monolithic |
tempo-distributed |
https://grafana-community.github.io/helm-charts |
grafana-community/helm-charts |
3.7.0 (3.0.3) |
Moved after 2026-01-30. Tempo 3 microservices: needs an external Kafka-compatible cluster (ingest.kafka.address) |
tempo (monolithic) |
https://grafana-community.github.io/helm-charts |
grafana-community/helm-charts |
3.0.0 (3.0.3) |
Moved after 2026-01-30 |
grafana |
https://grafana-community.github.io/helm-charts |
grafana-community/helm-charts |
13.2.5 (13.2.2) |
Moved after 2026-01-30 |
pyroscope |
https://grafana.github.io/helm-charts |
grafana/pyroscope (operations/pyroscope/helm/) |
2.3.1 (2.3.1) |
Maintained by Grafana Labs. Chart defaults to v2 storage since 2.1.0 |
alloy |
https://grafana.github.io/helm-charts |
grafana/alloy (operations/helm/charts/) |
1.12.1 |
Maintained by Grafana Labs |
k8s-monitoring |
https://grafana.github.io/helm-charts |
grafana/k8s-monitoring-helm |
4.5.2 (checked 2026-09-27) |
Maintained by Grafana Labs; manages Alloy instances for Kubernetes telemetry |
lgtm-distributed |
https://grafana.github.io/helm-charts |
grafana/helm-charts |
3.0.1 |
Deprecated (deprecated: true in Chart.yaml); not migrated |
loki-distributed, loki-stack, promtail |
https://grafana.github.io/helm-charts |
grafana/helm-charts |
0.80.6, 2.10.3, 6.17.1 |
Deprecated (deprecated: true in each Chart.yaml, checked 2026-09-27); superseded by loki and alloy. loki-simple-scalable and grafana-agent are no longer in grafana/helm-charts (the grafana-agent chart lives in grafana/agent, last version 0.44.2) |
OCI install
Community charts are also published as OCI artifacts: helm install <release> oci://ghcr.io/grafana-community/helm-charts/<chart>. Releases published before the end-of-January-2026 migration were signed with a different key.
Deployment Modes by Backend
All four backends compile every component into one binary and select roles with -target.
| Mode |
Mimir 3.x |
Loki 3.x |
Tempo 3.x |
Pyroscope 2.x |
Monolithic (-target=all) |
Yes; can be scaled horizontally with several all replicas |
Yes; about 20 GB/day guideline |
Yes; no Kafka needed; guideline 25-35 MB/s or 55k-80k spans/s; only one all instance is supported |
Yes (filesystem as object store allowed only in single-node) |
| HA monolithic |
Via multiple -target=all replicas |
Yes: RF=3, at least 3 replicas, memberlist, one main compactor; replaces SSD |
Not supported (backend scheduler is a singleton) |
Not supported: monolithic mode is for a single instance, and several instances do not share information (deployment-modes doc; its scaled-monolithic section is commented out) |
| Simple Scalable (read/write/backend) |
Experimental read-write mode removed in 3.0 |
Deprecated; will be removed in Loki 4.0 |
scalable-single-binary target removed in 3.0 |
Not offered |
| Microservices |
Recommended for production; mimir-distributed chart |
Recommended for very large clusters |
Recommended for production; requires Kafka-compatible queue |
Recommended at scale; v2 components |
| Kafka in the write path |
Ingest storage architecture (preferred since 3.0; flag -ingest-storage.enabled, default false in the binary, true in the Helm chart) |
Experimental dataobj / Kafka paths on main (not a supported mode as of 3.7) |
Mandatory in microservices mode |
No (segment-writers write straight to object storage) |
Components by Backend (Current Majors)
| Backend |
Write path |
Read path |
Background / state |
Removed or replaced |
| Mimir 3.x (ingest storage) |
distributor, Kafka, ingester (consumes one partition per zone), block-builder (optional) |
query-frontend, query-scheduler (now required), querier, store-gateway |
compactor, ruler, alertmanager, overrides-exporter, usage-tracker (experimental), compactor scheduler (experimental, 3.2) |
Redis cache backend removed in 3.0 (Memcached only); query-frontend embedded scheduler removed in 3.0 |
| Mimir 3.x (classic) |
distributor, ingester (RF=3 quorum, local WAL) |
as above |
as above |
— |
| Loki 3.x |
distributor, ingester |
query-frontend, query-scheduler, querier, index-gateway |
compactor, ruler, pattern-ingester, bloom-planner / bloom-builder / bloom-gateway (experimental), overrides-exporter |
Promtail removed as of 3.7.3 (merged into Alloy; lambda-promtail kept); Table Manager deprecated |
| Tempo 3.x |
distributor, Kafka, block-builder, live-store, metrics-generator (optional) |
query-frontend, querier (reads live-stores and object storage) |
backend-scheduler + backend-worker (compaction, retention, redaction) |
ingester and compactor removed; OpenCensus receiver removed; vParquet3 writes refused in 3.1 |
| Pyroscope 2.x (v2 storage) |
distributor, segment-writer |
query-frontend, query-backend |
metastore (Raft, the only stateful component), compaction-worker |
v1 ingester path still selectable with -write-path=ingester / -architecture.storage=v1 |
Default Ports and Endpoints
| Component |
HTTP |
gRPC |
Key write endpoints |
Key read endpoints |
| Mimir |
8080 |
9095 |
/api/v1/push (remote_write), /otlp/v1/metrics |
/prometheus/api/v1/query, /prometheus/api/v1/query_range |
| Loki |
3100 |
9095 |
/loki/api/v1/push, /otlp/v1/logs (OTLP/HTTP, exporter endpoint http://loki:3100/otlp) |
/loki/api/v1/query_range, /loki/api/v1/tail |
| Tempo |
3200 |
9095 |
OTLP 4317 (gRPC) / 4318 (HTTP) on distributors; Jaeger and Zipkin receivers when enabled |
/api/traces/<id>, /api/v2/traces/<id>, /api/search |
| Pyroscope |
4040 |
4040 |
/ingest, Push RPC |
/querier.v1.QuerierService/* (Connect API) |
| Alloy |
12345 (UI) |
— |
OTLP receivers as configured (usually 4317/4318) |
— |
| Grafana |
3000 |
— |
— |
— |
grafana/otel-lgtm (dev image) |
3000 (Grafana), 4317/4318 (OTLP) |
— |
— |
Uses Prometheus, not Mimir, as its metrics store |
Query Languages
| Signal |
Language |
Notes |
| Metrics |
PromQL |
Mimir 3.0 made the streaming Mimir Query Engine (MQE) the default; supports experimental Prometheus functions when enabled |
| Logs |
LogQL |
Stream selector {label="value"} required, then line filters, parsers (json, logfmt, pattern) and metric queries |
| Traces |
TraceQL |
Span selectors, structural operators (>>, >, ~), and TraceQL metrics (GA in Tempo 3.0; arithmetic in 3.1) |
| Profiles |
Label selectors on a profile type |
e.g. process_cpu:cpu:nanoseconds:cpu:nanoseconds{service_name="api"}. "FlameQL" was the query-language name in the original Pyroscope OSS project and is not used in current Grafana Pyroscope docs, which emphasise the queryless Profiles Drilldown UI |
Retention Settings
| Backend |
Key |
Default |
Notes |
| Mimir |
limits.compactor_blocks_retention_period (-compactor.blocks-retention-period) |
0 (keep forever) |
Per-tenant override supported. Not blocks_storage.tsdb.retention_period, which is the ingester-local TSDB retention (default 13h) |
| Loki |
limits_config.retention_period |
0s |
Only enforced when compactor.retention_enabled: true (default false) |
| Tempo 3.x |
backend_worker.compaction.block_retention (also under backend_scheduler.provider.compaction.compaction) |
336h (14 days) |
Per-tenant override compaction.block_retention. Tempo 2.x used compactor.compaction.block_retention |
| Pyroscope 2.x |
limits.retention_period (-retention-period) |
31d |
v1 storage used compactor_blocks_retention_period (default 0s) |
Multi-Tenancy Settings
| Backend |
Enable |
Default |
Cross-tenant queries |
| Mimir |
multitenancy_enabled (-auth.multitenancy-enabled) |
true |
tenant_federation.enabled: true, then X-Scope-OrgID: a\|b |
| Loki |
auth_enabled |
true |
querier.multi_tenant_queries_enabled: true, then X-Scope-OrgID: a\|b; push and tail reject multi-tenant requests |
| Tempo |
multitenancy_enabled |
false (single tenant single-tenant) |
Pipe-separated tenants in X-Scope-OrgID |
| Pyroscope |
multitenancy_enabled (-auth.multitenancy-enabled) |
false (tenant anonymous) |
Pipe-separated tenants in X-Scope-OrgID: the tenant interceptor accepts tenant-a\|tenant-b and the v2 query frontend reads all listed tenants (pkg/tenant/interceptor.go); no separate federation flag |
Tenant ID Rules
| Property |
Mimir |
Loki |
Tempo |
| Max length |
150 bytes |
150 bytes |
150 bytes |
| Allowed characters |
0-9 a-z A-Z ! - _ . * ' ( ) |
Same set |
Same set |
| Invalid values |
., .., __mimir_cluster |
., .. |
., .. |
Tempo's distributor resolves the tenant with dskit's tenant.TenantID (pkg/validation/tenantid.go), which applies the same ValidTenantID rules as Loki: 150-byte limit, the character set above, and no . or .. (dskit tenant/tenant.go, checked 2026-09-27).
Per-Tenant Limit Keys (Defaults)
| Backend |
Key |
Default |
Meaning |
| Mimir |
ingestion_rate |
10000 |
Samples/s per tenant |
| Mimir |
ingestion_burst_size |
200000 |
Burst samples |
| Mimir |
max_global_series_per_user |
150000 |
In-memory series across ingesters |
| Mimir |
max_fetched_chunks_per_query |
2000000 |
Per query |
| Mimir |
max_fetched_series_per_query |
0 (unlimited) |
Per query |
| Mimir |
max_global_exemplars_per_user |
0 (exemplars disabled) |
Must be > 0 for exemplar storage |
| Loki |
ingestion_rate_mb |
4 |
MB/s per tenant |
| Loki |
max_global_streams_per_user |
5000 |
Active streams |
| Loki |
per_stream_rate_limit |
3MB |
Per stream |
| Loki |
max_query_length |
30d1h |
Query range |
| Loki |
max_chunks_per_query |
2000000 |
Per query |
| Loki |
allow_structured_metadata |
true |
Requires schema v13 + TSDB index |
| Tempo |
ingestion.rate_limit_bytes |
30000000 |
Bytes/s |
| Tempo |
ingestion.burst_size_bytes |
30000000 |
Burst bytes |
| Tempo |
ingestion.max_traces_per_user |
10000 |
Live traces |
| Tempo |
global.max_bytes_per_trace |
5000000 |
Per trace |
Tempo 3.0 disables the legacy flat overrides format by default (enable_legacy_overrides: true is a temporary escape hatch). Use the scoped format shown above.
Object Storage Layout
Each backend must use its own bucket, or at least its own prefix. Never share a path.
| Component |
Recommended bucket |
Contents |
| Mimir (blocks) |
observability-mimir-blocks |
Per-tenant TSDB blocks (2 h ranges, compacted up to 24 h by default) |
| Mimir (ruler) |
observability-mimir-ruler |
Recording and alerting rule groups |
| Mimir (alertmanager) |
observability-mimir-alertmanager |
Alertmanager config and state |
| Loki (chunks + index) |
observability-loki-chunks |
Compressed log chunks + TSDB index (schema v13) |
| Loki (ruler) |
observability-loki-ruler |
Rule groups |
| Tempo (traces) |
observability-tempo-traces |
Apache Parquet blocks (vParquet5 default in 3.1, vParquet4 in 3.0) + bloom filters |
| Pyroscope (profiles) |
observability-pyroscope-data |
v2 segments and compacted blocks |
Kubernetes Deployment Matrix
Minimum HA replica counts are rules of thumb, not vendor requirements.
| Component |
Helm chart |
Pods (min HA) |
Scaling dimension |
Key resource |
| Grafana |
grafana-community/grafana |
2-3 |
HPA (CPU/memory) |
Memory |
| Mimir |
grafana/mimir-distributed |
distributor x2, Kafka (external), ingester x2 zones, querier x2, store-gateway x3, compactor x1 |
Per-component HPA/KEDA |
Memory (ingesters), CPU (queriers) |
| Loki |
grafana-community/loki |
Monolithic x3 (HA) or distributor x2, ingester x3, querier x2 |
Per-component |
Memory (ingesters), CPU (queriers) |
| Tempo |
grafana-community/tempo-distributed |
distributor x2, Kafka, block-builder = partitions, live-store = partitions (zone-aware), querier x2 |
Per-component; KEDA for live-store |
Memory (live-store) |
| Pyroscope |
grafana/pyroscope |
1 (monolithic) to 3+ per component |
Replicas |
Memory |
| Alloy |
grafana/alloy |
1 per node (DaemonSet) + clustered Deployment for scraping |
DaemonSet / clustering |
CPU, memory |
| PostgreSQL / MySQL |
External managed |
HA pair |
Managed service |
Disk IOPS |
| Memcached |
Bundled in mimir-distributed / tempo-distributed, or external |
2-3 |
Replicas |
Memory |
Caching
| Cache layer |
Purpose |
Technology |
| Query results cache |
Cache query responses in query-frontends |
Memcached (Mimir 3.0 removed Redis support; Loki and Tempo still accept Redis) |
| Chunks cache |
Cache chunks fetched from object storage |
Memcached |
| Index / postings cache |
Cache index lookups |
Memcached |
| Metadata cache |
Cache bucket index and block metadata |
Memcached |
Tempo 3.1 rewrote its experimental Redis cache: Redis Cluster (Redis 7+) is the default and Sentinel support was removed.
Deprecations and Removals (2025-2026)
| Change |
Version |
Replacement |
| Mimir experimental read-write (SSD-like) mode removed |
Mimir 3.0 |
Monolithic or microservices |
| Mimir Redis cache backend removed |
Mimir 3.0 |
Memcached |
| Mimir query-scheduler becomes required |
Mimir 3.0 |
Deploy a dedicated query-scheduler |
| Tempo ingester and compactor removed |
Tempo 3.0 |
block-builder, live-store, backend-scheduler/worker |
Tempo scalable-single-binary target removed |
Tempo 3.0 |
Monolithic or microservices |
| Tempo OpenCensus receiver removed |
Tempo 3.0 |
OTLP |
| Tempo vParquet3 writes refused |
Tempo 3.1 (rc) |
vParquet4 or vParquet5 |
| Loki Simple Scalable Deployment deprecated |
Loki 3.6/3.7 docs |
HA monolithic or microservices; removal planned for Loki 4.0 |
| Promtail removed |
Loki 3.7.3 |
Grafana Alloy (alloy convert --source-format=promtail) |
| BoltDB, BoltDB-shipper, Cassandra, DynamoDB, BigTable, gRPC stores removed |
Loki 3.8 (unreleased, on main) |
TSDB index on object storage |
| Pyroscope default write path flipped to segment-writer |
Pyroscope 2.0 |
v1 via -write-path=ingester |
lgtm-distributed umbrella chart deprecated |
2026 |
Install per-component charts |
| Grafana Agent end of support |
End of 2025 |
Grafana Alloy |
Figures below come from Grafana Labs publications and community reports; they are not benchmarks run for this note. Treat them as orders of magnitude.
| Component |
Metric |
Figure |
Source / conditions |
| Mimir |
Active series |
1B+ in one cluster |
Grafana Labs load test (2022), microservices mode |
| Mimir |
MQE peak memory |
Up to 92% lower than the Prometheus engine |
Mimir 3.0 release blog (2025-11) |
| Mimir |
Storage efficiency |
About 1-2 bytes/sample |
Prometheus TSDB compression; varies by data |
| Loki |
Monolithic guideline |
About 20 GB/day |
Loki deployment-modes docs |
| Loki |
SSD guideline |
Up to about 1 TB/day |
Loki 3.7 docs (mode deprecated) |
| Loki |
Compression ratio |
Roughly 10:1 or better on chunks |
Depends on log format and codec |
| Tempo |
Monolithic ceiling |
25-35 MB/s or 55k-80k spans/s |
Tempo 3 deployment-modes docs |
| Tempo |
Recent-data window |
Live-stores serve roughly the last 30-60 min |
Tempo 3 migration docs |
| Pyroscope |
v2 ingest latency |
Median below 500 ms (synchronous ingest) |
Pyroscope v2 architecture docs |
The following figures replace a carried-over April 2026 table (re-checked 2026-09-27). Figures that no source supports were removed: a Loki "sub-second label-filtered query" claim, a Tempo "trace-ID lookup under 200-500 ms" claim, and a Salesforce "70M metrics/min" figure.
| Component |
Metric |
Figure |
Source / conditions |
| Mimir |
Ingestion rate |
About 50M samples/s at 1B active series (20 s scrape interval), on about 7,000 CPU cores and 30 TiB RAM |
Grafana Labs 1B-series load test blog; replaces an older "30M+ samples/s" figure |
| Loki |
Ingestion per cluster |
Official sizing tiers go up to about 30 TB/day (about 1 PB/month) per cluster; Grafana Cloud Logs ingests hundreds of TB/day |
Loki "Size the cluster" docs, Grafana Cloud Logs memcached blog |
| Tempo |
Ingestion |
The monolithic ceiling above (55k-80k spans/s) already equals about 4.7-6.9 billion spans/day; microservices scale past it with Kafka partitions |
Derived from the Tempo 3 deployment-modes guideline |
| Pyroscope |
Agent overhead |
Docs say "minimal CPU overhead" and publish no percentage |
Pyroscope introduction; an older "under 1% CPU" figure had no source |
| Component |
Guideline |
Source |
| Mimir ingesters |
1 core, 2.5 GB RAM and 5 GB disk per 300,000 in-memory series; in-memory series = active series x replication factor (so 1M active series at RF=3 is about 10 cores and 25 GB RAM) |
Mimir capacity planning |
| Mimir distributors |
1 core and 1 GB RAM per 25,000 samples/s |
Same |
| Mimir store-gateways |
13 GB disk per 1M active series (1-year retention, 15 s scrape, 2 bytes/sample) |
Same |
| Mimir compactor |
One instance (1 core, 4 GB, 300 GB disk) per 20M active series |
Same |
| Loki (microservices), below 3 TB/day |
Base requests total about 38 CPU and 59 GiB RAM (6 ingesters, 4 distributors, 10 queriers and others) |
Loki "Size the cluster" docs |
| Loki, 3-30 TB/day |
About 431 CPU and 857 GiB RAM in base requests |
Same |
End-to-End Latency (Typical)
Qualitative ranges that follow from the read path of each backend; no project publishes latency SLOs for self-hosted clusters.
| Signal |
Typical latency |
Notes |
| Metrics (recent) |
Seconds |
From ingesters (classic) or partition consumers (ingest storage) |
| Logs (recent) |
Seconds |
From ingesters |
| Traces (recent) |
Seconds |
From live-stores; Tempo 3.0 fails recent queries when a live-store lags (live_store.fail_on_high_lag: true) |
| Traces (by ID, historical) |
Sub-second to a few seconds |
Bloom filter + Parquet row-group lookup |
| Traces (TraceQL search, historical) |
Seconds to tens of seconds |
Depends on time range and selectivity |
Cost Comparison Estimates
Rough estimates for 1M active series + 100 GB/day logs + 50M spans/day (mid-2026). They vary greatly with cloud, retention and discounts. The self-hosted and SaaS ranges are illustrative figures carried over from the April 2026 note, not published by any vendor; the Grafana Cloud row was recomputed from list prices on 2026-09-27.
| Stack |
Estimated monthly cost |
Operational burden |
Vendor lock-in |
| Self-hosted LGTM |
$1,000-3,000 |
High (4+ backends, plus Kafka for Mimir/Tempo at scale) |
Low |
| Grafana Cloud Pro |
About $7,600+ at list price: about $6,400 for 1M billable series ($6.50 per 1,000) plus about $1,200-1,300 for ~2,950 GB of billable logs; traces extra. Discounts and Adaptive Metrics lower it (the earlier $1,500-4,000 range was below list price) |
Low (managed) |
Low-Medium |
| SigNoz (self-hosted) |
$500-1,500 |
Medium |
Low |
| Datadog |
$5,000-17,000 |
Very low (SaaS) |
High |
| New Relic |
$2,500-8,000 |
Very low (SaaS) |
Medium |
| ELK + Prometheus + Jaeger |
$2,000-6,000 |
Very high (3 stacks) |
Low |
Self-Hosted Cost by Scale (Estimates)
Illustrative infrastructure-only figures carried over from the April 2026 note; no vendor or independent source publishes them. For your own numbers, start from the sourced sizing inputs and your cloud's instance prices.
| Scale |
Metrics (active series) |
Logs (GB/day) |
Traces (spans/day) |
Est. monthly self-hosted |
| Small |
100k |
10 GB |
5M |
$200-500 |
| Medium |
1M |
100 GB |
50M |
$1,000-3,000 |
| Large |
10M |
1 TB |
500M |
$5,000-15,000 |
| Enterprise |
100M+ |
10 TB+ |
5B+ |
$20,000-100,000+ |
Grafana Cloud list prices (secondary sources, 2026): Free tier; Pro = $19/month platform fee + usage (for example $6.50 per 1k billable metric series, $0.40/GB log and trace writes); Enterprise by contract (commonly cited from $25k/year). Check the official pricing page before quoting.
| Component |
Metric |
What it tells you |
| All |
*_request_duration_seconds |
Internal API latency |
| Mimir ingesters |
cortex_ingester_memory_series |
In-memory series |
| Mimir ingest storage |
cortex_ingest_storage_reader_receive_and_consume_delay_seconds |
Distributor-to-ingester lag through Kafka |
| Mimir distributors |
cortex_distributor_received_samples_total |
Ingestion throughput |
| Loki distributors |
loki_distributor_bytes_received_total |
Ingestion throughput |
| Tempo distributors |
tempo_distributor_spans_received_total |
Span throughput |
| Tempo distributors |
tempo_distributor_produce_records_total |
Kafka produce rate (alert TempoDistributorKafkaProduceFailing) |
| Object storage clients |
*_objstore_bucket_operation_duration_seconds |
Storage latency |
Pre-built dashboards and alerts ship as mixins: grafana/mimir operations/mimir-mixin/, grafana/loki production/loki-mixin/, grafana/tempo operations/tempo-mixin/.
Network Exposure
| Component |
Default ports |
Recommended exposure |
| Mimir distributor / query-frontend |
8080 (HTTP), 9095 (gRPC) |
Internal only, behind an auth gateway |
| Loki distributor / query-frontend |
3100 (HTTP), 9095 (gRPC) |
Internal only, behind an auth gateway |
| Tempo distributor |
4317/4318 (OTLP), 3200 (HTTP), 9095 (gRPC) |
Internal only, behind an auth gateway |
| Kafka (Mimir/Tempo ingest) |
9092 |
Cluster-internal; SASL + TLS |
| Object storage |
443 (HTTPS) |
Private endpoint or VPC-only |
| Grafana |
3000 (HTTP) |
TLS-terminated reverse proxy |
Hardening Checklist
| Area |
Recommendation |
| Network isolation |
All backends in private subnets; Kubernetes NetworkPolicies so only collectors reach distributors and only Grafana reaches query-frontends |
| TLS everywhere |
mTLS between components where supported; TLS at the gateway edge |
| Auth gateway |
Centralise authentication; the gateway sets X-Scope-OrgID (never trust a client-supplied value) |
| Per-tenant limits |
Rate limits and series/stream/trace limits for every tenant |
| Bucket isolation |
Separate bucket per component per environment; bucket policy limited to that component's IAM role |
| Encryption at rest |
SSE-KMS with customer-managed keys |
| Kafka |
Enable SASL and TLS: Mimir -ingest-storage.kafka.sasl-mechanism / -ingest-storage.kafka.tls-enabled (PLAIN, OAUTHBEARER, MSK IAM flags exist); Tempo 3.1 adds sasl_mechanism (PLAIN, SCRAM-SHA-256/512, OAUTHBEARER, AWS_MSK_IAM) and tls_enabled |
| Secrets management |
Storage credentials and TLS keys in Vault or a cloud secret manager |
| Supply chain |
Tempo 3.1 images are cosign-signed with SLSA provenance; verify signatures in admission control |
| Audit logging |
Access logs on gateway and object storage |
Sources