Skip to content

Personal Knowledge Base

LGTM Stack Architecture

Architecture¶

Deployment Modes¶

All LGTM backends (Mimir, Loki, Tempo) share the same deployment mode philosophy: a single binary with a -target flag that selects which component to run. This gives three deployment tiers:

Mode	Description	Use Case	Ops Complexity
Monolithic	All components in one process	Dev, testing, PoC, small prod	Low
Simple Scalable (SSD)	Read/Write/Backend targets as separate services	Mid-sized production	Medium
Microservices	Each component as independent pod	Large-scale, hyperscale production	High

Monolithic Mode¶

flowchart LR
    App["Applications"] --> Alloy["Alloy"]
    Alloy --> M["Mimir<br/>(all-in-one)"]
    Alloy --> L["Loki<br/>(all-in-one)"]
    Alloy --> T["Tempo<br/>(all-in-one)"]
    M --> S3["Object Storage"]
    L --> S3
    T --> S3
    G["Grafana"] -.-> M
    G -.-> L
    G -.-> T

    style M fill:#7b42bc,color:#fff
    style L fill:#2a7de1,color:#fff
    style T fill:#e65100,color:#fff
    style G fill:#ff6600,color:#fff

Simple Scalable Deployment¶

flowchart TB
    subgraph MimirSSD["Mimir (Simple Scalable)"]
        MW["Write Path<br/>(distributor + ingester)"]
        MR["Read Path<br/>(query-frontend + querier)"]
        MB["Backend<br/>(compactor + store-gateway)"]
    end

    subgraph LokiSSD["Loki (Simple Scalable)"]
        LW["Write Path<br/>(distributor + ingester)"]
        LR["Read Path<br/>(query-frontend + querier)"]
        LB["Backend<br/>(compactor + index-gateway)"]
    end

    subgraph TempoSSD["Tempo (Simple Scalable)"]
        TW["Write Path<br/>(distributor + ingester)"]
        TR["Read Path<br/>(query-frontend + querier)"]
        TB2["Backend<br/>(compactor)"]
    end

    S3["Object Storage"]
    MW --> S3
    LW --> S3
    TW --> S3
    MR --> S3
    LR --> S3
    TR --> S3

    style MimirSSD fill:#7b42bc,color:#fff
    style LokiSSD fill:#2a7de1,color:#fff
    style TempoSSD fill:#e65100,color:#fff

Microservices Mode (Production)¶

flowchart TB
    subgraph MimirMS["Mimir Microservices"]
        MD["Distributor"]
        MI["Ingester ×3"]
        MQF["Query Frontend"]
        MQ["Querier ×2"]
        MSG["Store-Gateway ×2"]
        MC["Compactor"]
    end

    subgraph LokiMS["Loki Microservices"]
        LD["Distributor"]
        LI["Ingester ×3"]
        LQF["Query Frontend"]
        LQ["Querier ×2"]
        LIG["Index Gateway"]
        LC["Compactor"]
    end

    subgraph TempoMS["Tempo Microservices"]
        TD["Distributor"]
        TI["Ingester ×3"]
        TQF["Query Frontend"]
        TQ["Querier ×2"]
        TMG["Metrics Generator"]
        TC["Compactor"]
    end

    S3["Object Storage<br/>(3 separate buckets)"]

    MI --> S3
    MSG --> S3
    MC --> S3
    LI --> S3
    LIG --> S3
    LC --> S3
    TI --> S3
    TC --> S3
    TMG -->|"remote_write"| MD

    style MimirMS fill:#7b42bc,color:#fff
    style LokiMS fill:#2a7de1,color:#fff
    style TempoMS fill:#e65100,color:#fff
    style S3 fill:#0d1117,color:#fff

Full Production Topology¶

flowchart TB
    subgraph Apps["Applications & Infrastructure"]
        App["Services<br/>(OTel SDK)"]
        K8s["Kubernetes<br/>Nodes"]
    end

    subgraph Collection["Collection Layer"]
        Alloy["Grafana Alloy<br/>(DaemonSet)"]
    end

    subgraph Backends["LGTM Backends (Microservices)"]
        Mimir["Mimir<br/>📊 Metrics"]
        Loki["Loki<br/>📝 Logs"]
        Tempo["Tempo<br/>🔍 Traces"]
        Pyro["Pyroscope<br/>🔥 Profiles"]
    end

    subgraph Storage["Object Storage"]
        B1["mimir-blocks (S3)"]
        B2["loki-chunks (S3)"]
        B3["tempo-traces (S3)"]
        B4["pyroscope-data (S3)"]
    end

    subgraph Grafana["Grafana (HA)"]
        G1["Pod 1"]
        G2["Pod 2"]
        G3["Pod 3"]
    end

    subgraph Support["Supporting Services"]
        PG["PostgreSQL<br/>(Grafana DB)"]
        Redis["Redis<br/>(Sessions)"]
        LB["Ingress / LB"]
        MC["Memcached<br/>(Query Cache)"]
    end

    Apps --> Alloy
    K8s --> Alloy
    Alloy -->|remote_write| Mimir
    Alloy -->|push| Loki
    Alloy -->|OTLP| Tempo
    Alloy -->|push| Pyro

    Mimir --> B1
    Loki --> B2
    Tempo --> B3
    Pyro --> B4

    LB --> G1 & G2 & G3
    G1 --> PG
    G1 --> Redis
    Mimir --> MC
    Loki --> MC

    Grafana -.-> Mimir
    Grafana -.-> Loki
    Grafana -.-> Tempo
    Grafana -.-> Pyro

    style Apps fill:#0d7377,color:#fff
    style Collection fill:#ff6600,color:#fff
    style Backends fill:#2a2d3e,color:#fff
    style Storage fill:#0d1117,color:#fff
    style Grafana fill:#ff6600,color:#fff
    style Support fill:#1a1d2e,color:#fff

Object Storage Layout¶

Critical rule: Each LGTM component must use separate buckets (or at minimum, separate prefixes within a bucket). Never share the same path.

Component	Recommended Bucket	Contents
Mimir (blocks)	`observability-mimir-blocks`	TSDB blocks (2h intervals)
Mimir (ruler)	`observability-mimir-ruler`	Recording and alerting rules
Mimir (alertmanager)	`observability-mimir-alertmanager`	Alertmanager state
Loki (chunks + index)	`observability-loki-chunks`	Compressed log chunks + TSDB index
Tempo (traces)	`observability-tempo-traces`	Parquet trace blocks + bloom filters
Pyroscope (profiles)	`observability-pyroscope-data`	Profile blocks

Kubernetes Deployment Matrix¶

Component	Helm Chart	Pods (Min HA)	Scaling Dimension	Key Resource
Grafana	`grafana/grafana`	2–3	HPA (CPU/mem)	Memory
Mimir	`grafana/mimir-distributed`	7+ (dist×1, ing×3, q×2, sg×1)	Per-component HPA	Memory (ingesters), CPU (queriers)
Loki	`grafana/loki`	6+ (dist×1, ing×3, q×2)	Per-component HPA	Memory (ingesters), CPU (queriers)
Tempo	`grafana/tempo-distributed`	5+ (dist×1, ing×3, q×1)	Per-component HPA	Memory (ingesters)
Pyroscope	`grafana/pyroscope`	1–3	Replicas	Memory
Alloy	`grafana/alloy`	1 per node (DaemonSet)	DaemonSet	CPU, Memory
PostgreSQL	External managed	HA pair	Managed service	Disk IOPS
Redis	External managed	HA pair	Managed service	Memory
Memcached	`bitnami/memcached`	2–3	Replicas	Memory

Shared Infrastructure Patterns¶

Hash Rings¶

Mimir, Loki, and Tempo all use consistent hash rings for sharding data across ingesters. The ring is backed by: - memberlist (default, gossip-based, no external dependency) - Consul or etcd (for environments that already run them)

Caching¶

Cache Layer	Purpose	Technology
Query results cache	Cache query responses	Memcached (recommended) or Redis
Chunks cache	Cache data chunks from object storage	Memcached
Index cache	Cache index lookups	Memcached
Metadata cache	Cache block metadata	Memcached

Memcached is strongly recommended over Redis for cache layers due to lower latency and simpler scaling.