Explanation¶

How Monoscope is built and why: component topology, the ingestion path, the TimeFusion storage engine, the LLM-driven query and agent features, deployment shapes, and the security model. Look-up values (ports, variables, schema, pricing) are in Reference. Tasks are in How-to Guides.

Read this first: storage is mid-migration

Marketing copy describes Monoscope as "all telemetry in your S3 bucket via TimeFusion". The code tells a more nuanced story (as of v0.6.27, 2026-09). Telemetry is dual-written to the legacy PostgreSQL/TimescaleDB table otel_logs_and_spans and to TimeFusion. TimeFusion reads and writes are feature-flagged (ENABLE_TIMEFUSION_READS, ENABLE_TIMEFUSION_WRITES). The upstream AGENTS.md states that the end state is TimeFusion only, and "Full migration to TimeFusion storage engine" is still an open roadmap item. The default docker-compose.yml starts only Monoscope and TimescaleDB, so a quickstart install keeps telemetry in Postgres, not S3.

Component Topology¶

The diagram shows the components of a full deployment and the flags that switch the optional parts on. Solid boxes run in every install. Kafka, Pub/Sub and TimeFusion are optional.

flowchart TB
    subgraph Clients["Instrumented clients"]
        SDK["OTel SDKs / agents<br/>monoscope-* SDK wrappers"]
        Col["OpenTelemetry Collector<br/>(optional)"]
        Web["@monoscopetech/browser<br/>RUM + rrweb replay"]
        CLI["monoscope CLI / MCP clients"]
    end

    subgraph Server["monoscope-server (Haskell, one binary)"]
        OTLP["Opentelemetry.OtlpServer<br/>gRPC :4317"]
        HTTP["Warp + Servant<br/>UI, REST, /api/v1/mcp :8080"]
        PM["ProcessMessage<br/>+ ExtractionWorker"]
        Jobs["odd-jobs BackgroundJobs<br/>monitors, reports, agents"]
        AI["Pkg.AI / LLM effect<br/>NL to KQL, issue analysis"]
    end

    subgraph Queue["Optional queue"]
        Kafka["Kafka topics<br/>+ dead-letter topic"]
        PubSub["Google Pub/Sub"]
    end

    subgraph Data["Storage"]
        PG["PostgreSQL + TimescaleDB<br/>metadata + legacy telemetry"]
        TF["TimeFusion<br/>pgwire :5432"]
        S3["S3 / MinIO / R2<br/>Delta Lake Parquet"]
    end

    LLM["OpenAI-compatible API"]

    SDK --> OTLP
    Col --> OTLP
    Web --> HTTP
    CLI --> HTTP
    OTLP --> PM
    Col -.->|"kafka exporter"| Kafka
    Kafka -.->|"ENABLE_KAFKA_SERVICE"| PM
    PubSub -.->|"ENABLE_PUBSUB_SERVICE"| PM
    PM --> PG
    PM -.->|"ENABLE_TIMEFUSION_WRITES"| TF
    TF --> S3
    HTTP --> PG
    HTTP -.->|"ENABLE_TIMEFUSION_READS"| TF
    Jobs --> PG
    Jobs --> AI
    HTTP --> AI
    AI --> LLM

Technology Breakdown¶

Component	Language	Main libraries	Purpose
Monoscope server (`app/`, `src/`)	Haskell (GHC 9.12.2)	effectful, Servant, Warp, Lucid, hasql, postgresql-simple, proto-lens, hw-kafka-client, odd-jobs	Ingestion, API, server-rendered UI, background jobs
Web components (`web-components/`)	TypeScript	Lit, Vite, ECharts 6, CodeMirror, Monaco, rrweb replay, elkjs	Log explorer, query editor, charts, service map, replay player
CLI (`cli/`)	Haskell	Shares the KQL grammar and wire types	Terminal and agent access. One static-ish binary
TimeFusion	Rust (edition 2024)	Apache DataFusion 54, delta-rs (fork), Arrow 58, pgwire (vendored), Foyer, Tantivy	S3-backed SQL store for telemetry
Metadata store	SQL	PostgreSQL + TimescaleDB (`timescale/timescaledb-ha:pg18-all` in compose)	Projects, users, monitors, issues, jobs, legacy telemetry

The frontend is server-rendered HTML (Lucid) with HTMX for partial updates, hyperscript for small behaviours, Tailwind CSS v4 + DaisyUI v5 for styling, and Lit web components where a rich client widget is needed.

Haskell Backend Internals¶

effectful algebraic effects. Capabilities such as DB, LLM, Notify, Time and UUID are effects in Data/Effectful/, each with a production and a test interpreter.
Servant route definitions (Web/Routes.hs) with Lucid HTML responses.
hasql / hasql-interpolate and postgresql-simple for PostgreSQL and TimeFusion access (TimeFusion is reached over the Postgres wire protocol through the same drivers).
Megaparsec KQL parser (Pkg/Parser), shared by the server and the CLI.
odd-jobs PostgreSQL-backed job queue for background work (monitors, reports, schema learning, digests).
hs-opentelemetry auto-instrumentation. Monoscope ingests its own traces ("dogfooding").
fourmolu and hlint for formatting and linting. Migrations are append-only, numbered SQL files in static/migrations/.

Ingestion Pipeline¶

OTLP over gRPC on port 4317 is the main path. The server also accepts browser SDK traffic over HTTP and can consume Kafka or Google Pub/Sub topics that an upstream collector fills. The sequence below shows both entry paths for one OTLP batch, with TimeFusion writes enabled.

sequenceDiagram
    participant App as "App (OTel SDK)"
    participant Coll as "OTel Collector"
    participant Grpc as "OtlpServer :4317"
    participant K as "Kafka or Pub/Sub topic"
    participant PM as "ProcessMessage + ExtractionWorker"
    participant PG as "TimescaleDB"
    participant TF as "TimeFusion"
    App->>Coll: OTLP spans, logs, metrics
    alt Direct gRPC (default)
        Coll->>Grpc: OTLP/gRPC with project API key
        Grpc->>PM: hand batch to the pipeline in-process
    else Queue-backed ingest
        Coll->>K: publish OTLP batch
        K->>PM: consume batch (consumer group)
    end
    PM->>PM: resolve project, normalise to otel_logs_and_spans rows
    PM->>PM: extract endpoints, schema, log patterns, error fingerprints
    PM->>PG: write legacy table (ENABLE_POSTGRES_TELEMETRY_WRITES)
    PM->>TF: INSERT over pgwire (ENABLE_TIMEFUSION_WRITES)
    TF-->>PM: ack after WAL append
    PM-->>K: commit offset, or route poison batch to the dead-letter topic

OTLP Ingestion¶

Signals: logs, traces and metrics. Logs and spans share one wide table (otel_logs_and_spans). Metrics have their own table, otel_metrics: one row per data point with typed value columns (value_double, value_int), histogram, exponential-histogram and summary arrays, and promoted resource/attribute columns, plus an otel_metrics_meta catalog. In Postgres it is a TimescaleDB hypertable with 1-hour chunks and a 30-day retention policy; the schema mirrors the TimeFusion table, and writes can target either or both (0108_otel_metrics.sql, checked 2026-09-28).
Authentication: the project API key travels either as an x-api-key gRPC header (Monoscope docs and Kubernetes guide) or as the OTel resource attribute at-project-key (vendor SDK READMEs). MCP and REST calls use Authorization: Bearer.
Queueing: gRPC batches are processed in-process. For queue-backed ingestion, a collector publishes to Kafka or Pub/Sub and Monoscope instances consume the topics. Ingest instances can run with CONSUMER_ONLY=True, so consumers and the extraction pipeline scale apart from the web tier. Since v0.6.24 a dead-letter consumer replays failed Kafka batches.
Extraction: the extraction worker derives the API catalog (endpoints), learns field schemas (v0.6.23), clusters log lines into patterns with the Drain algorithm (Pkg.Drain), and fingerprints errors.
Live Tail: matching events are streamed to the browser over SSE before the storage write. The transport is a live_tail Kafka topic when brokers are configured, or a Postgres relay table otherwise.

TimeFusion Storage Engine¶

TimeFusion (monoscope-tech/timefusion, MIT) is a separate Rust service that stores observability data as Delta Lake tables on S3-compatible storage and speaks the PostgreSQL wire protocol. Any Postgres client can read or write it. The diagram shows its internal write and read paths.

flowchart LR
    Client["Postgres client<br/>(Monoscope, psql)"] --> PGW["pgwire server<br/>(password auth)"]
    PGW --> DF["DataFusion planner<br/>+ executor"]
    subgraph Write["Write path"]
        WAL["Local WAL<br/>(single writer, fsync)"]
        Buf["In-memory buffer<br/>5-min time buckets"]
    end
    subgraph Durable["Object storage"]
        Delta["Delta Lake log<br/>+ Parquet (ZSTD)"]
        Tant["Tantivy index<br/>sidecars"]
    end
    Cache["Foyer cache<br/>memory L1 + disk L2"]
    DF -->|"INSERT"| WAL --> Buf
    Buf -->|"background flush"| Delta
    Buf --> Tant
    DF -->|"SELECT"| Buf
    DF -->|"SELECT"| Cache --> Delta

Key Properties¶

Property	Detail
Wire protocol	PostgreSQL-compatible (vendored `pgwire` + `datafusion-postgres`). `PGWIRE_PASSWORD` is required unless insecure auth is explicitly allowed
Query engine	Apache DataFusion (vectorised, predicate pushdown, partition pruning), plus a TimescaleDB-compatible `time_bucket`
Storage format	Delta Lake transaction log over Parquet files in your bucket (`TIMEFUSION_TABLE_PREFIX` path)
Write durability	`INSERT` is acknowledged after a WAL append (`sync_each` fsync by default). Buffered rows flush to Delta every 300 s by default
Read consistency	Reads union the in-memory buffer with Delta, so fresh rows are visible immediately
Updates and deletes	Merge-on-read: an update appends a new row version, `deleted` marks tombstones, dedup keeps the greatest `updated_at`
Compression	ZSTD tiers: level 3 hot, 9 warm, 19 cold (after 14 days)
Full-text	Tantivy indexes for fields marked `tantivy.indexed: true` in the schema YAML
Caching	Foyer: 1 GB memory + 500 GB disk by default, 7-day TTL
Throughput	Monoscope's README advertises "500K+ events/sec". TimeFusion's own README declines to publish a headline number and ships `bench/` scripts instead. Treat 500K/s as a vendor claim
Scale-out	One process per WAL directory (enforced with `flock`). No documented multi-writer mode

Stale claim corrected

Earlier versions of this page said TimeFusion had "no built-in auth", used "DynamoDB-based locking for multi-instance deployments", and defaulted to a 512 MB / 100 GB cache. The current TimeFusion README, DELTA_CONFIG.md and RUNBOOK.md (2026-09) show password-protected pgwire, a single-writer WAL, and 1 GB / 500 GB cache defaults.

Main Table Schema¶

The otel_logs_and_spans table stores logs and spans in one wide schema, partitioned by [project_id, date]. OTel attributes are flattened into typed columns with a triple-underscore separator (attributes___http___response___status_code), and the raw attributes, resource and body payloads are kept as a Variant type. The full column list is in Reference.

Schema evolution is file-based: table schemas are YAML files compiled into the TimeFusion binary. Adding a nullable column is safe (old Parquet files read as NULL). Adding a NOT NULL column breaks reads of existing files. Removed columns stay in old files until compaction rewrites them.

Data Model¶

Telemetry Storage (TimeFusion / S3)¶

The model is intentionally flat: one event row per log record or span, correlated by context___trace_id, parent_id and attributes___session___id. There are no separate trace, log or session tables for events, which keeps "show me everything around this error" to a single time-bounded scan filtered by project_id. Queries that omit project_id scan every tenant and are much slower.

Metadata Storage (PostgreSQL + TimescaleDB)¶

Projects and API keys: tenant boundary, encrypted keys (API_KEY_ENCRYPTION_SECRET_KEY), members and teams.
API catalog: endpoints, shapes and detected API changes.
Issues, error patterns and log patterns: fingerprinted errors, Drain patterns, incidents, investigations.
Monitors and dashboards: alert rules, dashboard definitions (also manageable as YAML through the CLI).
Jobs: odd-jobs queue for background work.
Legacy telemetry: the Postgres otel_logs_and_spans hypertable, still written by default.

Natural Language Query Engine¶

Monoscope's primary query language is KQL (a Kusto-style pipeline syntax with ==, has, summarize ... by). Natural language sits on top of it:

Prompt: the user types, for example, "Show me all errors in the payment service in the last hour".
Planning: an agentic query planner calls the configured LLM (OpenAI-compatible, OPENAI_BASE_URL, OPENAI_MODEL) with the project's learned schema and facets, and produces KQL.
Compilation: the KQL parser turns the query into SQL against Postgres or TimeFusion, always scoped to the project.
Rendering: results show as log lists, trace waterfalls or charts. The same flow is exposed as the MCP tool search_events_nl.

Because the LLM emits KQL rather than raw SQL, the model cannot address other tenants' partitions directly. Project scoping is added by the server during compilation.

AI Agent Scheduler¶

Scheduled AI agents and reports run as background jobs:

Intervals: hourly, daily or weekly, set per agent or report.
Inputs: aggregates, new or spiking error patterns, anomalies from spike detection, monitor states.
LLM step: summarises findings and proposes probable causes (analyze_issue exposes the same capability through MCP).
Outputs: email reports (weekly emails became denser "system reports" in v0.6.27) and alerts to configured channels.

Cost is driven by the LLM endpoint you configure. The server tracks token prices through AI_INPUT_*/AI_OUTPUT_* settings and has kill switches for LLM-based endpoint- and error-group review.

Error Fingerprinting¶

Error grouping combines deterministic and model-assisted steps:

Normalisation and fingerprinting (Pkg.ErrorFingerprint): stack traces and messages are normalised so variable parts do not split groups.
Pattern mining (Pkg.Drain): log lines are clustered into templates with the Drain algorithm.
Embedding-assisted merge (Pkg.PatternMerge): similar groups are candidates for merging. An "LLM judge" reviews candidates. Auto-apply for error-group merges is off by default (ENABLE_ERROR_GROUP_AUTO_APPLY), and so is promotion of merges to ingest-time masks.

Two further deterministic rules sit next to these steps (source checked 2026-09-28). Jaccard similarity is used, but on log patterns, not error groups: mergeByJaccard in Pkg.PatternMerge merges Drain templates whose non-placeholder token sets reach a 0.90 Jaccard score. Framework rollup is real for transport errors: isFrameworkTransportError in Pkg.ErrorFingerprint collapses an allowlist of connection-level errors (Warp InvalidRequest, Node ECONNRESET/ECONNREFUSED/EPIPE, Python BrokenPipeError, Java SocketException and others) into one issue across routes and stacks. Application errors such as Django Http404 are not in that list.

Session Replay¶

The browser SDK (@monoscopetech/browser) records DOM changes with rrweb, plus network timing, console errors and Web Vitals, and emits OTel spans.
Replay events are sent to Monoscope's ingest endpoint and processed from a queue topic (rrweb-client by default, RRWEB_TOPICS). A separate dead-letter topic quarantines bad payloads.
Frontend spans propagate trace context to backends that match propagateTraceHeaderCorsUrls, so a replay links to backend traces and logs through the shared trace and session ids.

Replay is gated by ENABLE_SESSION_REPLAY / ENABLE_REPLAY_SERVICE on self-hosted installs.

Deployment Topologies¶

Docker Compose (Development / Small Production)¶

The repository's docker-compose.yml runs two containers. Telemetry stays in TimescaleDB, and there is no Kafka, S3 or TimeFusion.

flowchart LR
    subgraph Host["Docker host (monoscope-network)"]
        M["monoscope-app<br/>ghcr.io/monoscope-tech/monoscope:latest<br/>:8080 HTTP, :4317 OTLP"]
        PG["monoscope-timescaledb<br/>timescaledb-ha:pg18-all<br/>:5432"]
        PGA["pgadmin :5050<br/>(profile dev)"]
    end
    M --> PG
    PGA -.-> PG

Self-Hosted Production¶

A larger install separates web, ingest and storage tiers using the documented flags. This layout is inferred from the configuration surface, not from an official reference architecture (no Helm chart is published).

flowchart TB
    LB["Load balancer / ingress<br/>TLS for :8080 and :4317"]
    subgraph Web["Web pods"]
        W1["monoscope-server<br/>UI, REST, MCP, jobs"]
    end
    subgraph Ingest["Ingest pods"]
        I1["monoscope-server<br/>OTLP gRPC"]
        C1["monoscope-server<br/>CONSUMER_ONLY=True"]
    end
    K["Kafka<br/>ingest, live_tail, DLQ topics"]
    PG["PostgreSQL + TimescaleDB<br/>(managed or HA)"]
    TF["TimeFusion<br/>single writer per WAL volume"]
    S3["S3 bucket<br/>Delta tables"]
    LB --> W1
    LB --> I1
    I1 --> PG
    I1 --> TF
    OC["OTel Collector<br/>kafka exporter"] --> K --> C1
    C1 --> PG
    C1 --> TF
    W1 --> PG
    W1 --> TF
    TF --> S3

Monoscope Cloud (SaaS)¶

The hosted service runs the same code. Plain Cloud keeps data for 30 days on Monoscope-managed storage. Cloud + S3 (BYOS) writes to a bucket you own and has unlimited retention. See Reference.

flowchart LR
    Apps["Your apps / collectors"] -->|"OTLP"| MC["Monoscope Cloud<br/>api.monoscope.tech"]
    MC --> MS["Monoscope-managed storage<br/>(Cloud plan, 30 days)"]
    MC -->|"Cloud + S3 plan"| US["Your S3 bucket<br/>(unlimited retention)"]

Security¶

Identity and Access¶

UI login: built-in basic auth (default admin/changeme in compose) or Auth0 SSO. The README lists "Auth & SSO: built-in" for Cloud and "DIY" for self-hosted.
Project permissions: members get view, edit or admin on a project (see Reference). Teams group members.
API keys: per project, stored encrypted, used for OTLP ingestion, the REST API, the CLI and MCP.

Multi-Tenancy¶

Projects are the tenant boundary. In Postgres, rows are scoped by project id. In TimeFusion, project_id is the first partition key. Shared tables serve most projects, and projects that need stronger isolation can get dedicated tables. The server adds project scoping to every compiled query. Organisation-level multi-tenant workspaces are still on the roadmap.

Data Protection¶

At rest: Parquet on S3 inherits the bucket's encryption (SSE-S3, SSE-KMS). Postgres relies on volume encryption. The TimeFusion WAL on local disk holds acknowledged but unflushed rows, so it needs the same protection.
In transit: TLS for client-to-ingest traffic is usually terminated at a load balancer. sslmode=require protects Postgres and TimeFusion connections. S3 calls use HTTPS unless AWS_ALLOW_HTTP is set.
Retention: Cloud plan data is kept 30 days. On BYOS and self-hosted, retention is your bucket's lifecycle policy. TimeFusion's VACUUM removes superseded files after 24 h, which limits Delta time travel to roughly that window.

Threat Model¶

Threat	Why it matters	Mitigation
Leaked project API key	Grants ingestion and API or MCP read access for that project	Rotate keys, scope keys per environment, monitor API usage
Public or over-permissive bucket	Full-fidelity telemetry often contains PII, tokens and request bodies	Block Public Access, bucket policy limited to the TimeFusion role, SSE-KMS, access logs
Direct TimeFusion or Postgres access	Bypasses project permissions in the UI	Network isolation. Strong `PGWIRE_PASSWORD`. Never enable insecure auth in production
Default credentials	Compose ships `admin`/`changeme` and a sample encryption key	Replace before exposing the instance
Prompt injection through NL queries or log content	The LLM sees user prompts and telemetry excerpts	The LLM emits KQL, not SQL, and the server enforces project scoping. Keep MCP keys least-privilege
Telemetry sent to a third-party LLM	Agents and NL search send schema, facets and event excerpts to `OPENAI_BASE_URL`	Point at a self-hosted OpenAI-compatible endpoint, or leave `OPENAI_API_KEY` unset

Setup steps for these controls are in How-to Guides, and the checklist is in Reference.

Design Trade-offs¶

Haskell core: strong typing (effect rows, NonEmpty, phantom types) and a single server binary, at the cost of a small contributor pool. The team offsets this with heavy self-instrumentation: Monoscope ingests its own traces and logs.
Postgres first, TimeFusion second: shipping on TimescaleDB gave a working product early. Dual-write lets TimeFusion reach feature parity without a flag day. The cost is double write load and two sources of truth until the migration completes.
Single-writer WAL: simpler correctness (no distributed lock or consensus) but no horizontal write scaling for one TimeFusion instance. Scale comes from larger nodes or more tables.
Wide-event table: one table for logs and spans matches the observability-2.0 "wide events" idea and makes cross-signal correlation cheap. The cost is a very wide schema that must be managed with YAML definitions and Variant columns.
LLM as a planner, not an executor: generating KQL keeps tenancy enforcement in deterministic code, and a user can inspect and edit the query.