How-to Guides¶

Task recipes for running Monoscope: deploy it, point telemetry at it, add TimeFusion/S3 storage, operate it day to day, and use the CLI and MCP server. Variable defaults, ports and schema are in Reference. Background is in Explanation.

Version-specific

Commands were checked against the upstream repositories on 2026-09-25 (Monoscope v0.6.27). Monoscope is pre-1.0. Pin image tags in production and re-check docs/ in the repository after upgrades.

Deployment¶

Docker Compose (Quick Start)¶

This starts Monoscope and TimescaleDB with the repository's docker-compose.yml. It needs Docker with about 4 GB of free RAM and ports 8080, 4317 and 5432 free.

git clone https://github.com/monoscope-tech/monoscope.git
cd monoscope
docker-compose up -d
docker-compose ps
# UI: http://localhost:8080  (basic auth admin / changeme)

Then create a project in the UI and copy its API key. To send test data, install the CLI (see CLI recipes) and run:

monoscope auth login --token <PROJECT_API_KEY>
monoscope send-event -m "Hello from Monoscope"
monoscope telemetrygen --kind=trace --rate=5 --count=50

Quickstart storage

The compose stack keeps telemetry in TimescaleDB. It does not start TimeFusion, S3/MinIO or Kafka. To use object storage, follow Add TimeFusion and S3 Storage.

Docker Compose (Production)¶

Keep the upstream compose file and put secrets and overrides in .env and docker-compose.override.yml, so upgrades stay a git pull.

cp .env.example .env
# Generate real secrets
echo "API_KEY_ENCRYPTION_SECRET_KEY=$(openssl rand -hex 32)" >> .env

# docker-compose.override.yml
services:
  monoscope:
    image: ghcr.io/monoscope-tech/monoscope:latest   # pin a digest in production
    env_file: .env
    environment:
      - ENVIRONMENT=PROD
      - HOST_URL=https://monoscope.example.com
      - DATABASE_URL=host=timescaledb user=monoscope password=${PG_PASSWORD} dbname=monoscope port=5432 sslmode=require
      # SSO instead of basic auth
      - BASIC_AUTH_ENABLED=False
      - AUTH0_DOMAIN=${AUTH0_DOMAIN}
      - AUTH0_CLIENT_ID=${AUTH0_CLIENT_ID}
      - AUTH0_SECRET=${AUTH0_SECRET}
      - AUTH0_CALLBACK=https://monoscope.example.com/auth_callback
      - AUTH0_LOGOUT_REDIRECT=https://monoscope.example.com
      # Email for alerts and reports
      - SMTP_HOST=${SMTP_HOST}
      - SMTP_PORT=587
      - SMTP_USERNAME=${SMTP_USERNAME}
      - SMTP_PASSWORD=${SMTP_PASSWORD}
      - SMTP_SENDER=alerts@example.com
    deploy:
      resources:
        limits:
          memory: 4G

To use an existing PostgreSQL server instead, enable the TimescaleDB extension on it, point DATABASE_URL at it, and remove the timescaledb service.

Add TimeFusion and S3 Storage¶

TimeFusion stores telemetry as Delta Lake tables in your bucket. Run it next to Monoscope and turn on the TimeFusion flags.

# docker-compose.override.yml (additions)
services:
  timefusion:
    image: ghcr.io/monoscope-tech/timefusion:<git-short-sha>   # images are tagged per commit
    ports:
      - "5433:5432"            # avoid clashing with TimescaleDB on the host
    environment:
      - AWS_S3_BUCKET=my-telemetry-bucket
      - AWS_DEFAULT_REGION=eu-central-1
      - AWS_ACCESS_KEY_ID=${TF_AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${TF_AWS_SECRET_ACCESS_KEY}
      - PGWIRE_PASSWORD=${TF_PGWIRE_PASSWORD}
      - TIMEFUSION_DATA_DIR=/data
    volumes:
      - timefusion_data:/data  # WAL + cache: keep on durable disk
  monoscope:
    environment:
      - TIMEFUSION_PG_URL=host=timefusion user=postgres password=${TF_PGWIRE_PASSWORD} dbname=postgres port=5432
      - ENABLE_TIMEFUSION_WRITES=True
      - ENABLE_TIMEFUSION_READS=True
volumes:
  timefusion_data:

Feature-flagged migration

Monoscope still dual-writes to Postgres by default (ENABLE_POSTGRES_TELEMETRY_WRITES=True). Keep it on until you have verified that TimeFusion reads cover the features you use. The upstream roadmap lists the full move to TimeFusion as unfinished. The TIMEFUSION_PG_URL format above mirrors DATABASE_URL. The upstream docs do not show a worked example, so confirm it against src/System/Config.hs.

For MinIO or Cloudflare R2, also set AWS_S3_ENDPOINT (and AWS_ALLOW_HTTP=true for plain-HTTP MinIO).

Kubernetes¶

No Helm chart is published. docs/kubernetes.md in the repository gives plain manifests. The steps are:

kubectl create namespace monoscope

kubectl create secret generic monoscope-secrets \
  --namespace=monoscope \
  --from-literal=database-url='postgresql://monoscope:CHANGE_ME@postgres:5432/monoscope?sslmode=require' \
  --from-literal=api-key-secret="$(openssl rand -hex 32)"

# Deployment + Service from docs/kubernetes.md:
#   image ghcr.io/monoscope-tech/monoscope:latest, ports 8080 (http) and 4317 (grpc),
#   env DATABASE_URL and API_KEY_ENCRYPTION_SECRET_KEY from the secret,
#   requests 1 CPU / 2Gi, limits 2 CPU / 4Gi
kubectl apply -f monoscope-deployment.yaml
kubectl -n monoscope get pods,svc

For TimescaleDB, use a managed service or an operator-managed cluster rather than the single-replica Deployment sketched in the upstream guide. If you run TimeFusion in-cluster, give it a StatefulSet with one replica and a PersistentVolume for TIMEFUSION_DATA_DIR. Use a TCP readiness probe on the pgwire port, as the TimeFusion runbook recommends.

Kubernetes Auto-Instrumentation with the OTel Operator¶

Monoscope exposes OTLP only over gRPC (4317), while the operator's Java, Node.js and Python agents default to OTLP/HTTP. Route them through an OpenTelemetry Collector that receives both protocols and forwards over gRPC with the API key.

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: monoscope
  namespace: default
spec:
  exporter:
    endpoint: http://otel-collector.monoscope.svc.cluster.local:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_always_on
EOF

# The operator reads the annotation from the pod template, not the Deployment object
kubectl patch deployment my-java-app -p \
  '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"monoscope"}}}}}'

Use inject-python or inject-nodejs for other runtimes. The collector config is in OpenTelemetry Collector Configuration.

Configuration¶

The full variable tables are in Reference: Monoscope Environment Variables and Reference: TimeFusion Configuration.

Enable LLM Features¶

Natural-language search, AI agents and issue analysis need an OpenAI-compatible endpoint:

# .env
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://api.openai.com      # or a self-hosted OpenAI-compatible gateway
# OPENAI_MODEL / OPENAI_SMALL_MODEL override the built-in model defaults

Leave OPENAI_API_KEY empty to keep telemetry away from any LLM. KQL search, dashboards and monitors still work.

Enable Email Reports and Alerts¶

# .env: SMTP (or SENDGRIDAPIKEY / POSTMARK_TOKEN)
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USERNAME=alerts
SMTP_PASSWORD=...
SMTP_SENDER=alerts@example.com
ENABLE_BACKGROUND_JOBS=True
ENABLE_DAILY_JOB_SCHEDULING=True

Then choose daily or weekly reports and their recipients in the project settings.

Scaling¶

Component	How to scale
Monoscope web tier	Run more replicas behind a load balancer. Sessions and state live in Postgres
Ingestion	Put Kafka (or Pub/Sub) in front and add `CONSUMER_ONLY=True` instances. Tune `KAFKA_GROUP_CONCURRENCY` and `MESSAGES_PER_PUBSUB_PULL_BATCH`
Background jobs	Raise `MAX_CONCURRENT_JOBS` in line with CPU cores
TimescaleDB	Vertical scaling, read replicas, connection pooling
TimeFusion	Scale vertically (buffer, cache, `TIMEFUSION_MEMORY_LIMIT_GB`). One process per WAL directory, so no multi-writer scale-out
S3	Effectively unlimited. Background compaction keeps file counts down. ZSTD tiers reduce size over time

Monitoring¶

Health Checks¶

# Monoscope HTTP (endpoint from docs/getting-started.md)
curl -sf http://localhost:8080/api/v1/health -H "X-API-Key: $MONOSCOPE_API_KEY"

# OTLP port reachable
nc -zv localhost 4317

# TimeFusion over pgwire (password required)
PGPASSWORD="$TF_PGWIRE_PASSWORD" psql -h localhost -p 5433 -U postgres -d postgres -c "SELECT 1"

# Container logs
docker-compose logs -f monoscope

Key Metrics to Watch¶

Monoscope ingests its own telemetry, so you can build these views in Monoscope itself. The thresholds are starting points chosen by the author, not vendor guidance.

Signal	Starting alert threshold	Why
OTLP ingestion rate	Drop over 50% against the previous hour	Pipeline or client breakage
Kafka consumer lag (if used)	Growing for 10 minutes or more	Consumers under-provisioned
Kafka dead-letter topic size	Any sustained growth	Poison batches or schema errors
TimeFusion `oldest_bucket_age_seconds`	Keeps growing	Flush stuck (runbook incident)
TimeFusion WAL disk and cache disk	Over 80%	WAL loss risk, cache thrash
Postgres connections	Over 80% of `max_connections` (compose sets 200)	Pool saturation

Upgrades¶

cd monoscope
git pull                      # refresh compose and docs
docker-compose pull           # fetch the new image
docker-compose up -d
docker-compose logs monoscope | grep -i migrat

Migrations run at startup when MIGRATE_AND_INITIALIZE_ON_START=True. They are append-only numbered SQL files, so do not edit or rename files in static/migrations/. Read the release notes on the releases page before upgrading. There is no formal compatibility policy for 0.x releases.

For TimeFusion, deploy stop-first with a single replica so the old process drains, flushes and releases the WAL lock before the new one starts (see the TimeFusion runbook).

Backup & Recovery¶

S3 Data¶

Delta tables on S3 are the authoritative store. Enable bucket versioning and lifecycle rules. Delta time travel plus versioning gives point-in-time recovery without a backup job.
Tantivy indexes and the Foyer cache are derived data and rebuild on their own.

TimeFusion WAL¶

Acknowledged rows live only in the local WAL until the next flush (default every 300 s). Put TIMEFUSION_DATA_DIR on durable storage if losing a host is a risk you cannot accept. TimeFusion does not upload WAL segments to S3.
To recover, point a new instance at the same bucket and TIMEFUSION_TABLE_PREFIX. If the old WAL directory survived, copy it to ${TIMEFUSION_DATA_DIR}/wal/ before startup. Replay is idempotent.

PostgreSQL Metadata¶

docker-compose exec timescaledb pg_dump -U postgres -d monoscope -Fc > monoscope-metadata-$(date +%F).dump
# restore
docker-compose exec -T timescaledb pg_restore -U postgres -d monoscope --clean < monoscope-metadata-2026-09-25.dump

With the default settings Postgres also holds telemetry, so size the dump and the restore window accordingly.

Recovery¶

Restore PostgreSQL from the latest dump.
Start TimeFusion against the same bucket, with the old WAL directory if you have it.
Start Monoscope. Kafka consumers resume from their committed offsets, so events still within topic retention are re-processed.

Secure a Self-Hosted Deployment¶

The checklist version is in Reference. The main steps are:

Replace defaults: new BASIC_AUTH_PASSWORD (or Auth0 with basic auth disabled), a random API_KEY_ENCRYPTION_SECRET_KEY, and a strong PGWIRE_PASSWORD.
Terminate TLS for 8080 and 4317 at an ingress or load balancer (NGINX, Envoy, or a cloud LB with gRPC support).

Isolate the data tier. In Kubernetes, allow only Monoscope pods to reach TimeFusion:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: timefusion-access
  namespace: monoscope
spec:
  podSelector:
    matchLabels:
      app: timefusion
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: monoscope
      ports:
        - protocol: TCP
          port: 5432

Harden the bucket: Block Public Access, SSE-KMS with a customer-managed key, versioning, access logs, and a bucket policy limited to the TimeFusion IAM role.
Strip sensitive attributes before they leave the cluster, with the Collector's attributes or transform processors, or with SDK redaction options (RedactHeaders, RedactRequestBody).

Common Issues¶

Issue	Cause	Resolution
OTLP connection refused	Wrong port, or an OTLP/HTTP exporter pointed at 4317	Use gRPC to 4317. Put a collector in front for HTTP clients
Data sent but project stays empty	Missing or wrong API key	Send `x-api-key` as a gRPC header, or `at-project-key` as a resource attribute
Login fails on a fresh install	Credentials changed in `.env` but container not recreated	`docker-compose up -d --force-recreate monoscope`
Port 5432 already in use	TimescaleDB and TimeFusion (or a local Postgres) collide	Remap one host port (`5433:5432`)
Natural-language search does nothing	No LLM configured	Set `OPENAI_API_KEY` (and `OPENAI_BASE_URL` for non-OpenAI gateways)
New TimeFusion pod waits forever at start	Old process still holds the WAL lock	Stop the old process. Never delete `wal.lock`
TimeFusion refuses to start	`PGWIRE_PASSWORD` unset	Set it (or `TIMEFUSION_ALLOW_INSECURE_AUTH=true` for local development only)
Slow TimeFusion queries	Missing `project_id` filter, cold cache	Always filter on `project_id`. Size `TIMEFUSION_FOYER_*` to the working set

Commands & Recipes¶

Docker Commands¶

# Start / stop / logs
docker-compose up -d
docker-compose down
docker-compose logs -f monoscope

# Optional profiles from the upstream compose file
docker-compose --profile dev up -d      # adds pgAdmin on :5050
docker-compose --profile test up -d     # ephemeral TimescaleDB on :5433

Run TimeFusion Standalone¶

# Zero-config local trial with MinIO (builds from source)
git clone https://github.com/monoscope-tech/timefusion.git
cd timefusion
docker compose up

# Against your own bucket (pick a tag from the GHCR packages page)
docker run -d --name timefusion -p 5432:5432 \
  -e AWS_S3_BUCKET=your-bucket \
  -e AWS_ACCESS_KEY_ID=your-key \
  -e AWS_SECRET_ACCESS_KEY=your-secret \
  -e PGWIRE_PASSWORD=change-me \
  ghcr.io/monoscope-tech/timefusion:<tag>

psql "postgresql://postgres:change-me@localhost:5432/postgres"

OpenTelemetry Collector Configuration¶

Basic OTLP Export to Monoscope¶

The collector receives gRPC and HTTP from applications and exports gRPC to Monoscope with the project key.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s

exporters:
  otlp:
    endpoint: monoscope.monoscope.svc.cluster.local:4317
    tls:
      insecure: true            # in-cluster plaintext. Use TLS across networks
    headers:
      x-api-key: ${env:MONOSCOPE_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Multi-Source Collector¶

This adds Prometheus scraping and Kubernetes metadata, following the pattern in the upstream docs/kubernetes.md.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  prometheus:
    config:
      scrape_configs:
        - job_name: my-app
          scrape_interval: 15s
          static_configs:
            - targets: ["my-app:8080"]
  kubeletstats:
    collection_interval: 30s
    auth_type: serviceAccount
    endpoint: "https://${env:K8S_NODE_NAME}:10250"
    insecure_skip_verify: true

processors:
  batch: {}
  k8sattributes:
    auth_type: serviceAccount
  resource:
    attributes:
      - key: k8s.cluster.name     # no receiver can discover the cluster name
        value: my-cluster
        action: upsert

exporters:
  otlp:
    endpoint: monoscope.monoscope.svc.cluster.local:4317
    tls:
      insecure: true
    headers:
      x-api-key: ${env:MONOSCOPE_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, k8sattributes, resource]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch, k8sattributes, resource]
      exporters: [otlp]
    metrics:
      receivers: [otlp, prometheus, kubeletstats]
      processors: [batch, k8sattributes, resource]
      exporters: [otlp]

For Monoscope Cloud, the vendor SDK READMEs send to otelcol.apitoolkit.io:4317 over gRPC and carry the key as the resource attribute at-project-key=<API_KEY>. Check the Cloud onboarding screen for the current endpoint.

SDK Setup¶

Zero-Code OpenTelemetry (Any Language)¶

# Python
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=my-app \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
OTEL_EXPORTER_OTLP_HEADERS="x-api-key=YOUR_API_KEY" \
opentelemetry-instrument python myapp.py

# Node.js
npm install --save @opentelemetry/auto-instrumentations-node
OTEL_SERVICE_NAME=my-app \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
OTEL_EXPORTER_OTLP_HEADERS="x-api-key=YOUR_API_KEY" \
node --require @opentelemetry/auto-instrumentations-node/register app.js

Java (Spring Boot)¶

curl -L -o otel-agent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
java -javaagent:otel-agent.jar \
  -Dotel.service.name=my-service \
  -Dotel.exporter.otlp.protocol=grpc \
  -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
  -Dotel.exporter.otlp.headers=x-api-key=YOUR_API_KEY \
  -jar my-app.jar

Python (Flask)¶

The monoscope-flask package adds request and response body capture on top of OTel.

pip install monoscope-flask opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
export OTEL_SERVICE_NAME=my-service
export OTEL_RESOURCE_ATTRIBUTES="at-project-key=YOUR_API_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
opentelemetry-instrument flask run --app app

from flask import Flask
from monoscope_flask import Monoscope

app = Flask(__name__)
monoscope = Monoscope(
    service_name="my-service",
    capture_request_body=True,
    capture_response_body=True,
)

@app.before_request
def before_request():
    monoscope.beforeRequest()

@app.after_request
def after_request(response):
    monoscope.afterRequest(response)
    return response

Django and FastAPI use monoscope-django and monoscope-fastapi. See the monoscope-python repository.

Node.js (Express)¶

npm install --save @monoscopetech/express @opentelemetry/api @opentelemetry/auto-instrumentations-node

import "@opentelemetry/auto-instrumentations-node/register";
import express from "express";
import { Monoscope } from "@monoscopetech/express";

const app = express();
const monoscopeClient = Monoscope.NewClient({ serviceName: "my-service" });

app.use(monoscopeClient.middleware);
app.get("/api/users", (req, res) => res.json({ users: [] }));
app.use(monoscopeClient.errorMiddleware);   // report unhandled errors with request context

app.listen(3000);

Run it with OTEL_EXPORTER_OTLP_PROTOCOL=grpc, OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES=at-project-key=<API_KEY> set.

Go¶

package main

import (
    "log"
    "net/http"

    monoscope "github.com/monoscope-tech/monoscope-go/native"
)

func main() {
    shutdown, err := monoscope.ConfigureOpenTelemetry()
    if err != nil {
        log.Printf("error configuring OpenTelemetry: %v", err)
    }
    defer shutdown()

    mw := monoscope.Middleware(monoscope.Config{
        ServiceName:       "my-service",
        RedactHeaders:     []string{"Authorization", "X-Api-Key"},
        RedactRequestBody: []string{"password", "credit_card"},
    })
    http.Handle("/", mw(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte(`{"users":[]}`))
    })))
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Set OTEL_RESOURCE_ATTRIBUTES=at-project-key=<API_KEY> and OTEL_SERVICE_NAME in the environment. Gin, Echo, Fiber, Chi and Gorilla have their own sub-packages.

Browser (RUM and Session Replay)¶

import Monoscope from "@monoscopetech/browser";

const monoscope = new Monoscope({
  apiKey: "YOUR_API_KEY",
  serviceName: "web-frontend",
  propagateTraceHeaderCorsUrls: [/api\.example\.com/],   // link frontend spans to backend traces
});
monoscope.setUser({ id: "user-123" });

CLI Recipes¶

# Install / upgrade (Linux, macOS)
curl https://monoscope.tech/install.sh | sh

# Point at a self-hosted instance
monoscope config set api_url https://monoscope.example.com
monoscope config set project <project-uuid>
monoscope auth login --token "$MONOSCOPE_API_KEY"

# Search (KQL). Bare strings become full-text search
monoscope logs search --service checkout-api --level error --since 30m
monoscope events search 'attributes.http.response.status_code >= 500' --since 1h

# Aggregate, with a CI gate
monoscope metrics query 'summarize percentile(duration, 99) by resource.service.name' --since 30m
monoscope metrics query 'summarize count()' --since 30m --assert '< 1000'

# Live tail and incident context
monoscope logs tail --service payment-api --level error
monoscope events context --window 5m --summary --at 2026-09-25T10:00:00Z

# Monitors and dashboards as code (YAML round-trip)
monoscope dashboards yaml <id> > dashboard.yaml
monoscope dashboards apply dashboards/
monoscope monitors apply monitors/
monoscope monitors mute <id> --for 30

For agents and scripts, set MONOSCOPE_AGENT_MODE=1 (or pass --agent) to get stable JSON envelopes and no prompts. The Claude Code skills plugin wraps these commands:

claude plugin marketplace add monoscope-tech/skills
claude plugin install monoscope-skills@monoscope-skills

Connect an MCP Client¶

{
  "mcpServers": {
    "monoscope": {
      "url": "https://api.monoscope.tech/api/v1/mcp",
      "headers": { "Authorization": "Bearer YOUR_API_KEY" }
    }
  }
}

For self-hosted instances, change the host and keep the /api/v1/mcp path. Smoke test:

curl -s https://api.monoscope.tech/api/v1/mcp \
  -H "Authorization: Bearer $MONOSCOPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | jq '.result.tools[].name'

Alert Channel Configuration¶

Slack¶

Create a Slack app and set SLACK_CLIENT_ID, SLACK_CLIENT_SECRET and SLACK_REDIRECT_URI=https://<host>/slack/oauth/callback/ (self-hosted).
In the project settings, connect Slack and pick the channel.

Discord¶

Set DISCORD_CLIENT_ID, DISCORD_CLIENT_SECRET and DISCORD_BOT_TOKEN (or DISCORD_WEBHOOK_URL), then connect it in the project settings.

PagerDuty¶

PagerDuty needs no environment variables. Create an Events API v2 integration on the PagerDuty service, then paste its integration key in the project's integrations settings (or add it to a team's PagerDuty services). The server posts trigger and resolve events to https://events.pagerduty.com/v2/enqueue, so allow outbound HTTPS to that host. Use the notification test in project settings to confirm delivery. Source: Pages/Projects.hs and Data/Effectful/Notify.hs (checked 2026-09-28).

Querying TimeFusion¶

Always include project_id. It is the partition and tenant key.

-- psql "postgresql://postgres:$TF_PGWIRE_PASSWORD@localhost:5432/postgres"

-- Recent 5xx spans for one project
SELECT timestamp, name, duration / 1000000 AS ms, attributes___error___type
FROM otel_logs_and_spans
WHERE project_id = '00000000-0000-0000-0000-000000000000'
  AND timestamp > NOW() - INTERVAL '1 hour'
  AND attributes___http___response___status_code >= 500
ORDER BY timestamp DESC
LIMIT 50;

-- Slowest server endpoints, 5-minute buckets
SELECT time_bucket('5 minutes', timestamp) AS bucket,
       name,
       COUNT(*) AS requests,
       AVG(duration / 1000000)::INT AS avg_ms
FROM otel_logs_and_spans
WHERE project_id = '00000000-0000-0000-0000-000000000000'
  AND kind = 'SERVER'
  AND timestamp > NOW() - INTERVAL '24 hours'
GROUP BY bucket, name
ORDER BY avg_ms DESC
LIMIT 20;