Skip to content

Operating an LLM Wiki

Operating an LLM Wiki requires treating the knowledge base as a codebase and the LLM as your primary engineer. As a human, your role shifts from manual data entry and cross-referencing to curation, orchestration, and review.

This document outlines the standard operating procedures, daily workflows, and tool configurations required to keep an LLM Wiki healthy. File roles, qmd flags, and the link contract are in Reference. The reasoning behind the workflows is in Explanation.

The Three Core Workflows

1. Ingestion

The Ingestion phase is triggered when you add new raw material. It is best to process sources individually or in small batches to maintain high synthesis quality.

Best Practices: - Place raw files in a dedicated 00 inbox/ or raw/ directory. - Invoke the agent and explicitly tell it the domain to focus on. - Review the agent's summary and specific file modifications before moving the raw source to an archive or permanent sources/ folder.

Clip and localize sources

Use the Obsidian Web Clipper to download articles as Markdown. In Obsidian settings (Files and links), set the attachment folder to raw/assets/ and bind "Download attachments for current file" to a hotkey, so images are stored locally and the LLM can view them after the original URLs rot.

2. Querying

Queries are how you extract value from the wiki. Instead of asking the agent to search the web, you ask it to search the local wiki.

Best Practices: - Force the agent to cite wiki notes, not URLs, following the link and citation contract. - If the agent synthesizes a particularly valuable insight (for example, a comparison matrix between two tools), instruct it to save that response as a permanent page in the wiki. Analytical work is then captured and compounds over time.

3. Linting

Just like code, knowledge rots. The wiki requires periodic maintenance passes to remain coherent.

Best Practices: - Schedule a weekly "lint pass" where the agent scans the wiki for issues. - Instruct the agent to look for: - Orphan pages (no inbound links). - Stale claims that newer ingestions superseded. - Structural drift (files placed outside of designated domain folders). - Missing metadata (missing YAML frontmatter).

Commands & Recipes

Because the wiki is stored as plain-text markdown, you can use standard developer tools to manage and query the knowledge base.

Managing the Log File

The log.md file tracks all agent actions chronologically. If your agent uses a strict formatting prefix, you can parse recent actions using standard Unix tools.

# View the last 5 actions taken by the LLM agent
grep "^## \[" log.md | tail -5

# Find all times the agent ingested sources related to "Docker"
grep -A 2 -i "ingest.*docker" log.md

Git Operations

An LLM Wiki is fundamentally a git repository of markdown files. This provides version control, branching (for experimental research), and collaboration out of the box.

# Check what the agent modified during its last ingestion run
git status

# Review the specific changes the agent made to existing concept pages
git diff knowledge/

# Commit a successful ingestion session
git add knowledge/
git commit -m "chore(knowledge): ingest Karpathy LLM Wiki concepts and update indices"

As your wiki scales beyond a few hundred pages, traversing the index.md becomes inefficient. Install qmd, a local, on-device hybrid search engine. It needs Node.js 22 or later (or Bun). On macOS, install Homebrew SQLite first (brew install sqlite). Commands verified against the qmd README for 2.8.3 (2026-08-16).

# 1. Install qmd globally via npm (or: bun install -g @tobilu/qmd)
npm install -g @tobilu/qmd

# 2. Add your Obsidian knowledge folder to the qmd index (Markdown only)
qmd collection add ~/Documents/obsidian-vault/knowledge --name knowledge --mask "**/*.md"

# 3. Describe the collection so results carry context for the agent
qmd context add qmd://knowledge "Agent-maintained technical wiki: topic hubs, how-to, reference, explanation"

# 4. Build embeddings (downloads the GGUF models to ~/.cache/qmd/models on first run)
qmd embed

# 5. Hybrid search with LLM re-ranking (best quality)
qmd query "how does the ingestion phase work in an LLM Wiki?"

# 6. Fast, exact-keyword BM25 search
qmd search "RAG vs LLM Wiki"

# 7. Check index health
qmd status

Re-embed after changes

Re-run qmd embed after large ingests so new pages are searchable by vector search. After switching the embedding model with QMD_EMBED_MODEL, run qmd embed -f, because vectors from different models are not compatible.

Connecting qmd to Claude Code over MCP

Once the index exists, expose it to the agent as native tools (query, get, multi_get, status).

# Option A: install the qmd plugin from its marketplace (recommended by the qmd README)
claude plugin marketplace add tobi/qmd
claude plugin install qmd@qmd

# Option B: register the stdio MCP server directly
claude mcp add --transport stdio qmd -- qmd mcp

# Optional: one long-lived HTTP server shared by several clients (localhost:8181)
qmd mcp --http --daemon
qmd mcp stop

Then add a line to CLAUDE.md / AGENTS.md such as: "Before answering, run a qmd query over the knowledge collection and read the returned pages."

Goal: every link the agent writes resolves in Obsidian and on the MkDocs site. Rules and the reasons for each are in Reference: Scoping Rules.

  1. For a note in another folder, write a wikilink scoped from knowledge/, with the heading after #: ai-agents/llm-fundamentals/how-to-guides#Speculative Decoding inside double brackets.
  2. For a page in the same folder, write a Markdown link with ./: [Reference](./reference.md#scoping-rules).
  3. Inside a Markdown table, use Markdown links, never an aliased wikilink.
  4. If a folder name contains a dot (observability-2.0), end the wikilink path with .md.
  5. If the alias would start with a digit, reword it ("Plan for 2026", not "2026 plan").
  6. Run the checks below before committing.
# Full structural audit (exits 1 on any ERROR in knowledge/)
python3 meta/scripts/kb_audit.py

# Only this topic's findings
python3 meta/scripts/kb_audit.py | grep "knowledge/ai-agents/llm-wiki/"

# Bare same-folder Markdown links (must print nothing)
grep -rnE '\]\([^)/ ]+\.(md|png|jpg)(#[^)]*)?\)' knowledge/ai-agents/llm-wiki

# Build the public site and show link warnings for one topic
DISABLE_MKDOCS_2_WARNING=true mkdocs build -f mkdocs-public.yml -d /tmp/site-check 2>&1 \
  | grep -E "WARNING|INFO +-  Doc file" | grep "ai-agents/llm-wiki/"

Troubleshooting

Issue: The Agent is Hallucinating Concepts

Cause: The agent is bypassing the index.md or qmd search and relying on its parametric memory (training data) instead of the local wiki. Fix: Update your AGENTS.md or CLAUDE.md schema file with strict instructions: "Always execute a qmd query or read the global index.md before writing a response. Never answer based on general knowledge."

Issue: Conflicting Information on Pages

Cause: A new source contradicted an older source, and the agent blindly appended the new fact without reconciling the conflict. Fix: Run a linting pass. Instruct the agent: "Review the Kubernetes explanation.md for contradictory claims about networking. Mark each conflict with a !!! warning admonition and cite both conflicting sources."

Issue: Token Limits Exceeded During Query

Cause: The index.md file has grown too large, and the agent is trying to load the entire wiki directory structure into context. Fix: Transition from manual file traversal to qmd local search. Restrict the agent from reading raw directory listings (ls -R) and force it to use semantic queries.

Issue: A Link Works in Obsidian but Points to the Wrong Page on the Site

Cause: The link has no / in its target (a bare wikilink or a same-folder Markdown link without ./). Obsidian resolves it locally, but mkdocs-roamlinks-plugin resolves it to the last file with that name anywhere under knowledge/. Fix: Rewrite it as a folder-scoped wikilink or a ./ Markdown link, then run python3 meta/scripts/kb_audit.py. Other causes (numeric aliases, anchors with spaced hyphens, links shown inside code) are listed in Reference: Syntax Gotchas.

Sources