Karpathy's LLM Knowledge Base, Wired Into a Real Agent

Andrej Karpathy posted a thread about using LLMs to compile a personal knowledge base — a two-layer pattern where raw clippings are processed into a structured wiki the agent actually reads. Most public implementations treat the wiki as a standalone tool. This video does something different: it wires the pattern into a real production AI agent.

A two-layer knowledge base, applied to a real agent

I build the Karpathy pattern into Emma — my AI operator with Obsidian vault access, custom slash commands, and a CRM. The raw layer lives in 0-Capture & Process/Clippings/ (populated by the Obsidian Web Clipper). The compiled layer is a structured wiki of pages plus atomic, cross-linked Zettelkasten notes. A compile skill drives the transformation. Emma reads the compiled layer at query time, not the raw clippings.

Claude Code plan mode for the architecture

The build uses Claude Code plan mode for the architectural decisions — where the wiki lives, how sources are classified, who owns the compiler, how it gets triggered. Claude’s sub-agent fan-out explores the codebase in parallel before proposing a plan. I edit the plan directly in the editor before approving execution. Then auto mode handles the implementation with /clear discipline to keep context clean across phases.

Local LLM, real agent, plan-mode discipline

The whole thing runs against a local LLM — Qwen 3.6-27B on a DGX Spark — instead of a hosted model. I walk through the compile skill end to end, including how the agent classifies sources, decides what gets atomized, and links new notes back into the existing graph.

If you want to apply this kind of architectural pattern to your own setup, I cover Claude Code workflows and agent architecture in 1:1 coaching. For related context on giving agents persistent memory, see I Gave My AI Agent Access to My Second Brain.