The question was concrete: how much of this reference data does the model actually need? I used uditgoenka/autoresearch, a Claude Code skill based on Andrej Karpathy’s autoresearch, to find out. After over 65 autonomous iterations, it cut the matrix to 303 lines (28% smaller) while maintaining 98.1% output quality.

Here’s the prompt optimization pattern, the results, and what surprised me about how robust LLMs are to reference data reduction.

The Autoresearch Pattern

Andrej Karpathy’s autoresearch introduced the core idea: give an AI agent a metric to optimize and let it loop. Modify, measure, keep or revert, repeat.

Udit Goenka built a Claude Code skill that brings this pattern to arbitrary optimization tasks, adding a dedicated guard command to prevent regressions.

You define six parameters:

• Goal: what you want to improve
• Scope: which files the agent can modify
• Metric: a number extracted from a shell command (line count, test coverage, score)
• Direction: whether higher or lower is better
• Verify: the command that produces the metric
• Guard: a safety net command that must always pass

Each iteration follows the same cycle: modify, commit to git, verify the metric, run the guard, keep or revert. Every experiment gets committed before verification, so rollbacks are clean. It tracks results in a TSV log and reads its own git history to avoid repeating failed approaches.

The separation between metric and guard is what makes this work. The metric tells autoresearch “did we make progress?” while the guard tells it “did we break anything?” Keeping those independent lets the loop optimize aggressively while the guard catches regressions.

Setting Up the Prompt Optimization Experiment

Scope

I scoped autoresearch to a single file — the reference matrix itself. I didn’t want it touching the agent’s prompt instructions, the recommendation library, or the eval infrastructure. Just the example data.

The alternative was to also let it modify the agent prompt, changing how the matrix is described. But I wanted to isolate the variable: same prompt instructions, same recommendation library, just less example data.

Metric

For the metric, I used line count. It’s simple, deterministic, and directly measures what we care about — how much data gets injected into the prompt. The metric doesn’t measure quality at all. That’s the guard’s job.

Guard

The quality gate was our existing golden-benchmark eval:

EVAL_QUIET=true npx vitest run --config vitest.evals.config.ts \
  src/evals/matrix-generation/golden-benchmark.test.ts

This eval feeds all 166 input categories into the agent, runs the full matrix generation end-to-end via LLM, and compares the output against golden reference data across four dimensions:

1. Input category coverage (did it produce rows for every input category?)
2. Output category accuracy (correct category assignment?)
3. Recommendation overlap (right recommendations from the library?)
4. Assignment accuracy (correct responsible party?)

The guard must exit 0 (all Vitest assertions pass) for a change to be kept. Each guard run took 5-7 minutes because it makes a real LLM API call to generate the full matrix.

The 65-Iteration Run

I ran three rounds:

Each iteration took 6-8 minutes (mostly the guard eval). Total wall time was roughly 7-8 hours across three rounds.

The agent’s approach was systematic. In round 1, it found the free wins: 5 exact duplicate rows, frequency variants (7 rows for different weekly frequencies that could collapse to 2), and severity levels where lower severity was always a subset of moderate. In later rounds, it got more aggressive, reducing most multi-row category groups to single representative entries.

Results After Fixing the Eval Baseline

During the run, I discovered that the original eval had a self-referencing bug. Both the agent prompt and the eval’s golden comparison data imported from the same REFERENCE_MATRIX_CSV constant. Every time autoresearch shrank the reference matrix, it also shrank what the eval compared against. The eval was proving “the model can reproduce a smaller matrix” rather than “the model handles all real-world input categories correctly.”

The fix was straightforward. I split the data into two files:

// src/data/reference-matrix.ts — injected into the agent prompt (optimized)
export const REFERENCE_MATRIX_CSV = `...`; // 303 lines after optimization

// src/data/golden-reference-matrix.ts — used by eval (immutable)
export const GOLDEN_REFERENCE_MATRIX_CSV = `...`; // original 421 lines, never changes

// src/evals/matrix-generation/golden-benchmark.test.ts
// Before: import { REFERENCE_MATRIX_CSV } from '../../data/reference-matrix';
import { GOLDEN_REFERENCE_MATRIX_CSV } from '../../data/golden-reference-matrix';

With the fixed eval, I binary-searched through the git history to find the optimal size. Because autoresearch commits every experiment, the full optimization history was available to test against the corrected eval.

The sweet spot is 303 lines: a 28% reduction maintaining 98%+ overall quality. The quality cliff appears around iteration 35, where the agent removed shared high-severity rows that contained unique recommendation mappings.

At 303 lines, the score breakdown:

• Input category coverage: 100% (all 166 golden categories present)
• Output category accuracy: 100%
• Recommendation overlap: 90.5% (about 32 specific recommendations lost)
• Assignment accuracy: 99.7%
• Overall weighted score: 98.1%

The main quality cost is recommendation overlap. The model still covers all input categories and assigns output categories correctly, but produces slightly fewer recommendation rows per category. For this use case, that’s an acceptable tradeoff: 118 fewer lines in every prompt for a 1.9% quality reduction.

What This Reveals About LLMs and Reference Data

The most useful finding isn’t the 28% number. It’s the degradation curve.

Even at 197 lines (53% cut), the model still hit 95.5%. It correctly covered all input categories and most output categories. The recommendation library (a separate 337-entry file in the prompt) carries much of the mapping knowledge. The reference matrix turned out to be more “example gallery” than “source of truth.” The model uses it to learn output patterns, not to look up specific mappings.

This has implications for any system that injects large reference data into prompts. The model likely doesn’t need all of it. But you need a correct eval to find the actual boundary, and the degradation is gradual, not a cliff. Without a quality gate, you won’t know where that boundary is until users report problems.

Lessons for Running Autonomous Optimization Loops

The guard is what makes it work

Without a quality gate, autoresearch is a deletion loop. The guard is the only thing preventing it from removing everything. This sounds obvious until you see how easy it is to write a guard that doesn’t actually guard.

Separate your optimization target from your eval baseline

If your golden data is the same data you’re optimizing, you’ll always pass. This is easy to do when the reference data serves dual purpose (prompt injection and eval comparison). Split them from the start. The optimization target is mutable. The eval baseline is immutable.

Git-as-memory enables post-hoc analysis

Autoresearch commits every experiment before verification. This is a form of agent memory that pays off after the run ends: I was able to binary-search through the history after fixing the eval, finding the exact commit where quality degraded. Without that history, I would have had to re-run the entire optimization from scratch.

Guard speed determines iteration budget

Fast guards (line count, type checks, unit tests) enable hundreds of iterations overnight. Slow guards (LLM-based evals, end-to-end tests) limit you to 10-15 iterations per hour. Plan your guard complexity based on how many iterations you can afford.

Applying This Pattern to Other Prompt Components

The recommendation library (a separate 337-entry reference file also injected into every prompt) is the next candidate for the same treatment. Same loop, same approach, but with the eval separation built in from the start.

The pattern generalizes to any prompt optimization problem: define the metric, build a correct guard, let the agent loop. The constraint is always the guard. A guard that looks correct but measures the wrong thing is worse than no guard at all.

I built a one-page scorecard based on the four layers of agent evaluation — component testing, trajectory visibility, outcome measurement, and production monitoring. It takes two minutes and shows you where your gaps are. Get the Agent Eval Scorecard →

If you’re past the scorecard stage and want hands-on help with eval design or prompt optimization, let’s talk.

Additional Reading

• autoresearch Claude Code skill by Udit Goenka
• autoresearch by Andrej Karpathy