The question was concrete. How much of this reference data does the model actually need? I used uditgoenka/autoresearch, a Claude Code skill based on Andrej Karpathy’s autoresearch, to find out. After over 65 autonomous iterations, it cut the matrix to 303 lines (28% smaller) while maintaining 98.1% output quality.

Here’s the prompt optimization pattern, the results, and what surprised me about how robust LLMs are to reference data reduction.

The Autoresearch Pattern

Andrej Karpathy’s autoresearch introduced the core idea: give an AI agent a metric to optimize and let it loop. Modify, measure, keep or revert, repeat.

Udit Goenka built a Claude Code skill that brings this pattern to arbitrary optimization tasks, adding a dedicated guard command to prevent regressions.

You define six parameters:

• Goal: what you want to improve
• Scope: which files the agent can modify
• Metric: a number extracted from a shell command (line count, test coverage, score)
• Direction: whether higher or lower is better
• Verify: the command that produces the metric
• Guard: a safety net command that must always pass

Each iteration follows the same cycle: modify, commit to git, verify the metric, run the guard, keep or revert. Every experiment gets committed before verification, so rollbacks are clean. It tracks results in a TSV log and reads its own git history to avoid repeating failed approaches.

The separation between metric and guard is what makes this work. The metric tells autoresearch “did we make progress?” while the guard tells it “did we break anything?” Keeping those independent lets the loop optimize aggressively while the guard catches regressions.

Setting Up the Prompt Optimization Experiment

Scope

I scoped autoresearch to a single file — the reference matrix itself. I didn’t want it touching the agent’s prompt instructions, the recommendation library, or the eval infrastructure. Just the example data.

The alternative was to also let it modify the agent prompt, changing how the matrix is described. But I wanted to isolate the variable: same prompt instructions, same recommendation library, just less example data.

Metric

For the metric, I used line count. It’s simple, deterministic, and directly measures what we care about — how much data gets injected into the prompt. The metric doesn’t measure quality at all. That’s the guard’s job.

Guard

The quality gate was our existing golden-benchmark eval:

EVAL_QUIET=true npx vitest run --config vitest.evals.config.ts \
  src/evals/matrix-generation/golden-benchmark.test.ts

This eval feeds all 166 input categories into the agent, runs the full matrix generation end-to-end via LLM, and compares the output against golden reference data across four dimensions:

1. Input category coverage (did it produce rows for every input category?)
2. Output category accuracy (correct category assignment?)
3. Recommendation overlap (right recommendations from the library?)
4. Assignment accuracy (correct responsible party?)

The guard must exit 0 (all Vitest assertions pass) for a change to be kept. Each guard run took 5 to 7 minutes because it makes a real LLM API call to generate the full matrix.

The 65-Iteration Run

I ran three rounds:

Each iteration took 6 to 8 minutes (mostly the guard eval). Total wall time was roughly 7 to 8 hours across three rounds.

The agent’s approach was systematic. In round 1, it found the free wins: 5 exact duplicate rows, frequency variants (7 rows for different weekly frequencies that could collapse to 2), and severity levels where lower severity was always a subset of moderate. In later rounds, it got more aggressive, reducing most multi-row category groups to single representative entries.

Results After Fixing the Eval Baseline

During the run, I discovered that the original eval had a self-referencing bug. Both the agent prompt and the eval’s golden comparison data imported from the same REFERENCE_MATRIX_CSV constant. Every time autoresearch shrank the reference matrix, it also shrank what the eval compared against. The eval was proving “the model can reproduce a smaller matrix” rather than “the model handles all real-world input categories correctly.”

The fix was straightforward. I split the data into two files:

// src/data/reference-matrix.ts — injected into the agent prompt (optimized)
export const REFERENCE_MATRIX_CSV = `...`; // 303 lines after optimization

// src/data/golden-reference-matrix.ts — used by eval (immutable)
export const GOLDEN_REFERENCE_MATRIX_CSV = `...`; // original 421 lines, never changes

// src/evals/matrix-generation/golden-benchmark.test.ts
// Before: import { REFERENCE_MATRIX_CSV } from '../../data/reference-matrix';
import { GOLDEN_REFERENCE_MATRIX_CSV } from '../../data/golden-reference-matrix';

With the fixed eval, I binary-searched through the git history to find the optimal size. Because autoresearch commits every experiment, the full optimization history was available to test against the corrected eval.

The sweet spot is 303 lines: a 28% reduction maintaining 98%+ overall quality. The quality cliff appears around iteration 35, where the agent removed shared high-severity rows that contained unique recommendation mappings.

At 303 lines, the score breakdown:

• Input category coverage: 100% (all 166 golden categories present)
• Output category accuracy: 100%
• Recommendation overlap: 90.5% (about 32 specific recommendations lost)
• Assignment accuracy: 99.7%
• Overall weighted score: 98.1%

The main quality cost is recommendation overlap. The model still covers all input categories and assigns output categories correctly, but produces slightly fewer recommendation rows per category. For this use case, that’s an acceptable tradeoff: 118 fewer lines in every prompt for a 1.9% quality reduction.

What This Reveals About LLMs and Reference Data

The most useful finding isn’t the 28% number. It’s the degradation curve.

Even at 197 lines (53% cut), the model still hit 95.5%. It correctly covered all input categories and most output categories. The recommendation library (a separate 337-entry file in the prompt) carries much of the mapping knowledge. The reference matrix turned out to be more “example gallery” than “source of truth.” The model uses it to learn output patterns, not to look up specific mappings.

This has implications for any system that injects large reference data into prompts. The model likely doesn’t need all of it. But you need a correct eval to find the actual boundary, and the degradation is gradual, not a cliff. Without a quality gate, you won’t know where that boundary is until users report problems.

Lessons for Running Autonomous Optimization Loops

The guard is what makes it work

Without a quality gate, autoresearch is a deletion loop. The guard is the only thing preventing it from removing everything. This sounds obvious until you see how easy it is to write a guard that doesn’t actually guard.

Separate your optimization target from your eval baseline

If your golden data is the same data you’re optimizing, you’ll always pass. This is easy to do when the reference data serves dual purpose (prompt injection and eval comparison). Split them from the start. The optimization target is mutable. The eval baseline is immutable.

Git-as-memory enables post-hoc analysis

Autoresearch commits every experiment before verification. This is a form of agent memory that pays off after the run ends: I was able to binary-search through the history after fixing the eval, finding the exact commit where quality degraded. Without that history, I would have had to re-run the entire optimization from scratch.

Guard speed determines iteration budget

Fast guards (line count, type checks, unit tests) enable hundreds of iterations overnight. Slow guards (LLM-based evals, end-to-end tests) limit you to 10 to 15 iterations per hour. Plan your guard complexity based on how many iterations you can afford.

Applying This Pattern to Other Prompt Components

The recommendation library (a separate 337-entry reference file also injected into every prompt) is the next candidate for the same treatment. Same loop, same approach, but with the eval separation built in from the start.

The pattern generalizes to any prompt optimization problem: define the metric, build a correct guard, let the agent loop. The constraint is always the guard. A guard that looks correct but measures the wrong thing is worse than no guard at all.

I built a one-page scorecard based on the four layers of agent evaluation — component testing, trajectory visibility, outcome measurement, and production monitoring. It takes two minutes and shows you where your gaps are. Get the Agent Eval Scorecard →

If you’re past the scorecard stage and want hands-on help with eval design or prompt optimization, let’s talk.

Additional Reading

• autoresearch Claude Code skill by Udit Goenka
• autoresearch by Andrej Karpathy

The Problem

The following is adapted from a real production system but generalized for this post.

Take a theoretical SaaS application. The goal: generate a report of users who have low activity and identify those who are likely to churn. Let’s take a look at an example prompt:

require 'anthropic'
require 'json'

class ChurnRiskAnalyzer
  SYSTEM_PROMPT = <<~PROMPT
    You are a customer success analyst. Your job is to analyze user engagement
    data and identify customers at risk of churning.

    When analyzing users, identify which users are recently converted AND at high
    risk of churning due to low engagement. Pay special attention to:
    - Login frequency relative to their plan type
    - Feature adoption breadth
    - Time since trial conversion

    For each user, state:
    1. Days since conversion
    2. Whether they qualify as a "recent conversion"
    3. Your churn risk assessment and reasoning
  PROMPT

  def initialize
    @client = Anthropic::Client.new
  end

  def analyze(low_engagement_users)
    user_prompt = build_user_prompt(low_engagement_users)

    response = @client.messages.create(
      model: 'claude-sonnet-4-20250514',
      max_tokens: 1024,
      system: SYSTEM_PROMPT,
      messages: [
        { role: 'user', content: user_prompt }
      ]
    )

    response.content.first.text
  end

  private

  def build_user_prompt(users)
    <<~PROMPT
      Analyze the following low-engagement users from the past 30 days:

      #{JSON.pretty_generate(users)}
    PROMPT
  end
end

# Example usage
analyzer = ChurnRiskAnalyzer.new

low_engagement_users = [
  {
    engagement_id: 'eng_001',
    last_login: '2025-12-28',
    logins_past_30_days: 2,
    features_used: ['dashboard'],
    user: {
      id: 'usr_4821',
      email: 'sarah@acme.co',
      plan: 'pro',
      trial_converted_at: '2025-02-15',
      company: 'Acme Corp'
    }
  },
  {
    engagement_id: 'eng_002',
    last_login: '2025-12-20',
    logins_past_30_days: 1,
    features_used: [],
    user: {
      id: 'usr_9174',
      email: 'mike@newstartup.io',
      plan: 'pro',
      trial_converted_at: '2025-12-01',
      company: 'NewStartup'
    }
  }
]

puts analyzer.analyze(low_engagement_users)

Let’s assume the date is December 29th, 2025.

In this example we have two users with low engagement. sarah@acme.co converted back in February 2025 and has 2 logins in the past 30 days. mike@newstartup.io converted December 1st, 2025 and has just 1 login.

The expected behavior: flag Mike as at risk since he converted recently and has minimal engagement. What actually happened: both Mike and Sarah were flagged. Let’s look at why.

Lack of guidance

In the first version of our system prompt, we mention that we want to include recent conversions—but we never define what “recent” means. This leaves it up to the model to decide, which leads to non-deterministic and confusing results. The fix is to provide explicit guidance:

class ChurnRiskAnalyzer
  SYSTEM_PROMPT = <<~PROMPT
    You are a customer success analyst. Your job is to analyze user engagement
    data and identify customers at risk of churning.

    A "recent conversion" is defined as a user who converted from
    trial to paid within the past 30 days.

    When analyzing users, identify which users are recently converted AND at high
    risk of churning due to low engagement. Pay special attention to:

    - Login frequency relative to their plan type
    - Feature adoption breadth
    - Time since trial conversion

    For each user, state:
    1. Days since conversion
    2. Whether they qualify as a "recent conversion"
    3. Your churn risk assessment and reasoning
PROMPT

Now the model has explicit guidance on what makes a “recent conversion.” But we can’t stop here.

Providing a reference date

We’ve updated the prompt to provide explicit guidance on what makes a recent conversion, but there’s still one problem—the model doesn’t know what the current date is. LLMs have no system clock access; they only know what you tell them. One way to resolve this is to provide the date as part of the prompt:

  def build_user_prompt(users)
    <<~PROMPT
      Today's date is: #{Date.today.iso8601}.

      Analyze the following low-engagement users from the past 30 days:

      #{JSON.pretty_generate(users)}
    PROMPT
  end

Now the model has everything it needs for accurate reporting. Running this updated version correctly excludes Sarah, who signed up months ago, and flags only Mike.

Conclusion

When working with LLMs and time-sensitive data:

1. Be explicit about definitions - Don’t assume the model interprets terms like “recent” the same way you do.
2. Always provide the current date - LLMs have no awareness of real-time; include today’s date in your prompt.
3. Test with edge cases - Run your prompts with data that spans different time periods to catch these issues early.

These might seem like small details, but in production systems where accuracy matters, they make the difference between useful analysis and misleading results. Subtle errors like these erode trust quickly.

Building LLM features into a Rails app and running into issues like this? My book Building LLM Applications in Rails covers the patterns that hold up in production.