Sheldon: how model adaptation took a local model from 53% to 91%

For most of this year I've been chasing one goal: getting the most intelligence possible out of small models running on-device. 1 to 4 billion parameters, on a 16 GB MacBook Air, documents never leaving the machine. Models that small fumble hard questions, so you compensate with engineering, and the engineering I bet on was context: retrieval, temporal resolution, clause extraction, budgeted packets with citations, a whole workflow to hand the model exactly the right evidence before it ever speaks.

Then I ran a controlled experiment to check which part of the system was producing the accuracy, and the answer was: almost none of it.

A small adapter, trained in a few hours on the same laptop, moved accuracy on frozen Kubernetes questions from 53.00% to 91.17%. My context engine, the thing with the passes and the budgets and the conflict warnings, moved it by half a point. +0.50, interval [-1.50, 2.67]. The interval spans zero.

This post is the story of how I got that number and why I trust it more than the much better number I had before it. It has the full recipe (the adapter trains on nothing but the domain's own documents: no questions, no answers, no teacher model), the controls, the 329 broken rows and where they went, and the exact boundary of what one experiment on one domain can claim. The short version: the part of the pipeline everyone tunes turned out to be the part that didn't matter.

Watching a model fail with the answer in hand

The failure that defines the whole problem fits in one case. A question about pod-level resource limits. The model, Gemma 3 4B running locally, receives 289 tokens of context that contain the complete answer. It writes a fluent paragraph about the shared pool behavior and drops the 2 CPU limit, which is the entire point of the question.

The same model with a small adapter, reading the same 289 tokens byte for byte, keeps both clauses.

Retrieval did not fail here. Nothing upstream failed. The model had everything it needed sitting in its prompt and still dropped the load-bearing clause. Multiply this case across a corpus and you get the number this post is about: for a small model, having the evidence and using the evidence are different skills. I had spent my year improving the wrong one.

The machine I built first

Sheldon started as a context engine, because of course it did. Better context is the move everyone reaches for.

The engine compiles retrieved evidence into a small, cited packet. Fixed passes: temporal resolution (which facts are still current), clause extraction, evidence ordering, structured rendering, dead-context elimination against a token budget. Conflict warnings when sources disagree. A generation gate that refuses to answer when the sources have drifted since indexing. pip install sheldon-context.

And the packets performed. Seven local models, 1,050 answers per method, on the development benchmark:

82.67% against 37.62% for raw retrieval, and every other way of stuffing the prompt far below. Across all seven models.

That number was real. I can regenerate it on demand. I no longer believe it means what I thought it meant.

82.67%, and why I stopped believing it

When I transferred pieces of the system to new settings, the margin kept not showing up. Not collapsing, just quietly failing to be there. And the more I stared at the original run, the less I could say about what had produced it, because the system changed everything at once: the vocabulary, the source structure, the retrieval behavior, the answer format, and, more quietly than I'd registered, the model's own conditioning. Five treatments, one number. A bundled win can't tell you which part did the work.

So I stopped shipping and built the experiment that could.

The adapter that trains on nothing

The other lever, the one RAG treats as fixed, is the model itself. A frontier model behind an API is untouchable, and the field's instincts formed there. But a local model's weights are sitting on my disk.

Sheldon's adaptation path onboards a domain the way you'd onboard the corpus itself: parse it into source-native sections, build a completion curriculum from those sections, train a LoRA adapter locally, pick the checkpoint by validation loss on held-out source material, freeze.

The detail that matters: the curriculum contains no questions and no answers. Nobody wrote gold responses. No teacher model distilled anything. 520 rows built from the domain's 2019 to 2024 documents, each shaped as evidence in, complete cited answer out. RAFT-style methods teach these behaviors with synthesized questions and chain-of-thought answers written by a frontier model. This trains on the documents themselves, and that turns out to be enough.

The mechanics, for anyone reproducing: LoRA rank 8 on the final 16 layers, AdamW at 5e-5, seed 617, checkpoints at 150/300/450/600 steps with the earliest minimum validation loss winning, 110 held-out validation rows from a disjoint H1-2025 split. Hours of training, on the same 16 GB MacBook Air that serves inference.

Two switches, everything else frozen

The experiment is a 2x2. Switch A is the reader: base model or frozen adapter, and the contexts each pair receives are byte-for-byte identical, so the adapter is the only difference. Switch B is the context: my specialized profile or plain generic hybrid retrieval, with the reader held fixed. A fifth control adds version-filtered source-native sections.

The benchmark had to be one I couldn't fool myself on. 150 questions about 2026 Kubernetes material, 30 each across feature state, mechanism, constraints, migration and operations, and cross-source synthesis, drawing on 180 sections from 47 official reports, scored against 957 atomic required propositions. Written and frozen before any answer existed. Training data ends in 2024, so the model can't have memorized the targets. The judge is blinded to methods and models, every row carries a SHA-256 identity, and scores opened only after all judgments closed.

Results

Flip the reader and 47 answers move, under identical contexts. Flip the context and one answer moves.

And there is the Sheldon context engine:

I wanted the other result. I had built the other result's infrastructure. This is what the matched controls said instead.

The costs, so nobody reads this as a free lunch. The adapted model is slower: 8.93 seconds mean answer latency on Gemma against 6.38 for the fastest control. And context lengths are matched at roughly 500 tokens across all five methods, so the adapter is not seeing extra evidence. I claim accuracy. I do not claim speed, and I do not claim token savings.

Outcomes

The same frozen cases, re-run across seven public model families with an independently trained adapter each:

Positive in seven of seven, with the biggest gains in the smallest models.

I label this run robustness rather than confirmation, and the reason is a gate that failed. 329 of the 4,500 new outputs came back structurally broken: 267 unresolved token caps, 65 released control echoes, 65 blocked prompt boundaries, with overlaps. The tempting move is to rerun them. The rule I froze before opening any score was the opposite: every broken row stays in the denominator and counts as a strict failure. My numbers got slightly worse and considerably harder to argue with.

One observation I am not allowed to promote to a claim: 320 of the 329 broken rows came from unadapted models. Adaptation probably also fixes termination. It wasn't randomized for that, so it stays an observation.

What the adapter actually learned

Not facts. You can see it in the gap between semantic and strict scoring.

Base Llama 3.2 1B gets 51 of 150 answers semantically right, meaning the content is in there somewhere, but only 8 survive strict scoring. It knows things and cannot execute an answer: it runs into the token cap, echoes prompt control text, drops clauses mid-list. Adapted, on the same contexts: 144 semantic, 142 strict.

Proposition completeness rises 18.8 points under both context types with no provenance regression, and none of it can come from retrieval, because retrieval is byte-identical. What the curriculum installs is behavior. I've taken to calling it an evidence-use contract, because "the model learned to answer the way the documents answer" is what the failure logs actually show.

Where it still fails

At 90.67% there are 14 failures left on Gemma, and they decompose into two different problems. Five are retrieval-incomplete two-source questions: 29 missing propositions, and every provenance failure in the run. Nine are generation omissions with complete evidence sitting in the context: 22 propositions dropped. Both halves of the chain still fail, in measurably different ways, and the fixes are not the same.

By question type, the adapted profile is perfect on constraints (30/30) and worst on cross-source synthesis (22/30), under a preregistered bar of 24. The full quality gate fails on three of six thresholds: completeness 906/957 against a floor of 910, provenance 145/150 against a required 150, and the synthesis stratum. Those are fails, and I report them as fails.

What this proves, and what it doesn't

One domain. The protocol-clean evidence is Kubernetes on Gemma 3 4B. The seven-model run is post-hoc. A second-domain replication doesn't exist yet. The judge is automated, blinded, hash-frozen, audited, and mine: zero independent human reviews are complete.

Peer reviewers read this work and concluded it isn't a paper yet, for exactly these reasons, and suggested it would make a better blog. They were right on both counts.

So the defensible claim is narrow: on a recurring, versioned domain where retrieval already reaches the needed evidence, the accuracy lived in the reader, not in the context construction. That is a statement about this setting, with controls behind it. The general lesson I would defend anywhere is smaller and sharper: if you haven't held the reader fixed, you don't know what your RAG numbers are measuring.

If you want to run this

The recipe applies when three things hold: questions recur within a stable domain, the corpus contains examples of the answer operations you need, and evidence is already reaching the prompt. If retrieval is missing documents, fix retrieval first. No adapter recovers evidence that isn't there.

pip install sheldon-context installs the engine. python run_benchmark.py verifies all 5,250 output rows against their hashes and regenerates the report every number in this post comes from. The evaluation suites and frozen results are on Hugging Face: ombhojane/sheldon-bench.

Next, in order: independent human labels on the frozen outputs, a curriculum fix for the synthesis stratum, and a preregistered second domain. If you run this on your own corpus, I want the result either way. Failures more than successes: those are the ones that tell me whether this transfers.