How to Measure a Small On-Device Coding Agent Without Fooling Yourself

This is a methodology post, not a results post. I am building an on-device model proxy and an evaluation suite that runs a small local model inside a real coding-agent harness on real projects. The suite has not run yet, so nothing below is a measurement from it. Numbers are either from a cited source or arithmetic on a made-up example, and I say which. Results will follow in a separate post.

The question is narrow: when a model of about 3 billion parameters with an 8,192-token context tries to edit code, how do you decide whether it succeeded, and how far should you trust the answer? Benchmark designers have worked on this for years, so the goal is to reuse their practice rather than invent mine. Opinions are marked as opinions.

The problem

Two facts about the setup shape the design.

1. The model reports false success. In an earlier feasibility spike, single-step read, search and replace tasks worked. On one multi-file edit, the edit did not apply, a shell redirect then overwrote a file, and the model still claimed success. That is one run, so an anecdote, but it is enough to say the model’s “done” cannot be evidence.
2. The model is not reproducible. In an SDK probe, a seed parameter was accepted but not honoured: each of four fresh sessions gave four distinct outputs for every seed tried. Temperature 0 gave one distinct output in four runs. The probe ran each case once on one machine, so it is indicative only.

The first fact means the verdict must come from somewhere other than the model. The second means one run per task cannot be trusted.

What counts as a correct result

The standard answer in code benchmarks is functional correctness through execution, not similarity to a reference. HumanEval (Chen et al., 2021) is 164 hand-written problems checked by unit tests in a sandbox. SWE-bench moved the idea to real repositories: 2,294 tasks from GitHub issues in 12 Python projects, where the model gets the codebase and issue text and must produce a patch. The metric is the percentage of instances resolved: the patch applies and the tests pass.

The SWE-bench Verified announcement states the scheme most clearly:

• FAIL_TO_PASS tests fail before the reference fix and pass after it. They check that the issue is solved.
• PASS_TO_PASS tests pass before and after. They check that nothing unrelated broke.
• A task is resolved only if both sets pass, and the tests are not shown to the agent.

This maps directly onto small real projects. “The previously failing test now passes” is a fail-to-pass check; “the existing tests still pass” is a pass-to-pass check. My suite uses both and treats the second as a hard floor.

The same announcement is a warning about test quality. Its annotators screened 1,699 SWE-bench samples and report that 38.3% had underspecified problem statements and 61.1% had tests that may reject valid solutions; 68.3% were filtered out overall (the authors call the filtering likely overzealous). A failing test is not automatically a failing model. If tasks are generated rather than curated, they need validation.

Two more benchmarks contribute one idea each. Terminal-Bench (89 tasks in the paper) ships every task with its own environment, a human-written solution and verification tests, so each check is shown to be passable. Aider’s polyglot benchmark (225 hard exercises in six languages) was built because the older benchmark saturated: the top score was 84.2% (112 of 133), so a new record meant one or two extra solves. A benchmark near its ceiling or floor separates nothing, which is why my task set has an “expected to pass” tier, a boundary tier, and an “expected to fail” tier kept for calibration.

Not every useful task has a test suite. For “list all usages of a function”, the natural check is precision and recall against the true list. The suite uses a small fixed set of check kinds: an answer contains an expected value, a set comparison with precision and recall thresholds, an absence check (the old name is gone after a rename), a command that must exit 0, and a check that changed lines stay inside an expected range.

One run is an anecdote

Repeated trials: pass@k versus pass^k

• pass@k (Chen et al.): the chance that at least one of k samples passes. The paper gives an unbiased estimator from n samples with c passing, because computing pass@k directly from k samples has high variance, and the tempting plug-in estimate 1−(1−p̂)^k (p̂ is the observed pass@1) is biased.
• pass^k (tau-bench): the chance that all k trials succeed, averaged across tasks. The authors report that a strong function-calling model with roughly 61% single-trial success in the retail domain fell to about 25% at pass^8.

A made-up illustration (not from my suite): if a task succeeds independently with probability 0.7 per attempt, pass^1 is 0.7, pass^3 is 0.7³ ≈ 0.34, and pass@3 is about 0.97. For an agent editing files unattended, the all-trials number describes the experience. My suite records a trial index and supports repeats, but the default is one trial per task, so the first report cannot say anything about consistency. Because seeds are not reproducible, repeated trials are the only way to see run-to-run variation.

What 24 tasks can and cannot show

The planned suite is about 24 tasks: four projects (three Python, one Go), six tasks each. The simple normal-approximation interval, p ± 1.96·√(p(1−p)/n), misbehaves for small n and for p near 0 or 1 (it can collapse to zero width or leave the 0–1 range; see this secondary source). The Wilson score interval is the usual fix. I computed these myself with z = 1.96 and n = 24:

At 50% the interval is about ±19 points: 12 of 24 is compatible with a true rate from about a third to about two-thirds. And 0 of 24 does not mean “cannot do it”; it is compatible with a true rate up to about 14%.

Comparing two configurations. Suppose, hypothetically, A solves 12 of the same 24 tasks and B solves 15. Treated as independent, the difference of 0.125 has standard error √(0.5·0.5/24 + 0.625·0.375/24) ≈ 0.142, so the 95% interval is about −15 to +40 points. Pairing per task, as Evan Miller’s “Adding Error Bars to Evals” recommends, helps: with 9 tasks solved by both, 3 only by A, 6 only by B and 6 by neither, the per-task differences have mean 0.125 and standard error about 0.122, giving roughly −11 to +36 points. The interval still includes zero. At this size, a 12-point gap is not a finding.

Tasks from one project are also not independent. Miller recommends clustered standard errors for related questions and reports that on popular evals he analysed, clustered errors can be over 3 times the naive ones (a ratio of 3.05 on DROP, but only 1.10 on RACE-H). I would expect a similar direction here (opinion; I have no data on this setup), so any interval I report is probably optimistic.

What 24 tasks can show: whether the model handles easy tasks at all, whether false success is frequent enough to matter, and which failure modes exist. What they cannot show: small differences between prompts, tools or thresholds. My plan says the same: per-signal tables (pass rate with and without a signal) are descriptive only. Miller also covers what would add power later: resampling each question several times, with diminishing returns once within-question noise is small relative to variance across questions, and a power analysis before the run.

Who judges the judge

Some tasks lack a clean programmatic check, so the plan adds an LLM judge on top of hard checks. The standard reference for its problems is Zheng et al. (2023), on MT-Bench and Chatbot Arena:

• Position bias: judges favour a position; only GPT-4 gave consistent results in more than 60% of swapped-order cases, and swapping positions is a simple mitigation.
• Verbosity bias: most judges were fooled by a padded “repetitive list” answer; GPT-4 resisted better.
• Self-enhancement: some judges rated their own model higher, but the authors say their data cannot establish it.
• Agreement with humans: GPT-4 reached 85% agreement with experts on non-tie votes versus 81% human-to-human, with a random-judge baseline reported alongside.

Two caveats. The paper reports raw agreement, not Cohen’s kappa; kappa is my addition. It is (p_o − p_e) / (1 − p_e), agreement corrected for chance (secondary source), and it matters when classes are imbalanced: a judge that always says “solved” looks good on a mostly-solved set. And the paper’s setting is chat quality, not code patches, so its numbers do not transfer.

My responses:

1. Hard checks win. A run that fails project tests, leaves an unparsable file or deletes tracked files cannot be marked solved, whatever the judge says. The judge refines; it does not override.
2. The judge sees only supplied inputs: the task, the diff, test output before and after, and the model’s final answer, with tools disabled.
3. Judge inputs and outputs are stored, with the judge prompt, model name and CLI version, and a command re-judges stored runs without re-running the model.
4. Self-preference is probably less of a concern here (opinion). The judge is from a different model family than the model under test. Verbosity bias might still apply to long, confident final answers.
5. A gap: verdicts are accepted automatically, and a hand-labelled sample with a chance-corrected agreement figure is not yet in the plan. I want one before trusting judge-dependent numbers.

Generated tasks need the same care. They are produced by an LLM and accepted without manual review, so the generator validates them mechanically: an injected-bug test must fail with the patch and pass without it; a rename or edit needs a reference solution that passes; an expected answer is recomputed from the code at the base commit. This is SWE-bench Verified’s human validation replaced by machine checks. It catches broken tasks, not valid but uninteresting ones.

False success

False success is the model claiming the task is done when it is not. Benchmarks guard against it by never asking the model. In SWE-bench the tests are hidden from the agent and the verdict comes from running them. In τ-bench the evaluation compares the database state at the end of the conversation with the annotated goal state rather than judging the conversation. Terminal-Bench ships tests with each task.

My suite adopts the same stance and makes false success a headline metric instead of burying it in the failure count:

• Each run records whether the final answer claims completion.
• False-success rate is, among runs that claim completion, the share whose verdict is not solved (partial counts as not solved), reported with its count and also as a share of all runs. Runs without a completion claim, such as a plain answer to a question, are outside the first denominator.

The limits: it depends on the judge’s reading of “claims done”, itself a classification with error; the denominator is small, so one more or fewer case moves the rate by several points and it needs an interval like any proportion; and it misses a model that says nothing and does damage. That is why damage (deleted files, unparsable code) is a separate flag, and why runs have hard caps on time and requests: a run killed at the cap is recorded as cap_hit instead of quietly counted as a failure.

Fixtures, contamination and comparability

Public benchmarks are scraped from public repositories, and the SWE-bench Verified announcement itself says large models trained on internet text are likely contaminated on them. LiveCodeBench answers by continuously collecting new contest problems over time (I read only its abstract, so I cite it for that design only). My answer is cruder: my own small projects of roughly 2–4K lines, each required to pass its tests offline from a clean checkout so that a failure cannot be an environment fault. I cannot check what any model has seen of similar code.

The costs are real:

• No comparability. Nobody else can run these tasks, so I will not put the numbers next to published scores.
• No outside scrutiny. Readers can audit descriptions, not tasks.
• Selection. Four projects by one author is a convenience sample, and the Go project has few tests, so more of its verdicts lean on the judge.

Harness effects add another caveat. The SWE-bench Verified announcement notes that GPT-4 on SWE-bench Lite varied between 2.7% with an early retrieval-based scaffold and 28.3% with another. For a 3B model I expect the effect to be at least as large (opinion): in the spike, one phrasing of a shell-tool description led to a wrong argument in 3 of 3 runs, and a version with a platform hint and examples fixed it in 3 of 3. Whatever I report is a property of model, proxy, harness and tool descriptions together, and I will word it that way.

What this setup will report and what it will not

Will report after the first run: pass rate overall and per tier (solved and partial separate) with sample size and Wilson intervals; false-success, loop, damage and cap-hit rates; median and 90th-percentile per-request latency, median prompt tokens and requests per run; the environment (OS version, hardware class, SDK, harness and proxy versions, judge model and CLI version); aggregates only, with neutral project descriptions (language, size class).

Will not claim: that differences of a few points mean anything at n ≈ 24; that numbers are comparable to public benchmarks; that a pass proves correctness beyond the checks used; anything about consistency beyond what the trials show; anything about cloud models, since this first pass runs only the local one. The first report is not a go/no-go verdict. Its main product is a labelled dataset: per task, the observable signals of the run next to the most objective outcome I can get.

For disclosure I follow the spirit of two projects. HELM evaluates models under the same conditions and releases raw prompts and completions. The lm-evaluation-harness README stresses public prompts for comparability and supports per-sample logging and saved configs (I found nothing there about task versions or seeds). I cannot publish transcripts of private projects, so aggregates go public and full records stay local. That is weaker than HELM-style transparency, and I would rather say so.

Practice to setup at a glance

Sources

• Chen et al., Evaluating Large Language Models Trained on Code (HumanEval, pass@k)
• Jimenez et al., SWE-bench
• OpenAI, Introducing SWE-bench Verified
• Yao et al., τ-bench
• Merrill et al., Terminal-Bench (abstract page only)
• Aider, Polyglot benchmark
• Jain et al., LiveCodeBench (abstract page only)
• Miller, Adding Error Bars to Evals
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Liang et al., HELM (abstract page only)
• lm-evaluation-harness README
• Wikipedia: binomial proportion confidence interval, Cohen’s kappa (secondary; formulas only)

Written October 23, 2026. The evaluation suite described here has not yet been run.