There's a benchmark problem hiding in plain sight. Every major coding agent evaluation — SWE-bench, HumanEval, LiveCodeBench — measures whether an agent can write correct code for a given specification. Once. From scratch. Against a known test suite.
That's not how software gets built. Software gets extended. The code you write today is the context your future self — or your future agent — has to work with tomorrow.
A new paper from UW-Madison, MIT, and Washington State University ran the experiment nobody else was running: they made agents extend their own code across multiple evolving specifications, measured the code quality at every step, and compared it against how human-maintained open-source projects evolve. The results should change how you think about deploying coding agents in production.
The Setup: SlopCodeBench
SlopCodeBench contains 20 language-agnostic problems spanning 93 checkpoints. At each checkpoint, the agent receives an updated specification and must extend its own prior solution — not a reference implementation, not a clean slate. Its own code. The design decisions made at checkpoint 1 become the foundation for checkpoint 2, which becomes the foundation for checkpoint 3.
This matters because it's the causal chain that actual software development has. A coding agent that hardcodes language-specific logic in checkpoint 1 faces cascading rewrites by checkpoint 3 when the spec demands multi-language support. An agent that builds extensible abstractions from the start pays almost nothing at checkpoint 3. The benchmark makes agents live with their own architectural decisions.
They tested 11 models across open and closed weight families. They tracked two quality signals at every checkpoint:
- Structural erosion — the fraction of total complexity mass concentrated in high-complexity functions (cyclomatic complexity > 10). When agents patch new features into existing functions instead of distributing logic cleanly, this rises.
- Verbosity — redundant or duplicated code as a fraction of total lines. Measured via 137 AST-Grep rules for common wasteful patterns plus clone detection.
These are independent metrics. A codebase can be eroded but terse, or verbose but structurally sound. Tracking both gives a picture of how "slop" actually accumulates.
The Results
No agent solved any problem end-to-end. Not one. Across all 11 models and all 20 problems, zero complete solutions. The best checkpoint solve rate across the full benchmark was 17.2%.
Quality degraded steadily and predictably:
- Structural erosion rose in 80% of trajectories
- Verbosity rose in 89.8% of trajectories
- Agent code is 2.2x more verbose than 48 maintained open-source Python repositories
- Human repositories degrade less often and by smaller margins across their git histories — their code stays roughly flat while agent code deteriorates with each iteration
The degradation pattern is consistent: agents pass tests by patching new logic into existing functions rather than refactoring for the new requirements. Each patch makes the next patch harder. The complexity mass concentrates in fewer and fewer functions. The verbosity compounds.
The Prompt Intervention That Didn't Fix It
The researchers also ran a prompt intervention study: what happens if you explicitly instruct the agent to write clean, well-structured code? The answer is instructive.
Quality-aware prompts reduce initial verbosity and erosion by up to a third. That's real. But they don't slow the degradation rate. They don't improve pass rates. They don't reduce cost. The agent starts cleaner and then degrades at the same rate as without the instruction.
This suggests the problem isn't that agents don't know what clean code looks like — they can produce less verbose code when explicitly asked. The problem is that iterative extension under evolving specifications creates structural pressure that the agent doesn't have the context or architectural discipline to resist turn by turn.
Why Pass-Rate Benchmarks Are Hiding This
SWE-bench scores an agent on whether it can fix a specific bug in an existing codebase. HumanEval scores on whether generated functions pass unit tests. Both are single-shot evaluations. Neither measures whether the code the agent writes today makes the code it writes tomorrow harder.
This is a systematic blind spot. A coding agent could rank highly on every existing benchmark while generating code that erodes and bloats in exactly the way SlopCodeBench documents — because existing benchmarks don't give the agent's code a future that it has to build on.
For teams deploying coding agents in production workflows, this matters directly. If your agent is writing code that will be extended — by itself, by other agents, or by your engineers — single-shot benchmark scores are telling you almost nothing about how that code will age.
What This Means for Production Deployment
A few practical implications:
Architectural review can't be automated away. The slop the benchmark documents isn't random noise — it's systematic: complexity concentrating in hot functions, logic duplicating instead of abstracting, interfaces hardcoded when they should be generic. These are exactly the patterns a human architect reviews for in a code review. Agents can write the code; they still need external architectural discipline applied at each iteration.
The longer the horizon, the less the benchmark scores mean. A coding agent that scores 60% on SWE-bench might perform well on your two-week sprint task but degrade on a multi-month feature that requires iterative extension. Scope matters, and current benchmark designs don't capture it.
Quality-aware prompting is a partial mitigation, not a solution. Starting clean helps. It doesn't prevent the compounding. If you're using coding agents for iterative work, you need architectural checkpoints that are independent of the agent's self-assessment of its own code quality.
Verbosity has real costs. 2.2x more verbose than well-maintained human code isn't just an aesthetic problem — it's more tokens to process at each subsequent turn, more context for both the agent and any engineer who touches the code, more surface area for bugs and confusion.
The Benchmark Name Is The Point
"Slop" is a colloquial term in the LLM world for low-quality, high-volume AI-generated output that's superficially plausible but structurally poor. The paper uses it deliberately. The coding equivalent is code that passes tests while becoming progressively harder to extend — the definition of technical debt, accelerated.
SlopCodeBench provides the first measurement of how much slop coding agents produce under the conditions closest to real software development. The answer: a lot, and it compounds. No model is exempt. Quality guidance helps at the start and doesn't slow the decline.
Pass-rate is what you optimize for when you're building demos. Extension robustness is what you need when you're building software. Those are different properties, and right now only one of them is being measured.
Paper: arxiv.org/abs/2603.24755 · Benchmark and leaderboard: scbench.ai
Originally published at vsebude.it