Summary
A paper on arXiv describes drydock-eval, a local evaluation kit for small coding agents, tested on Gemma 4 E4B. The authors say a hosted score in the Gemma 4 Developer Agent competition is one noisy number, so they built a local evaluator that is fast, safe for agent-written code, and matches the hosted scorer's prompts. According to the source, the kit evaluates the agent at about 250 runs per hour on one RTX 5090. Its Contain component runs each job in a fresh container without network access and passed 200 scripted escape tests.
AI-assisted summary based on the listed source.
What happened
A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. Contain...
Why it matters
Developers building or benchmarking small coding agents may get a way to test locally before relying on a single hosted leaderboard score. Teams that need sandboxed execution of agent-written code can look at the containment approach and its escape-test results. The source snippet is cut off before describing the full set of components, so the complete scope and results should be checked in the paper.
Signal Intelligence
Signal Strength 93%
Technical label WATCH
Public Interest 22
Category RESEARCH
Reader Depth TECHNICAL
Signal Strength reflects source quality, relevance, freshness and evidence. Public Interest helps organize discovery; it is not proof of truth.
Public Interest components
Recognizable Entity Score 0
Practical Impact Score 20
Novelty Interest Score 48
Consequence Score 34
Curiosity Score 16
Shareability Score 21
