How well do agents use test/verification techniques?

An extensive empirical evaluation tests whether prompting coding agents (codex with GPT-5.6 Sol) to use specific testing techniques or libraries (TDD, formal methods like Verus/Lean 4/TLA+/Alloy/Kani/ACL2/Creusot/Spin, property-based testing via QuickCheck/Hegel/Proptest/rstest, fuzzing, mutation testing, differential testing, and various testing skills) improves implementation correctness on a Zstd Rust implementation eval (plus IMAP RFC checks). Across 26 conditions and multiple skills, agents largely failed to meaningfully use the named technique or library, instead defaulting to superficial or traditional unit tests regardless of instruction. TDD underperformed as predicted, formal methods mostly produced trivial or irrelevant proofs, and the Default (no instructions) condition scored above average. Fuzzing and property-based testing did marginally better than formal methods on high effort, but no condition dramatically outperformed. A small custom skill nudging away from default failure modes scored highest but still had significant limitations. The findings suggest current agents lack an internalized sense of effective testing methodology, and RL training has not addressed this gap.

Nguồn: https://danluu.com/agentic-testing. 8 Sync News chỉ tóm tắt và dẫn link; bản quyền nội dung thuộc tác giả và nguồn gốc.