Red Flags to Watch for When Choosing Between AI Pentesting Vendors
Choosing between AI pentesting vendors requires looking past marketing labels like 'autonomous' or 'agentic' because the underlying systems vary widely. A Stanford-led study ran different AI agent frameworks against a live 8,000-host network and found that between 55% and 82% of reported findings were confirmed as real vulnerabilities. Swapping only the underlying model moved one framework's valid finding rate from 57% to 80%, yet that same framework still scored lowest overall since its findings stayed shallow. On a CTF-style benchmark two frameworks scored near-identically (48.6% vs 45.9%), but on the live network one scored nearly three times higher on severity and complexity. Buyers should demand proof of exploitation, real-network results, clear safety controls, data handling answers and clarity on what 'continuous' actually means.
