AI AppSec: Scanners Agree on Just 5%, Triage Costs $128,000

Contrast Security's AppSec Overflow 2026 report reveals three AI scanners testing the same codebase agree on only 5% of findings, while a single scanner reproduces just 17% of its own results across runs. Scanning 2 million lines costs roughly $315 in API fees; triaging the output costs about $128,000 — a 400-to-1 asymmetry.

On August 31, 2026, Contrast Security published the AppSec Overflow 2026 report, drawing on telemetry from hundreds of thousands of applications and APIs in production. The data exposes a systemic instability in AI tools for security assessment: three scanners agree on only 5% of findings on the same codebase, while a single scanner run three times reproduces just 17% of its own results. The cost of triage exceeds the cost of scanning by a factor of four hundred.

Key Takeaways

Three AI scanners tested by Contrast Labs agree on only 5% of findings on the same codebase [SOURCE 1]
A single AI scanner run three times on the same code reproduces only 17% of its own results [SOURCE 1]
Scanning 2 million lines costs roughly $315 in API charges; triaging the findings costs roughly $128,000 [SOURCE 1]
Monitored applications carry an average of 106 vulnerability findings, 22 of them high or critical, with a mean 92-day remediation time [SOURCE 1]

The Numbers Behind the Disagreement: 5% Convergence, 17% Reproducibility

Contrast Labs subjected the same codebase to three AI scanners. The result: a 5% overlap. The remaining 95% of findings are disjoint — alerts generated by one tool and ignored by the others, with no external criterion to decide which is right.

The variability is not only inter-tool. A single scanner executed three times on the same code reproduces only 17% of its own results. The remaining 83% vary from run to run, making temporal comparison impossible and prioritization arbitrary.

The source does not specify which scanners were tested or their operational parameters (model temperature, versions, prompt engineering). The quantitative data stands, but the methodology is not independently reproducible. It is also unclear whether the 5% figure covers all vulnerability categories or specific subsets.

The Economic Asymmetry: $315 to Scan, $128,000 to Triage

The report quantifies the cost of an AI scan on 2 million lines of code: roughly $315 in API tokens. Triaging the generated findings — the manual verification of what is real, exploitable, and priority — costs roughly $128,000. The ratio is about 1 to 400.

This asymmetry turns triage into the economic bottleneck of AppSec, not scanning. Security teams no longer spend to find vulnerabilities; they spend to decide which of the findings are real. AI has automated alarm generation, not validation.

According to the cited sources, monitored applications carry an average of 106 vulnerability findings in proprietary code, 22 classified as high or critical. Remediating a critical vulnerability takes an average of 92 days.

The Timeline Compresses: From Years to Weeks, From Weeks to Hours

The report tracks the acceleration of weaponization time. According to Contrast data, the mean time to exploit fell from over 2 years in 2018 to under 1 year in 2021, to a majority within 3 weeks in 2025. In 2026, some vulnerabilities are weaponized in hours.

Meanwhile, remediation time has not shortened comparably. The 92-day average for a critical vulnerability in custom code leaves a gap of months between exposure and fix.

Contrast records 42 confirmed, viable exploit attempts per application per month, with an average of 11,382 attacker-to-application contacts — roughly one every 4 minutes. More than 60% of applications suffer fewer than 3,000 attacks monthly; more than a quarter absorb over 30,000.

CVE-2022-22965 (Spring4Shell, CVSS 9.8 CRITICAL) and CVE-2021-44228 (Log4Shell, CVSS 10.0 CRITICAL) remain widely present in production years after disclosure, evidence that the problem is not only the speed of discovery but the speed of removal.

"These tools disagree with each other; they disagree with themselves from one run to the next, and none of them can tell me how my application behaves when someone is actually attacking it. That is fine when AI is one input among several. It is a problem when it becomes the system of record, because that is what decides what my team works on Monday morning." — David Lindner, CISO Contrast Security

Epistemic Instability: When AI Becomes the "System of Record"

David Lindner, CISO of Contrast, frames the problem in terms of work governance: when AI tools become the "system of record" for prioritization, they decide what the security team tackles on Monday morning. If results change with every run, prioritization becomes institutionalized gambling.

Jeff Williams, CTO of Contrast, shifts the emphasis to the temporal dimension: "For twenty years the discipline of AppSec has been organized around a race: find the vulnerability, decide if it matters, and fix it before somebody with bad intent finds it first. AI ended that race, and defenders lost it. We are now seeing vulnerabilities weaponized in hours while the average critical fix takes weeks or months."

Lindner adds: "AI is not going to triage its way out of this problem, and we have the data to prove it." The statement closes the door on a purely algorithmic solution: if the instability is structural, more AI in triage does not resolve the instability, it amplifies it.

Market Context: Bounty Programs Under Strain, AI on Offense Too

The report is not an isolated signal. According to cited sources, HackerOne suspended new submissions to the Internet Bug Bounty program in March 2026; Node.js did the same. The asymmetric pressure between vulnerability generation and management is warping the incentives of coordinated disclosure programs.

On the other side of the field, offense increasingly relies on AI. Cited sources mention tools such as Google Mandiant AVDH for automated vulnerability discovery, accelerating the weaponization cycle the Contrast report documents. Defense is playing catch-up, but with detection tools that do not even agree with each other.

Why It Matters

The dossier does not specify independently tested operational mitigations nor verify whether the 5% and 17% instability extends to all market vendors or remains confined to the three scanners analyzed. It is unclear whether different configuration parameters (zero temperature, structured prompt engineering, retrieval-augmented generation) could stabilize results.

The source does not document specific corrective measures organizations can adopt beyond the shift toward runtime visibility proposed commercially by Contrast. The report is produced by a vendor that sells that very solution: selection bias should be kept in mind when reading it.

No infrastructure overlaps emerge in the dossier between the tested tools and specific development ecosystems (cloud-native, embedded, legacy monolith) that might explain part of the variability.

The fundamental datum remains: when the systemic reliability of a tool falls below the threshold of operational use, the problem is not technical but decisional. The 5% agreement is not a precision metric; it is a reliability metric, and it says the tool is not reliable as a sole input.

For CISOs, the question is not whether to use AI in security assessment, but whether to use it as one of several inputs or as the "system of record." The Contrast report argues the second option is risky. The numbers on disagreement and intra-tool instability provide a quantitative foundation for that caution.

For the industry, the report raises an ROI question: if AI investments in security produce non-reproducible findings and a triage that costs 400 times the scan, the return is measured on the throughput of alarms generated or on the throughput of vulnerabilities actually closed. The report's answer is implicit: runtime protection is the only domain where the measure is behavioral, not interpretive.

Information has been verified against cited sources and is current as of publication.

Sources

Sources and references

Get DeafLetter