AI Agent Re-grades 10,000 Cloud Alerts: Three-Quarters Had the…

Sola Security study of 9,967 HIGH findings: 58.5% downgraded, 16.6% upgraded. Static CSPM vendor severities do not reflect real risk in production environments.

On October 13, 2026, Sola Security published the results of a production experiment: an AI agent with read-only access to cloud environments re-evaluated 9,967 findings classified HIGH by two commercial platforms and changed the severity grade in 75.1% of cases. The study, not yet independently replicated, is limited to HIGH findings across eight AWS accounts and does not measure incident data or the agent's recall rate. The data is not proof of superintelligence, but a measure of the systemic misalignment between CSPM vendor rules and the operational reality of the environments under review.

The stakes are the remediation queue that cloud teams manage daily, ordered by severity that overestimates risk in 58.5% of cases and underestimates it in 16.6%.

Key Takeaways

Sola Security tested an AI agent on 9,967 HIGH-severity findings from two commercial platforms across eight production environments: 58.5% were downgraded, 16.6% upgraded.
32.3% of downgrades involved inert, orphaned, or unconnected resources with no real attack path.
49.0% of the agent's verdicts relied on evidence external to the flagged resource, drawn from connected resources or other security products.
Read-only AWS probes confirmed the agent's facts in 99.4% of verifiable cases; conservative accuracy drops to 81.8% if all non-verifiable findings are counted as errors—cases where the probe could not establish facts due to lack of visibility or insufficient data.

How Contextual Reclassification Works

The agent operates via a two-phase methodology: a "theory of exploitation" that models the specific attack path for each resource, and an "evidence collection" phase that verifies connectivity, internet exposure, live services, and compensating controls via cloud APIs and an asset graph. It does not merely reapply the CVSS environmental score; it reconstructs the actual state of the network, security groups, and resource dependencies.

Study authors Leon Goldberg and Gal Engelberg write that "a static severity reflects the risk of the condition in general, and a queue ordered by it is ordered by the general case." The vendor rule, in other words, encodes the context its author could codify once for all environments; the context that decides what that condition is worth in a given environment "lies elsewhere."

The agent retrieves it, and the result is that 599 vendor rules emitted a single severity across 99,073 total findings, regardless of resource or environment.

The 32.3% Pattern: Resources That Cannot Be Attacked

The most frequent downgrade pattern, nearly a third of the total, involves inert, orphaned, or unconnected resources. These are not technical false positives: the condition detected by the vendor rule exists, but the resource is unreachable, not associated with a live service, or isolated such that the theoretical exploit has no practicable path.

Static severity measures the risk of the condition in the abstract; the agent measures the risk of the condition in that specific network topology. The finding connects to a known problem in cloud security management: CSPM platforms generate alert volumes that teams cannot clear, and priority is determined by a score that does not incorporate runtime state.

The Sola Security study quantifies, for the first time on a population of nearly ten thousand findings, the extent of this misalignment in a production environment.

"The rule-severity gap is large in production, and closing it requires evidence from the whole environment"

Rules That Miss in Both Directions

A less intuitive data point emerges from analysis of the 138 rules with the most findings: 62 produced both downgraded and upgraded results, and for 57 of those, both outcomes occurred within the same environment. The same rule, applied to different resources in the same account, required opposite corrections. This is not a problem of a rule being too sensitive or too rigid; it is a problem of a rule without context.

The study's evidence map documents an exemplary case: a security group rule was downgraded when associated with an internal load balancer, and raised when the same rule type sat on an internet-exposed path. The condition is identical, the vendor assigns the same severity, the actual risk is different.

49.0% of verdicts relied on evidence external to the flagged resource, confirming that risk assessment requires a field of visibility wider than that of the single asset.

What Changes

The transition from static severities to evidence-based re-grading is not a complete solution, but it recalibrates the trade-off between coverage and precision that cloud teams face daily. CSPM platforms tend to favor coverage: more active rules generate more findings, and more findings cover more attack surface. The cost is alert fatigue, with teams spending time on inert resources while exposed resources sit in the queue with the same HIGH label.

Goldberg and Engelberg clarify that "verified, environment-specific reading of each finding is a precondition for acting on findings autonomously." The distinction is sharp: contextual reading is a prerequisite, not an enabler, of automation. The study's agent operated in read-only mode; the re-grading is a priority recommendation, not a remediation action.

The industry dilemma remains open: CSPM vendors use static rules by design, not negligence. A single severity per condition is scalable and deterministically reproducible. Integrating runtime context requires continuous access to asset state, the ability to reason about network dependencies, and governance of permissions for a third-party agent that reads the entire environment. 92% of re-graded findings came from AWS, with only 8% from other clouds: generalizability to other platforms remains to be verified.

For operational teams, the implication is more modest and more immediate: before reordering the remediation queue, verify whether the HIGH finding at the top of the list concerns a genuinely exposed resource or an orphaned resource that the vendor rule classified with the same severity as a real attack path.

This article is based on a single empirical study published by Sola Security on Help Net Security. The data has not been independently replicated and the scope is limited to eight AWS environments and HIGH-severity findings.

Information is based on the cited source and current as of the time of publication.

Sources

Sources and references

Get DeafLetter