Fabricated conditions in AI security reports

I noticed a repeating pattern with AI security reports lately: The attack chain just assumes a pre-existing breach, then continues to frame normal and expectable behavior as a security issue.

Some examples that were actually reported to me in the past:

• A web application serves static files from a specific public directory to users. If that directory contains a symlink to /etc/passwd, and the file is readable by the web server, an attacker may be able to request it and leak sensitive information.
• A web framework provides a method for adding custom headers to HTTP responses. If an application developer ignores all security practices and passes untrusted user input directly into response headers, and if the HTTP stack serving the application also ignores the spec and forwards headers with invalid characters to clients, then an attacker may use this for header injection or cache poisoning.
• Signed cookies are parsed with cPickle on the server side. If the signing key is leaked, malicious cookies may trigger RCE.
• An internal function decodes bytes to strings without catching UnicodeException. The report claims this could be used for denial-of-service attacks, but does not show how this exception could be triggered through any of the public APIs. The provided PoC calls the internal function directly, bypassing the safeguards that usually surround it.

All of these reports share a pattern: they assume a prior breach unrelated to the software under audit, or exploit conditions that violate the software's trust boundaries and assumptions, and then label the direct consequences of those fabricated preconditions as separate security issues in the audited software.

Under fabricated conditions, even correct behavior can be exploited. Garbage in, garbage out. An API that assumes trusted input cannot be expected to magically detect and sanitize malicious input. A pencil can be used as a weapon, but that is not the pencils fault. Banning pencils or expecting them to break on impact does not improve security, it breaks pencils.

There are exceptions, of course. If the reported issue is an actual bug or undefined behavior and can be used in an attack chain to drastically increase the impact of another common bug, then yes: it should be fixed. But does that make it a security issue that requires responsible disclosure and a CVE? Usually no. A public issue may be enough.

AI often ignores the threat model implied or defined by an API and fabricates preconditions that break with established trust boundaries or assumptions, just to be able to report a finding. In my experience, this happens most often when AI is explicitly asked to find vulnerabilities in software that does not have any obvious issues. It would rather hallucinate a finding than admit that it found nothing.

So when you use AI to audit software, or when you deal with AI-assisted security reports, here is a checklist:

• Ask for a PoC that demonstrates the entire attack chain under realistic circumstances, or for tangible real-world examples of how the preconditions could happen by accident or be provoked by an attacker. If preconditions cannot be explained, they should not be assumed.
• If the preconditions assume insecure configuration, operation, or usage practices, check documentation and defaults. They should be clear and secure by default. It does not hurt to have additional runtime checks for common configuration or usage errors, but the absence of those checks is not automatically a security vulnerability. If someone wants to shoot themselves in the foot, they will always find a way.
• If the preconditions assume a prior breach, re-evaluate impact and severity after factoring out any out of scope conditions. Sometimes the preconditions are more severe than the actual finding, which drastically reduces the real impact. If the preconditions are completely unrealistic or out of scope, then the solution is also out of scope. We cannot fix bugs in other people's software.