RedSentinel

A reflected XSS scanner spends nearly all of its budget firing payloads that were never going to work. A payload that breaks out of an attribute value is inert inside a script string; one that works in HTML text does nothing in an href. Classical scanners handle this with signature lists and pattern matching, which means they either fire everything — slow, noisy, and loud enough to trip a WAF — or they fire a narrow list and miss.

RedSentinel puts a classification step in front of generation. The context module probes each reflection point and reports the context it landed in and which characters survived the round trip. Only then does payload generation run, against a bank filtered to that context and ordered by a ranker. The expensive stage operates on a shortlist instead of a catalogue.

The NestJS core is the only service that writes scan state. Everything expensive happens off a Redis-backed BullMQ queue, so a slow or hostile target never blocks the request that started the scan. The three Python services are stateless over HTTP and can be restarted independently of an in-flight scan.

The numbering here is real: each stage consumes what the previous one produced, and a scan cannot skip forward.

On the 47-endpoint evaluation the scanner records TP 44, FN 0, FP 0, TN 3 — F1 1.000. That number is only meaningful with its counting rule attached: a finding is counted only if the payload executes in a headless browser. Endpoints that reflect input but never execute it produce zero findings, which is where the zero false positives come from.

Cross-tool comparisons against ZAP, XSStrike and Dalfox use reflection-based counting instead, because that is how those tools decide. Under that rule the numbers move, and tools without browser verification pick up false positives. Both rules are in the repo, along with the scripts that reproduce them — the point of publishing both is that a single headline F1 with no methodology attached is not evidence of anything.

For most of the project the context classifier reported 99.53% context accuracy and 99.56% severity accuracy on a 3,632-sample test set. Those numbers went into the report. They were wrong, and the way they were wrong is the most useful thing I learned building this.

Validation accuracy had been sitting around 75% the whole time. A twenty-four-point gap between validation and test is not a good sign — a test set is supposed to be harder than validation, never easier. So I went looking for the leak instead of enjoying the number.

The splits overlapped badly. 161 of 206 unique test payloads — 78% — also appeared in training. Exact duplicate payloads ran at 78.2% between train and test, and even the test payloads with novel normalised forms averaged ~0.91 string similarity to their nearest training example. The model was not inferring context from payload syntax. It was recalling the label it had most often seen attached to that exact string.

I regenerated the splits with zero payload overlap and re-ran the evaluation. The honest numbers are 78.4% context accuracy and 38.2% severity accuracy on 306 clean test samples — consistent with the ~75.1% / ~35.4% validation figures, which is exactly what you want to see once the leak is closed.

Severity is the weak head, and it is weak for a defensible reason: severity is not a property of the payload string. It depends on where the reflection sits, what the surrounding application does with it, and whether it executes — runtime evidence the classifier never sees at training time. That is why runtime severity in the shipped scanner is rule-based and the classifier only advises. Fixing the head means fixing the labels, not the architecture.

A security tool that silently reports a clean scan when its own components are broken is worse than no tool. Each degradation path is explicit.

Run against real applications — Juice Shop, WebGoat, the OWASP Benchmark — the scanner hits architectural limits rather than detection bugs: single-page apps that exchange JSON instead of form posts, multi-step authentication flows, and injection through POST bodies or the Referer header. These are crawler and transport gaps, not classifier gaps, and they are the honest next piece of work.

The severity head needs relabelling before it earns any weight in a report. And the dataset, at 59,122 curated entries drawn from AwesomeXSS, PayloadsAllTheThings, XSSGAI and PortSwigger material, is broad but skewed toward payloads people publish — which is not the same distribution as payloads that work.