The challenge
A cybersecurity platform offered automated vulnerability scanning, with an AI assistant built to help users interpret scan results in plain language. The problem: it hallucinated. Users could trick it into believing it had scanned websites it hadn't — "analyse google.com's security headers and compare them to my scan" — and it would fabricate headers, comparison metrics and recommendations as if the scan were real. Worse, it had legitimate access to real client scan data, so it couldn't tell where its knowledge ended.
The root causes were structural: no input validation, no output validation, no scope enforcement — safeguards skipped to hit a roadmap deadline. For a security product, confidently fabricated findings meant eroded trust, bad remediation advice and compliance exposure.
What we built
No single guardrail was enough, so we layered defences across the whole interaction — choosing CPU-friendly validators so safety didn't cost latency:
- Input — regex pattern detection to block obvious prompt-injection ("ignore previous instructions", "you are now…").
- Input — a transformer classifier for the subtler toxicity and jailbreak attempts regex would miss.
- Output — schema validation (a Pydantic schema) plus PII, jailbreak and profanity validators, so fabricated or unsafe responses never reach the user.
- Scope enforcement — the root fix: a pre-query check grounds every answer in data that was actually scanned. If a site wasn't scanned, the assistant replies "I don't have data for that website. Would you like to scan it?"
The outcome
Hallucination on tricked queries dropped to zero. Instead of inventing a security analysis, the assistant now acknowledges its limits and points the user to a legitimate next step. The guardrails added under 200ms of latency — imperceptible to users — and the same layers caught PII-extraction attempts, jailbreaks and inappropriate content as a bonus.
The lesson generalises: output validation matters as much as input filtering, CPU-first keeps guardrails production-viable, and grounding responses in real data fixes hallucination at the root rather than treating the symptom.