Attack → Defend → Govern Lab
One real, measured vulnerability class, walked through seven stages -- from what an unguarded LLM system actually leaks, to what closes it, to how that translates into governance language a non-technical stakeholder can act on.
The Build/Break/Detect/Defend/Measure stages below are real, measured evidence. Govern's risk-register example and the Monitor stage are explicitly labeled where they extend into illustrative territory -- that distinction stays visible throughout, not just here.
1. Build
Personal Project — LLM Data Leakage Lab. A deliberately vulnerable LLM-enabled Flask application, built to demonstrate a real cross-user data leakage vulnerability class, not a hypothetical one. The vulnerable baseline is preserved unmodified alongside two progressively stronger defended modes, so the same attack set can be replayed against all three and compared directly.
Stack: Python, Flask, Flask-CORS.
2. Break
Unauthorized disclosure rate -- Vulnerable Baseline
25% (3/12 adversarial cases)
tests/run_evaluation.py, response text scanned for any other user's stored sensitive field values, independent of the code's own self-reported findings.
Research (in progress) — Adaptive Moving Target Defense for LLM Security. The attack side of this MSc research uses red-team tooling (Promptfoo, with broader testing planned via Garak and PyRIT) plus a canary-string method: a unique marker is planted in protected context, and if an adversarial prompt causes the model to reveal it, that's a measured, unambiguous leak signal -- not a subjective judgment call.
3. Detect
Before a request ever reaches the model, the Defense-in-Depth mode scores it for risk and blocks anything at or above a set threshold -- a real, measured detection control, not a downstream cleanup step.
Adversarial requests blocked before reaching the model -- Defense in Depth
33% (4/12)
Pre-model input risk inspector score >= 60 (high) triggers a block; measured, not asserted.
4. Defend
Unauthorized disclosure rate -- Partial Defense (context isolation only)
0% (0/12 adversarial cases)
Unauthorized disclosure rate -- Defense in Depth
0% (0/12 adversarial cases)
Research — Three-Layer Defense Framework for Secure LLM Applications (University of Guelph, collaborative project). This companion project independently arrives at the same architectural pattern the Data Leakage Lab measures: context isolation, prompt sanitization, and response redaction, layered rather than relied on individually.
What the metrics above don't show on their own: context isolation alone stops the disclosure vector, but the separate identity-spoofing (IDOR) vector is only closed once the full Defense-in-Depth mode resolves identity from the server-side session token instead of trusting a client-declared field. Only Defense in Depth (1/1); Baseline and Partial Defense both allow it.
5. Measure
Every number on this page comes from a reproducible evaluation script (tests/run_evaluation.py), not hand-written claims: a stated threat model, a fixed set of adversarial test cases including paraphrased variants (not a single hardcoded phrase), replayed identically against the vulnerable baseline and both defended modes.
Skills demonstrated: threat modeling, context isolation, prompt/input risk inspection, output inspection and redaction, structured security event logging, authorization boundary design, IDOR identification and remediation, test-driven security evaluation.
6. Govern
Turning a measured technical result into governance language a non-technical stakeholder can act on -- without overstating it as a formal compliance assessment.
Sample risk-register entry
A concrete example of the technical-to-governance transition, built from the real measured results above.
| AI system / use case | LLM-enabled customer data assistant (based on the LLM Data Leakage Lab's architecture) |
|---|---|
| Identified risk | Unauthorized cross-user data disclosure -- via prompt-based context leakage, and via an identity-spoofing (IDOR) vector where a request carries a valid session token for one user but a claimed identity for another |
| Risk rating | High, pre-mitigation (25% measured disclosure rate) -> Low, post Defense-in-Depth (0% measured, both vectors closed) |
| Controls | Context isolation; pre-model input risk inspection (blocks at risk score ≥ 60); output redaction; server-side session identity resolution (replaces the client-declared identity field that made the IDOR vector possible) |
| Framework relevance | NIST AI RMF (Map / Measure / Manage functions); OWASP Top 10 (A01: Broken Access Control); MITRE ATT&CK-style adversarial-variation testing |
| Evidence requirement | Reproducible test set + evaluation script results (see the Measure stage above); a logged record of blocked vs. allowed requests and the risk score behind each decision |
| Owner | Sample role: Application Security / Platform Engineering |
| Residual risk | Low, contingent on the identity-resolution fix remaining in place and the input-risk threshold being periodically re-tuned against new attack variants |
| Monitoring | Structured security event logging is implemented (see the Monitor stage below); a production deployment would alert on blocked-request rate spikes and repeated high-risk-score requests from the same session |
- Relevant to: NIST AI Risk Management Framework
- The Map → Measure → Manage → Govern structure of this lab mirrors NIST AI RMF's own function areas: Break/Detect map and measure a concrete risk (cross-user data disclosure); Defend manages it with layered technical controls; this section is the govern step.
- Human oversight & disclosure
- Per the AI governance thinking demonstrated elsewhere on this site (see AI Governance), a system that can leak data if under-defended should disclose that risk to affected users and stakeholders rather than relying on silent technical controls alone.
Try this on your own AI system
The same rules-based questionnaire behind the risk-register framing above, open for any AI system you're evaluating.
Answer for a hypothetical or real AI system you're considering. Nothing you select is sent anywhere or stored -- this runs entirely in your browser against a small, fixed rule set (see the reasoning link at the end of each result).
7. Monitor
The Defense-in-Depth mode includes structured security event logging as a foundation for monitoring (see the Measure stage's skills list). A monitoring view built on that logging would reasonably surface:
- Alerting on a sustained rise in the blocked-request rate.
- Flagging repeated high-risk-score requests from the same session.
- Periodic re-evaluation of the risk-inspector threshold against new attack variants surfaced in the Break stage.