AI / LLM Security Testing Checklist

shadowe1ite — Tue, 08 Sep 2026 00:00:00 +0000

A working checklist for testing an LLM-backed application end to end. It’s the AI Hacking 101 notes</a> squeezed into “did I actually do this?” form — tick-boxes grouped by phase, from scoping through to writing it up.</p>

Only run this against a target you have written authorization</strong> for, inside an agreed scope. Several sections (agentic tools, ticket close/escalate, load testing) change state or cost money — get explicit sign-off before you touch them.</p> </blockquote>

LLMmap</a></strong> — minimal-query model fingerprinting (“nmap for LLMs”).</li>
P4RS3LT0NGV3</a></strong> — encoding/obfuscation + prompt-mutation workbench.</li>
Burp Suite</strong> — inspect the chat API and its response metadata.</li>
Small scripts for determinism and rate-limit probing.</li> </ul> </blockquote>
0. Scope & rules of engagement</h2>

Written authorization / signed RoE in hand</li>
In-scope endpoints, accounts, and data confirmed</li>
Out-of-scope systems and actions written down</li>
Rate-limit / load ceiling agreed (avoid DoS and “denial of wallet”)</li>
Point of contact + a kill-switch for anything touching real data</li> </ul>
1. Threat model the app first</h2>

Threat actors</strong> listed — malicious user, curious insider, competitor, automated bots, criminal actors</li>
Assets</strong> listed — model parameters, training data, private RAG data, user PII, agent tools, API keys / secrets</li>
Attack surface</strong> mapped — input channels, retrieval/RAG layer, tools & plugins, supply chain, logging, admin/debug endpoints, training data</li>
Risks</strong> prioritized — PII/sensitive-data leakage, model/data theft, jailbreak, internal access</li> </ul>
2. Recon & fingerprinting</h2>
Model & behavior</h3>

Identify model family/version — identity probes + LLMmap</a></li>
Estimate determinism / temperature — send one identical prompt N times, count unique replies</li>
Estimate context window and where input truncates (long A</code>×N prompt)</li>
Probe tokenizer / unicode handling (literal separators, mixed-script strings)</li>
Detect tools / agentic capabilities; probe a hypothetical web_fetch</code> for SSRF surface</li> </ul>
Retrieval & prompt</h3>

Confirm whether RAG is used — ask it to cite sources; watch response metadata and latency</li>
Plant a canary doc in a test index, then ask for it</li>
Try to surface the system prompt / hidden instructions</li>
Probe input handling — special chars, unicode, whether Base64/hex is decoded</li>
Fingerprint moderation — “is phrase X allowed?”, then paraphrase / obfuscate the same idea</li> </ul>
Infrastructure</h3>

Inspect the chat API in Burp for metadata (rag_used</code>, tickets_used</code>, response_time_ms</code>, …)</li>
Find rate limits empirically (watch for 429</code> / “too many requests”)</li>
Look for admin/debug endpoints, trace IDs, or debug flags surfaced to the user</li>
Multi-turn memory probe — ask it to remember a note, recall it a turn later</li> </ul>

The system prompt is not</strong> a security control. Treat anything you extract from it (URLs, emails, key formats, org IDs) as a lead to chase, not as the finding itself.</p> </blockquote>
3. Prompt injection</h2>
Direct injection</h3>

Ignore-previous-instructions / system-prompt extraction</li>
“Debug mode” / “maintenance mode” prints instructions</li>
Role switching (e.g. “you are now a configuration editor”)</li>
Continuation and verbatim-repetition tricks</li>
Tag spoofing — <SYSTEM>…</SYSTEM></code>, ===END SYSTEM PROMPT===</code>, [INST]</code></li>
Developer impersonation / authoritative “the user is admin” command</li>
Translation obfuscation (pig latin, other languages)</li> </ul> Indirect injection</h3> Plant a payload in app-controlled content the model later reads — ticket titles/bodies, uploaded docs, emails</li> Ask the bot about that item so it retrieves and summarizes it</li> Confirm whether retrieved content is treated as authoritative over the system prompt</li> </ul> Indirect injection rides in as “trusted” business data, so it isn’t scrutinized like something typed into the chat box — it’s usually far more effective than a direct prompt, and it’s the path a real attacker takes.</p> </blockquote> Multi-turn injection</h3> Build a persona over several benign turns, then escalate to sensitive asks</li> Reference earlier turns to legitimize the request</li> Check whether conversation memory overrides earlier refusals</li> </ul> Obfuscation & encoding</h3> Base64 / Base32 / hex / URL / unicode-escape payloads</li> ROT13 / Caesar / Atbash, Morse and emoji encodings</li> Leetspeak, custom-symbol, reversed text, case-flipping, whitespace steganography</li> Record which encodings the pipeline decodes vs. treats as literal text</li> </ul> 4. Jailbreaks</h2> Persona jailbreaks — DAN, “unshackled AI”, “evil assistant”</li> Mode tricks — Opposite Mode, Chaos Mode</li> “Rewrite your guidelines” / helpful-assistant contradiction framing</li> Threats & coercion (“I’ll overload your tokens”)</li> Log what’s refused vs. what leaks brand-damaging or false output</li> </ul> 5. Harmful & off-topic output</h2> Direct harmful-phrase repetition (profanity, hostile phrases)</li> The same phrase wrapped in a jailbreak (Opposite / Chaos mode)</li> Non-English / phonetic spelling to slip the filter</li> Off-topic requests (recipes, travel, code) — does it stay on-brand or comply then redirect?</li> </ul> 6. RAG / retrieval abuse</h2> Direct query for internal/dev docs — API, security, and access details</li> Name a specific internal document and request it verbatim</li> “Fishing” — error codes, config-file names, admin-panel and webhook questions</li> “Pray-and-spray” keyword dump (keys, api, tokens, admin, secret, credentials…)</li> Canary-token retrieval — does a planted doc steer output? are sources leaked verbatim?</li> Compare unauthenticated vs. authenticated retrieval rates</li> Watch for leaked infra — admin URLs, API keys, webhook URLs, DB paths, .env</code> / config.json</code></li> </ul> Treat any partial hit as a foothold. Even a sliver of useful info is worth keeping — note what came back and how you asked, then reshape the next prompt to dig down that same path.</p> </blockquote> 7. Access control & excessive agency</h2> Conversation IDOR — request another user’s conversation_id</code> through the bot</li> Ticket IDOR — reference an arbitrary ticket number</li> Unauthorized actions — close / escalate a ticket with no auth or ownership</li> Confirm ownership and authorization checks on every</strong> tool the agent can call</li> </ul> These tests change state. Only run close / escalate / delete against test data you’re explicitly cleared to touch, and confirm nothing persists to real users.</p> </blockquote> 8. Map findings to a framework</h2> Map each finding to the OWASP LLM Top 10</a>: LLM01 Prompt Injection</li> LLM02 Sensitive Information Disclosure</li> LLM03 Supply Chain</li> LLM04 Data & Model Poisoning</li> LLM05 Improper Output Handling</li> LLM06 Excessive Agency</li> LLM07 System Prompt Leakage</li> LLM08 Vector & Embedding Weaknesses</li> LLM09 Misinformation</li> LLM10 Unbounded Consumption</li> </ul> </li> Map techniques and tactics to MITRE ATLAS</a></li> Record success rates and reproducible prompts for the report</li> </ul>

shadowe1ite - owasp

AI / LLM Security Testing Checklist

0. Scope & rules of engagement</h2> Written authorization / signed RoE in hand</li> In-scope endpoints, accounts, and data confirmed</li> Out-of-scope systems and actions written down</li> Rate-limit / load ceiling agreed (avoid DoS and “denial of wallet”)</li>

2. Recon & fingerprinting</h2>