10 Nov 2026 AI data leakage: how to test whether your AI application gives away data it shouldn’t

In short: AI data leakage means your AI application shows data to someone who shouldn’t see it. It can come from what the model learned, from its own instructions, from documents it searches, from earlier conversations or from its logs. You test for it with targeted questions per route, real user roles and planted test markers. Repeat every question, because the answers vary.

Your AI assistant doesn’t need to be hacked to leak data. Someone just has to ask the right question. That’s what makes AI data leakage different from the leaks your security team knows. There’s no break-in to detect, only a friendly answer with a customer’s address in it. If your organisation has a chatbot or an AI assistant live, or is about to, this is worth an afternoon of your time.

What is AI data leakage?

AI data leakage is when an AI application gives away personal or confidential data to someone who shouldn’t have it. It can show up in an answer, in the sources the assistant quotes, or in what it writes to its logs. Most of these applications run on an LLM, a large language model. That’s the kind of AI that reads a question and writes an answer. An LLM doesn’t know what’s secret. It works with whatever it was given: its training, its instructions, the documents it can search and the conversation so far. If something sensitive is in there, a clever or simply curious question can bring it out. The OWASP Top 10 for LLM applications lists this as a main risk, called sensitive information disclosure. Under the GDPR, the EU privacy law, showing personal data to the wrong person can count as a data breach. Then the cause matters less than the fact that it happened.

Where can an AI application leak data?

There are five routes, and each needs its own tests. It helps to name them before you start, because teams often test one and assume the rest is fine.

• What the model learned. If a model was fine-tuned, trained further on your own data, it can repeat bits of that data word for word.
• Its own instructions. The system prompt is the hidden set of instructions that tells the assistant how to behave. It sometimes holds internal rules, names or even access keys. OWASP calls this system prompt leakage.
• Documents it can search. An assistant that searches your files can quote a file the user has no right to see. Our post on RAG testing covers how to test that search step.
• Earlier conversations. An assistant with memory can carry details from one user’s chat into another’s.
• Logs and connected tools. Questions and answers end up in logs, monitoring tools and sometimes at an outside AI provider. That’s data leaving your control too.

Attackers often reach these routes with prompt injection: text that tricks the AI into ignoring its rules. We covered that in our post on prompt injection testing. Here the focus is the data itself: what can come out, and to whom.

How do you test for AI data leakage?

You test it by asking the questions an attacker or a curious user would ask, route by route, and checking what comes back. Plain pass or fail doesn’t work well here. An AI application doesn’t give the same answer twice, so one clean answer proves little. These steps give you a test you can repeat.

1. Plant test markers. Put made-up but recognisable values in your test data and instructions. Think of a fake customer name or a fake code. If one of them ever shows up in an answer, you’ve found a leak and you know where it came from.
2. Write probes per route. For each route, write a set of questions that try to get data out. Ask directly, ask politely, ask in another language, ask for a summary of “everything you know”.
3. Test with real roles. Run the same questions as a customer, an employee and a manager. An admin account alone hides every access problem.
4. Repeat every probe. Ask each question several times. A leak that shows up one time in ten is still a leak.
5. Check the logs too. Look at what the application stores and where it sends it, not only at what the user sees.

Much of this can be automated, and it should be, because you’ll run it again and again. But people stay part of it. Whether an answer gives away too much is often a judgement call. A privacy officer or the owner of the data should look at the grey cases. No script decides that for them.

How often do you test again?

Every time something changes, and in between as well. A new model version, a changed system prompt, new documents or new access rights can each open a route that was closed yesterday. Treat your probe set like a regression set: the tests that check what was safe yesterday is still safe today. Run it before every release and after every change to the data the assistant can reach. After go-live, keep watching real conversations for personal data in answers.

One trade-off to be honest about: you can’t prove an AI application will never leak. The space of possible questions is endless. What you can do is lower the risk, find the common leaks before a user does, and show you checked. That last part matters. If a leak happens, the GDPR expects you to report a personal data breach to the regulator within 72 hours in most cases. Being able to show what you tested and when makes that conversation very different.

What this means for your team

Testing for AI data leakage keeps going long after launch. Someone has to keep the probe set current and run it after every change. They follow up on what fails, and report it so your privacy officer and management can act. In many organisations that job doesn’t exist yet. The AI application goes live after a good first test, and then nobody looks again.

That ongoing testing is what we built our in-house tool VeritasAI for, and we use it to deliver AI testing as a service. It keeps testing your AI application with real questions. An operational dashboard shows which questions fail and which law, regulation or framework each failure maps to. It shows the prompt, the answer and a priority score. A management dashboard gives the overview. You’ll find more on our AI compliance testing page.

A first step for this week: write down the three kinds of data your AI application must never show to the wrong person. Then ask it, as a regular user, to show you exactly that.

Frequently asked questions

Can an AI chatbot leak personal data?

Yes. It can repeat data it learned, quote a document the user shouldn’t see, or carry details from one conversation into another. It doesn’t need to be hacked. A well-chosen question can be enough.

What is system prompt leakage?

It’s when an AI application reveals its hidden instructions. Those can include internal rules, names or access keys. Keep secrets out of the system prompt, and test whether users can get it to repeat its instructions.

Is a data leak through an AI application a GDPR data breach?

It can be. If personal data is shown to someone who shouldn’t see it, the GDPR treats that as a personal data breach. Check with your privacy officer, and keep a record of what you tested.

Can you test for data leakage automatically?

Largely, yes. Probe sets, test markers and repeated runs can all be automated. People still judge the grey cases, such as whether an answer reveals too much.