What is LLM data leakage?
LLM data leakage is the exposure of sensitive data through a large language model (LLM) application. It happens when personal, health, payment or business data reaches a model, a log, a vector store or a person who should not see it. The leak can occur in the prompt, the retrieved context, the model’s answer or a tool call.
For example, a support rep pastes a customer’s full chat history, with an email address, a home address and a phone number, into an AI assistant to draft a reply. The text travels to the model provider, lands in the app’s prompt log and may stay in a conversation history that other teams can search. No one meant to share it. The data simply followed the task.
Where LLM data leakage happens
- Prompts. Users and applications send raw records, documents and transcripts to a model hosted by a third party.
- Retrieved context. A retrieval-augmented generation (RAG) system pulls documents the user is not cleared to read and puts them into the answer.
- Outputs. The model repeats sensitive values from its context to the wrong person, or a fine-tuned model reproduces data it memorized during training.
- Logs and memory. Prompt logs, traces, caches and AI agent memory keep copies of sensitive values long after the task ends.
- Tool calls. An AI agent passes customer data to an external API or a Model Context Protocol (MCP) server as part of a multistep task. See how to stop AI agents and MCP tools from leaking PII.
LLM data leakage vs a data breach
A breach usually involves an attacker or a misconfigured system. Most LLM data leakage involves no attacker at all. It comes from normal work, as people and AI agents move data to complete tasks. That is why perimeter controls and traditional data loss prevention (DLP) often miss it: the data leaves through an approved app, in free text, mixed with the context the model needs.
| Aspect | Data breach | LLM data leakage |
|---|---|---|
| Typical cause | Attack or misconfiguration | Normal use of AI tools and AI agents |
| Where it shows up | Databases, file shares, endpoints | Prompts, context, outputs, logs, tool calls |
| Common controls | Perimeter security, encryption, DLP | Detection and masking in the AI data path |
How to reduce LLM data leakage
Start by finding where sensitive data enters your AI workflows. Then decide, per data type and per user, what the model may see. Common steps:
- Detect sensitive values in prompts, documents and tool outputs before they reach the model.
- Replace them with consistent tokens so the model keeps the context it needs to answer.
- Reveal real values only to users and tools that policy allows.
- Keep raw sensitive values out of prompt logs, traces and vector stores.
- Record each access in an audit log.
Here is what the model receives after masking:
Original: Call Jane Doe at 415-555-0142 about the refund to jane.doe@acme.com To model: Call <PERSON>tok_19bd</PERSON> at <PHONE>tok_3c7e</PHONE> about the refund to <EMAIL>tok_8f2a</EMAIL>
How Protecto helps
Protecto finds sensitive data in prompts, retrieved context, tool outputs and logs using an agentic detection and classification engine that covers 200+ entity types plus custom entities. It replaces each value with a consistent token, and the same value always gets the same token, so the model keeps its context. Authorized users and tools receive real values only where policy allows, and everyone else sees the data masked. Learn more about data leak prevention for AI.
Frequently asked questions
Can ChatGPT or other LLMs leak my data?
Yes, if sensitive data is in what you send. The provider may keep prompts under its retention terms, and your own app may log them. The safest step is to keep raw sensitive values out of the prompt.
Is LLM data leakage the same as prompt injection?
No. Prompt injection is an attack that tricks a model into following hidden instructions. LLM data leakage is the exposure itself, and it often happens with no attacker. An injection can cause a leak, which is one reason to limit what sensitive data a model can see.
Does masking data reduce LLM accuracy?
Simple redaction can, because it removes context the model needs. Consistent tokens keep the links between values, so the model can still follow who did what.
Related terms
Data leakage · AI DLP · Prompt injection · Tokenization · Retrieval-augmented generation (RAG) · Shadow AI