LLM Data Leakage

What is LLM data leakage?

LLM data leakage is the exposure of sensitive data through a large language model (LLM) application. It happens when personal, health, payment or business data reaches a model, a log, a vector store or a person who should not see it. The leak can occur in the prompt, the retrieved context, the model’s answer or a tool call.

For example, a support rep pastes a customer’s full chat history, with an email address, a home address and a phone number, into an AI assistant to draft a reply. The text travels to the model provider, lands in the app’s prompt log and may stay in a conversation history that other teams can search. No one meant to share it. The data simply followed the task.

Where LLM data leakage happens

  • Prompts. Users and applications send raw records, documents and transcripts to a model hosted by a third party.
  • Retrieved context. A retrieval-augmented generation (RAG) system pulls documents the user is not cleared to read and puts them into the answer.
  • Outputs. The model repeats sensitive values from its context to the wrong person, or a fine-tuned model reproduces data it memorized during training.
  • Logs and memory. Prompt logs, traces, caches and AI agent memory keep copies of sensitive values long after the task ends.
  • Tool calls. An AI agent passes customer data to an external API or a Model Context Protocol (MCP) server as part of a multistep task. See how to stop AI agents and MCP tools from leaking PII.

LLM data leakage vs a data breach

A breach usually involves an attacker or a misconfigured system. Most LLM data leakage involves no attacker at all. It comes from normal work, as people and AI agents move data to complete tasks. That is why perimeter controls and traditional data loss prevention (DLP) often miss it: the data leaves through an approved app, in free text, mixed with the context the model needs.

Aspect Data breach LLM data leakage
Typical cause Attack or misconfiguration Normal use of AI tools and AI agents
Where it shows up Databases, file shares, endpoints Prompts, context, outputs, logs, tool calls
Common controls Perimeter security, encryption, DLP Detection and masking in the AI data path

How to reduce LLM data leakage

Start by finding where sensitive data enters your AI workflows. Then decide, per data type and per user, what the model may see. Common steps:

  1. Detect sensitive values in prompts, documents and tool outputs before they reach the model.
  2. Replace them with consistent tokens so the model keeps the context it needs to answer.
  3. Reveal real values only to users and tools that policy allows.
  4. Keep raw sensitive values out of prompt logs, traces and vector stores.
  5. Record each access in an audit log.

Here is what the model receives after masking:

Original:  Call Jane Doe at 415-555-0142 about the refund to jane.doe@acme.com
To model:  Call <PERSON>tok_19bd</PERSON> at <PHONE>tok_3c7e</PHONE> about the refund to <EMAIL>tok_8f2a</EMAIL>

How Protecto helps

Protecto finds sensitive data in prompts, retrieved context, tool outputs and logs using an agentic detection and classification engine that covers 200+ entity types plus custom entities. It replaces each value with a consistent token, and the same value always gets the same token, so the model keeps its context. Authorized users and tools receive real values only where policy allows, and everyone else sees the data masked. Learn more about data leak prevention for AI.

Frequently asked questions

Can ChatGPT or other LLMs leak my data?

Yes, if sensitive data is in what you send. The provider may keep prompts under its retention terms, and your own app may log them. The safest step is to keep raw sensitive values out of the prompt.

Is LLM data leakage the same as prompt injection?

No. Prompt injection is an attack that tricks a model into following hidden instructions. LLM data leakage is the exposure itself, and it often happens with no attacker. An injection can cause a leak, which is one reason to limit what sensitive data a model can see.

Does masking data reduce LLM accuracy?

Simple redaction can, because it removes context the model needs. Consistent tokens keep the links between values, so the model can still follow who did what.

Related terms

Data leakage · AI DLP · Prompt injection · Tokenization · Retrieval-augmented generation (RAG) · Shadow AI

Book a demo

See Protecto on your own data

Book a demo to see how Protecto finds and masks sensitive data before it reaches any AI model.

New: Protecto now runs inside LiteLLM. Control what sensitive data your models and tools see.
Read the launch blog