Short answer. AI data leakage now includes policy-enforcement failure, not only undetected values. When a protection layer assigns a sensitive value the wrong specific data type, downstream masking, retention, and access rules select confidently on a false premise. The control exists and runs. It is pointed at the wrong category of data.
Most AI data leakage conversations measure one thing: what the protection layer missed. That is a real risk and it is the easier one to catch, because a missed value eventually surfaces as an incident. The failure mode we think is underweighted is the opposite one. The layer finds the value, assigns it a specific data type, and gets the type wrong. Nothing in that output looks unresolved, so nothing downstream treats it with caution.
A missed value can be surfaced through an audit. A specific but incorrect label may be harder to uncover because the control appears to operate successfully while applying the wrong policy.
Recall was the right metric for DLP. It is not sufficient here.
Traditional data loss prevention evaluation emphasized recall, and that fit the original use case. A loss-prevention system blocks what it finds. Nothing downstream depends on understanding what was blocked, because nothing downstream ever sees it again.
AI workloads changed the requirement. Sensitive data is no longer blocked. It is rewritten and passed forward to a model, an agent, a retrieval index, or a business system. Three things now depend on the label attached to it.
The model still has to answer the question. If a service prompt reads “the April invoice is wrong” and the protection layer replaces “April” with a generic person tag, the model can no longer identify the invoice. Nothing leaked and the response is wrong.
The policy engine still has to decide what to do. Retention, redaction strictness, access rules, and regional handling all depend on the data type. A driving license number and a bank account number may be subject to different jurisdiction-specific requirements. A generalized label gives the pipeline nothing to select on. A wrong specific label gives it something to select on that is false.
The audit trail still has to hold. A reviewer will ask which fields were protected, under which classification, and on what basis. “The tool said it was a phone number” is a poor answer when the value was a Social Security number.
The 14 misclassifications we recorded
We ran a comparison between OpenAI PrivacyFilter and Protecto on 150 synthetic enterprise prompts in July 2026, scored against 344 expected annotations assigned before either product was run. Against those 344 annotations, PrivacyFilter detected roughly 68% under some label and 37% arrived with a data type precise enough for a policy engine to act on. Protecto correctly classified 90%.
Inside that scorecard, one outcome category was scored separately: detected, assigned a specific type, and the type contradicts what the value is. Fourteen annotations landed there, and they are worth looking at individually.
| Expected type | Type returned | Annotations |
|---|---|---|
| IP address | PRIVATE_URL | 3 |
| US Social Security number | PRIVATE_PHONE | 3 |
| Card CVV | PRIVATE_ADDRESS | 2 |
| Payment card | PRIVATE_DATE or PRIVATE_PHONE | 2 |
| Aadhaar | PRIVATE_PHONE | 1 |
| Bank account | PRIVATE_PHONE | 1 |
| India PAN | PRIVATE_PERSON | 1 |
| Medical code | PRIVATE_ADDRESS | 1 |
Those counts are annotations, not test cases. Fourteen annotations, scored as a separate outcome category. It is a small count and we are not going to inflate it. The reason we pulled it out is that the consequences do not scale with the count.
Take the IP address labeled as a private URL. Indicator search, threat correlation, geolocation, and IP-specific retention logic can all fail on that value, and each of those is a control a security team believes it has. The failure is silent because the pipeline received a well-formed, specific answer.
Take the card CVV labeled as a private address. Card-security data can be routed into address handling instead of card-specific controls. Address handling is a real control path. It is the wrong one for a CVV.
Take the Social Security number labeled as a private phone number. Identity and tax handling can be replaced by phone-number policy, which in most enterprises is a looser policy with a shorter retention floor and wider internal access.
Why the generalized case is easier to live with
A generalized label signals uncertainty. When a system returns a broad type, an engineering team can build for that: route those values to stricter default handling, queue them for review, or add an application-side resolution step. The output is telling you it does not know.
A specific incorrect label can be harder to manage, because a downstream system may confidently apply the wrong rule and there is no signal in the output that would cause anyone to check. That is the asymmetry. Both outcomes are classification failures. Only one of them is visible to the system consuming it.
This matters more as pipelines get more autonomous. In a manual review workflow, someone eventually looks at the underlying value and notices the mismatch. In an agentic pipeline where classification output feeds masking, routing, and access decisions with no person in the loop, the mismatch persists and nothing trips an alarm. The wrong rule is applied consistently to a specific slice of data.
Generalization was far more common than misclassification in our run: 28% of the 344 annotations, against 4%. The generalized case has its own operational cost, which we cover in PII redaction vs. PII classification.
Where we got it wrong
Protecto correctly classified 90% of the 344 annotations, which leaves 10% that were not correctly classified. Two specific misses are relevant to anyone building an adversarial test set. One email address written with Cyrillic homoglyph characters was not detected by Protecto. One phone number written in Arabic-Indic numerals was not detected either. On the edge case and adversarial prompt category, 43 annotations, Protecto correctly classified 79%. Character-spaced values, 6 annotations, came in at 67%.
We publish these because a classification layer that claims no blind spots is not describing a real system.
What to put in an adversarial test set
Detection rate on clean text will not surface any of this. Build the test set to be confusable on purpose.
Pair values that look like each other. A CVV next to a postal code. An IP address next to a URL. A nine-digit Social Security number next to a ten-digit phone number. Score whether the returned type is correct, not whether something was returned.
Include obfuscation that a person would still read correctly. Homoglyphs, zero-width characters, fullwidth digits, spaced digits. Across our 43 edge and adversarial annotations, PrivacyFilter correctly classified 5% and detected 19%. Protecto reached 79%. Both numbers moved sharply against the clean-text categories, which is the point of running them.
Score four outcomes, not two. Correct, generalized, misclassified or not detected are different failures with different remediation paths. Collapsing them into “found or missed” hides the thing you most need to see.
Check what the API hands back. In our run, Protecto returned rewritten protected text on 149 of 150 prompts. The tested PrivacyFilter interface returned spans, labels, and character offsets and did not return a rewritten version of the input. That difference concerns the tested API response contracts, not detection accuracy or model quality. It does mean an integrating team controls the transformation itself and needs an additional application-side step when rewritten text is required.
What this comparison does not establish
The run measured detection, classification, and the transformation the API returned. It did not evaluate architecture, extensibility, Protecto Vault capability, tenancy, access controls, deployment model, or operational integrations. Nothing here supports a claim about any of those. False positives were outside the scope, and latency, throughput, and cost were not measured.
How we ran this comparison
Protecto designed the prompt set, ran both APIs, wrote the scoring rules, and scored the results. The comparison was not independently validated and no third party reviewed it. Both products were called through their public APIs under default configuration, on the same input set, in a single run on 29 July 2026. The 150-prompt set deliberately includes difficult and adversarial cases, so these rates describe this test set rather than universal production performance. False positives were outside the scope, so none of these figures are precision or F-score measurements. The full prompt set, the expected annotations, the scoring rules, and both products’ raw API responses are available on request from sales@protecto.ai.
FAQ
What is AI data leakage?
It is the exposure of sensitive data through an AI system’s inputs, retrieved context, tool calls, memory, or outputs. The definition has widened as pipelines moved from blocking sensitive values to rewriting them and passing them forward, which added policy-enforcement failure to the list of ways data can end up outside its intended controls.
Is a misclassification a data breach?
Not on its own. It is a control failure. Whether it becomes an exposure depends on what the downstream rule does with the wrong type, which is why the consequence differs so much between a CVV routed into address handling and an IP address labeled as a URL.
How do I detect this in a system already in production?
Sample the classification output against a set of values whose types you already know, then compare returned types rather than counting detections. Misclassification does not appear in a detection-rate dashboard, so it has to be tested for directly.
Does a higher detection rate reduce AI data leakage risk?
It reduces one category of risk. It says nothing about whether the values that were detected carry a type your policy engine can act on correctly. Those are separate measurements and should be reported separately.
The full analysis, including the by-family and by-domain breakdowns, is in the pillar post, OpenAI PrivacyFilter vs Protecto: what 150 prompts and 344 annotations told us about classification.
To run this kind of test against your own pipeline, schedule a walkthrough or request the prompt set and scoring rules from sales@protecto.ai. More at www.protecto.ai.
