OpenAI PrivacyFilter vs Protecto: what 150 prompts and 344 annotations told us about classification
Most evaluations of a PII protection layer ask one question: did it find the sensitive data?
That was the right question when data loss prevention meant blocking. A DLP system that found a Social Security number stopped the email, and nothing downstream needed to understand what had been stopped.
AI workflows are different. Sensitive data can enter through prompts, tool calls, retrieved RAG context, agent memory, and model outputs, including paths that may sit outside traditional DLP inspection points. Blocking every document or prompt containing sensitive data is also impractical: it can interrupt the agent, prevent legitimate work, and make the application unusable. The data often needs to be protected while the workflow continues.
The consequential question is now the second one: did the protection layer name the data type precisely enough for a policy engine to select a rule?
We ran OpenAI PrivacyFilter and Protecto against the same 150 enterprise prompts to measure that difference. Across 344 expected annotations, PrivacyFilter detected roughly 68% under some label. Only 37% came back with a data type specific enough to act on. Protecto correctly classified 90% under the same documented scoring rules.
The distance between 68% detected and 37% correctly classified is what this post is about.
Methodology and disclosure
This comparison was designed, run, and scored by Protecto. It was not independently validated. Both products were called through their public APIs under default configuration on the same input set on 29 July 2026. The 150-prompt set intentionally includes difficult and adversarial cases, so these rates describe this test set rather than universal production performance. False positives were outside the scope, so the results are not precision or F-score measurements. The full prompt set, expected annotations, scoring rules, and raw API responses are available on request from sales@protecto.ai.
Two further points belong here rather than in a footnote. The analysis was updated on 3 August 2026 after we found two rows in the source dataset with output columns transposed. Both were corrected before scoring. One correction lowered PrivacyFilter’s measured performance and the other raised it. And the 344 expected annotations were assigned before either product was run, which makes them the agreed basis of the comparison, not independently established ground truth.
What we tested
We wrote 150 prompts to look like what an enterprise AI system actually receives: pasted records, casual service phrasing, structured document formats, mixed languages, and a deliberate set of hard cases such as OCR-like text, partially masked identifiers, character-spaced digits, decoys, homoglyphs, zero-width characters, and fullwidth digits.
Before calling either API, we annotated each prompt with the expected sensitive value and its correct data type, which gave us 344 expected annotations across 51 data types. Each was scored into one of four outcomes:
| Outcome | Definition |
| Correctly classified | Detected and assigned the correct specific data type |
| Generalized | Detected, but the returned type is too broad to select a specific rule |
| Specifically misclassified | Detected and assigned a specific type that contradicts the value |
| Not detected | No span, label, or output returned |
The headline result, with the nuance attached
Across the 344 expected annotations:
| Outcome | PrivacyFilter | Protecto |
| Correctly classified | 37% | 90% |
| Generalized | 28% | n/a |
| Specifically misclassified | 4% | n/a |
| Not detected | 31% | n/a |
| Not correctly classified | n/a | 10% |
Read the middle two rows before the top one. PrivacyFilter returned some label for about 68% of the values we checked, so it is a capable detector. For 28% the type was too broad for a downstream system to use, and for another 4% it was a specific type that contradicted the value.
So the accurate statement is not that PrivacyFilter missed most of the data. It found most of it and did not name most of it. A span without a usable type only tells the pipeline that something is probably sensitive, not which retention window, masking strictness, residency rule, or access policy applies.
The four layers, and the one we did not measure
We think about runtime data protection in four layers. This comparison covers three of them.
| Layer | What it produces | What depends on it | Evaluated here? |
| 1 · Detection | A span of sensitive text | Whether the protection layer sees the value at all | Yes |
| 2 · Classification | A specific data type | Which policy rule a system can select | Yes |
| 3 · Transformation | Protected text the application can consume | Whether the pipeline runs without a custom substitution layer | Yes |
| 4 · Enterprise control | Governed re-identification, audit, tenancy | Whether it can be deployed in a regulated environment | No |
Layer 4 is a hard boundary on what this report can say. API output shows observed detection, classification, and masking behavior. It shows nothing about architecture, extensibility, vault capability, tenancy, access controls, deployment model, or operational integrations, for either product. Protecto Vault, Context-Based Access Control (CBAC), and GPTGuard MCP were not part of this test, so none of this is a Layer 4 comparison. If those properties matter to you, test them in a proof of concept.
Why detection alone stopped being enough
Three things now depend on what the protection layer returns.
The model still has to answer the question. If a prompt says “the April invoice is wrong” and the layer swaps “April” for a generic person tag, the model can’t identify the invoice. Nothing leaked, and the answer is still wrong.
The policy engine still has to pick a rule. Retention, redaction strictness, access, and regional handling all key off the data type, and a generalized label gives the pipeline nothing to select on.
The audit trail still has to hold. A reviewer will ask which fields were protected, under which classification, and why. “The system flagged it as sensitive” doesn’t answer that.
The gap widens as the input gets harder
| Category | Prompts | Annotations | Protecto correct | PrivacyFilter correct | PrivacyFilter detected (any label) |
| Production conversational | 92 | 164 | 89% | 38% | 78% |
| Document and form-style | 9 | 82 | 94% | 40% | 68% |
| Dense multi-entity | 5 | 24 | 92% | 46% | 62% |
| Multilingual and code-switched | 15 | 31 | 97% | 55% | 84% |
| Edge case and adversarial | 29 | 43 | 79% | 5% | 19% |
On ordinary support and internal-request phrasing the gap is already 89% against 38%. On the adversarial set, PrivacyFilter correctly classified 5% of 43 annotations and detected 19% under any label. A test built only from tidy single-entity text would understate the difference an enterprise sees in production.
Where the gap concentrates: regulated identifiers
Breaking the same 344 annotations down by PII family shows the weakness isn’t evenly spread.
| PII family | n | PF correct | PF generalized | PF misclassified | PF not detected | Protecto correct |
| Contact and person | 145 | 69% | 1% | 0% | 30% | 87% |
| Government and national IDs | 72 | 0% | 75% | 7% | 18% | 90% |
| Financial and payment | 45 | 9% | 51% | 11% | 29% | 98% |
| Healthcare and PHI | 40 | 15% | 10% | 2% | 72% | 92% |
| Technical and asset | 42 | 38% | 29% | 7% | 26% | 88% |
The healthcare row totals 99% after rounding.
Government IDs are the clearest case. Of 72 government and national ID annotations, PrivacyFilter correctly classified none. That wasn’t a detection failure: 75% came back generalized and only 18% were not detected. The values were found and never named.
Healthcare goes the other way. 72% of the 40 healthcare and PHI annotations were not detected at all, because clinical language doesn’t look like an identifier. Two different problems, two different fixes, and a single recall number would have hidden both.
Grouped by enterprise domain, the pattern holds. These groupings are analytical and can overlap with the families above, so don’t add them together.
| Domain | n | PrivacyFilter correct | Protecto correct |
| Banking and financial | 110 | 30% | 93% |
| Healthcare and clinical | 72 | 33% | 93% |
| Insurance and claims | 31 | 19% | 87% |
| Identity and KYC | 26 | 42% | 96% |
| IT and security | 17 | 24% | 76% |
One more domain group was omitted because its denominator was too small.
The generalized-label problem, inside one label
PrivacyFilter returned 111 ACCOUNT_NUMBER labels in this run. That is a raw label count, not comparable with the 344-annotation scorecard, and the two should never be combined. This is what was inside:
| Composition | Count |
| Actual bank account numbers | 13 |
| Identifiers Protecto classified across approximately 22 other specific types | 92 |
| No corresponding Protecto value | 6 |
| Total | 111 |
The 92 included 17 Social Security numbers, 9 India PAN numbers, 8 credit cards, 7 passport numbers, 7 national IDs, 6 driving license numbers, 5 vehicle identification numbers, and 3 routing numbers.
Not all 111 labels are wrong. For 13 of them ACCOUNT_NUMBER is correct, and we scored those as correct. The problem is one label carrying more than twenty data types with different handling requirements. Here is the cost on one line of synthetic input:
New employee Sarah Johnson’s PAN is AAAPS4321Q and her bank account for salary credit is 100234567890.
| Tax identifier | Salary account | |
| Protecto | PAN | BANK_ACC_NO |
| PrivacyFilter | ACCOUNT_NUMBER | ACCOUNT_NUMBER |
A payroll system consuming that output can’t tell which value is the destination account. Both are sensitive, but only one is a payment instruction, and the label doesn’t separate them.
When the label is specific and wrong
Fourteen expected annotations came back with a specific type that contradicts the value.
| Expected type | PrivacyFilter output type | Annotations |
| IP address | PRIVATE_URL | 3 |
| US Social Security number | PRIVATE_PHONE | 3 |
| Card CVV | PRIVATE_ADDRESS | 2 |
| Payment card | PRIVATE_DATE or PRIVATE_PHONE | 2 |
| Aadhaar | PRIVATE_PHONE | 1 |
| Bank account | PRIVATE_PHONE | 1 |
| India PAN | PRIVATE_PERSON | 1 |
| Medical code | PRIVATE_ADDRESS | 1 |
These are annotations, not test cases, and 14 is a small count that we won’t inflate. It matters for another reason. A generalized label signals uncertainty, so a downstream system can escalate, quarantine, or apply the strictest rule. A specific wrong label signals confidence. Nothing flags it for review, and the pipeline applies the wrong rule.
Consequences from the report:
- IP address returned as PRIVATE_URL: indicator search, threat correlation, geolocation, and IP-specific retention logic can fail.
- Card CVV returned as PRIVATE_ADDRESS: card-security data can be routed into address handling instead of card-specific controls.
- US Social Security number returned as PRIVATE_PHONE: identity and tax handling can be replaced by phone-number policy.
Clinical language: 19 of 21 against 0 of 21
We isolated 21 annotations covering clinical concepts.
| Concept | Expected | Protecto | PrivacyFilter |
| Medical condition | 6 | 5 | 0 |
| Medical procedure | 4 | 4 | 0 |
| Blood type | 4 | 4 | 0 |
| Medication | 3 | 3 | 0 |
| Health beneficiary number | 3 | 2 | 0 |
| Biometric data | 1 | 1 | 0 |
| Total | 21 | 19 | 0 |
Clinical language isn’t identifier-shaped. A diagnosis, a procedure name, or a blood type has no fixed pattern, checksum, or length rule, and “Patient was prescribed metformin” is a sentence, not a format. A system tuned to structured identifiers can do well on Social Security and card numbers and still return nothing for a whole protected class. Recognizing clinical language takes semantic classification, not format matching. If your workload touches PHI, reproduce this row on your own data first.
Payment cards: found, but not typed
Sixteen expected payment-card number annotations were scored separately from CVVs and expiry dates.
| Result | |
| Protecto correctly classified | 16 of 16 · 100% |
| PrivacyFilter detected | 9 of 16 · 56% |
| PrivacyFilter correctly classified | 0 of 16 · 0% |
Card numbers are among the most format-regular data in the set. PrivacyFilter located most of them and none arrived with a payment-card type. Detection without the correct type leaves a card-security control nothing to select on. It is the cleanest case we have of locating a numeric span without identifying what it is.
Span granularity and the context the application still needs
The two products also divided text differently. Take a composite address:
My driving license number is DL-0420110149646, renewed last year, and my current address is 45 Baker Street, London, NW1 6XE.
| Protecto | PrivacyFilter |
| DL-0420110149646 → DRIVING_LICENSE_NUMBER | DL-0420110149646 → ACCOUNT_NUMBER |
| 45 Baker Street → ADDRESS | 45 Baker Street, London, NW1 6XE → PRIVATE_ADDRESS |
| London → CITY | |
| NW1 6XE → PINCODE |
A merged span is not unsafe in itself. It reduces the policy granularity available and may remove context the application still needs. With separate spans you can mask the street and the license number while keeping city and postcode for residency, regional entitlement, or fraud analysis. One merged span forces an all-or-nothing choice on the whole string.
A note on the API response contract
Protecto returned rewritten, protected text on 149 of 150 prompts. The tested PrivacyFilter interface returned detected spans, labels, and character offsets but no rewritten input. That difference concerns the tested API response contracts, not detection accuracy or model quality. Returning spans leaves the transformation to the integrating application, which then needs its own step when rewritten text is required. Which you want depends on your architecture.
On replacement consistency, 344 of 347 measured replacements were consistent across the 150 prompts (a count of replacements, unrelated to the 344 expected annotations). Replacements kept the shape of structured identifiers and were weaker on dates and IP addresses, where separators stayed but character classes and field widths did not.
A second cut of the data, and why we score it against ourselves
A secondary analysis in the report uses a different denominator: 351 reference values that Protecto identified. It doesn’t combine with the 344-annotation scorecard.
| Result | |
| Protecto | 351 of 351 |
| PrivacyFilter not detected | 87 |
| PrivacyFilter detected, not identified precisely enough for policy action | 149 |
| PrivacyFilter detected and identified | 115 · 33% |
The 351 of 351 is coverage by definition within this internal comparison, not independently validated recall. Protecto identified all 351 values used to build the reference set, so the set can’t contain a value Protecto missed. It is not a 100% accuracy claim and shouldn’t be quoted as one.
The more useful disclosure is how we handled ambiguity. Three scoring rules cover ambiguous cases, and each one resolves in PrivacyFilter’s favor:
- Label synonyms scored as correct. PRIVATE_EMAIL counted as email.
- ACCOUNT_NUMBER scored as correct wherever it was applied to an actual bank account. That is the 13 instances above.
- Span overlap treated as sufficient for detection.
Remove all three rules and PrivacyFilter’s correctly identified total in this cut falls from 115 to 2. We kept the rules, and we are telling you they exist because the generous reading is the one we published.
What we got wrong
Protecto made errors in this run:
- One US Social Security number was classified as a Canadian SIN.
- One phone number written in Arabic-Indic numerals was not detected.
- One email address using a Cyrillic homoglyph was not detected.
These are specific cases, not a complete accounting of our misses. The aggregates say the same: Protecto correctly classified 79% of the 43 edge and adversarial annotations, so about one in five of the hardest cases did not come back with the right type. On the 6 character-spaced values it classified 67%. The 90% headline is what happened on this test set, in this run, under these rules. It is not a ceiling or a guarantee.
If you evaluate us, push hardest on the adversarial and character-spacing cases, where we are weakest.
What to actually test in a proof of concept
If you are evaluating any PII protection layer, including ours, this is the test design we would use. Nothing in it is specific to Protecto.
- Annotate before you run. Pull 100 to 200 prompts from your own logs and write down the expected value and data type for each, before any vendor sees them. The annotation is the whole experiment.
- Score the type, not the hit. Grade each result as correctly classified, generalized, misclassified, or not detected. One recall number hides the gap between the government-ID pattern and the healthcare pattern.
- Use your worst traffic. Pasted spreadsheet rows, OCR output, partially masked values, code-switched sentences. Clean text flatters every vendor equally.
- Trace one label end to end. Take the most common generalized label and list what is inside it. If one label carries twenty data types, your policy engine has twenty rules it can’t select.
- Measure false positives separately. We didn’t, and you should. Over-flagging breaks prompt utility as surely as under-flagging breaks compliance.
- Check that the answer survives. Run the protected prompt through your actual model and grade the response.
- Test Layer 4 directly. Re-identification controls, audit records, tenancy isolation, residency, and deployment model can’t be inferred from API output. Get answers in writing and test them yourself.
The question we would put first: is the type this system returns the type I would have assigned myself? Everything downstream depends on the answer.
