PII redaction and PII classification get used as if they were the same operation. They are not. PII redaction removes or replaces a value that a system has flagged as sensitive. Classification names what that value actually is, so a policy engine can select a rule for it. In an AI pipeline, the second step is the one that decides whether a policy can run at all.
Short answer. PII redaction removes or masks a value a system has flagged as sensitive. PII classification assigns that value a specific data type, such as passport number or routing number, so a policy engine can select the correct rule. Redaction without classification gives a pipeline something to hide and nothing to decide with.
We ran a comparison between OpenAI PrivacyFilter and Protecto in July 2026 to see how wide that gap gets in practice. The numbers below come from that run. It was our own comparison, not an independently validated one, and the methodology section at the end says exactly what it does and does not establish.
What PII redaction actually does
Redaction, in the sense most tools use the word, means locating a span of text that looks sensitive and replacing, flagging, or removing it. The output is close to binary: sensitive or not sensitive. Many tools attach a tag, something like PERSON or ACCOUNT_NUMBER, but the tag is often broad enough to cover a dozen unrelated data types.
That is enough for a narrow goal, such as keeping a value out of a log file or a screen share. It is not enough for a goal like “apply the correct retention period to this field” or “mask this for a support agent and reveal it to a fraud analyst.” Both of those decisions require knowing precisely what the value is.
What classification adds
Classification assigns a specific data type. Not “this is an identifier” but “this is a driving license number,” “this is a routing number,” “this is a card verification value.” A policy engine can only select a rule when it knows which rule applies, and rules differ by type. A driving license number and a bank account number are both identifiers. They can carry different retention, access, and jurisdiction-specific requirements.
This is why detection rate on its own is a weak buying signal, and why we scored classification separately.
What we measured
We built 150 synthetic enterprise prompts containing 344 expected annotations across 51 expected data types. Every annotation was assigned before either product was run. We then called both public APIs under default configuration on the same input set and scored each expected value into one of four outcomes.
Against those 344 expected annotations:
| Outcome | PrivacyFilter | Protecto |
| Correctly classified | 37% | 90% |
| Generalized (detected, type too broad to act on) | 28% | n/a |
| Specifically misclassified | 4% | n/a |
| Not detected | 31% | n/a |
| Not correctly classified | n/a | 10% |
PrivacyFilter detected roughly 68% of the 344 expected annotations under some label. Only 37% arrived with a data type precise enough for a policy engine to act on. The gap between those two figures is the redaction-versus-classification gap, measured.
The 111 ACCOUNT_NUMBER labels
The clearest illustration of that gap is a single label. In this run, PrivacyFilter returned 111 raw ACCOUNT_NUMBER labels.
| Composition of the 111 labels | Count |
| Actual bank account numbers | 13 |
| Identifiers Protecto classified across approximately 22 other specific types | 92 |
| No corresponding Protecto value | 6 |
Two caveats matter here. Not all 111 labels are incorrect, and 111 is a raw label count from PrivacyFilter’s output. It is not comparable with the 344-annotation scorecard above and should never be combined with it.
Inside that one label sat, among others, 17 Social Security numbers, 9 India PAN values, 8 credit cards, 7 passport numbers, 7 national IDs, 6 driving license numbers, 5 vehicle identification numbers, and 3 routing numbers.
Here is why that matters operationally. One of our synthetic prompts reads:
New employee Sarah Johnson’s PAN is AAAPS4321Q and her bank account for salary credit is 100234567890.
| Tax identifier | Salary account | |
| Protecto | PAN | BANK_ACC_NO |
| PrivacyFilter | ACCOUNT_NUMBER | ACCOUNT_NUMBER |
A payroll system consuming that output cannot tell which value is the destination account. Both values were found. Neither was named. Redaction succeeded and classification did not, and the downstream system is the thing that pays for it.
Government IDs: located, rarely named
Breaking the 344 annotations down by PII family shows the weakness is not evenly distributed.
| PII family | n | PF correct | PF generalized | PF misclassified | PF not detected | Protecto correct |
| Contact and person | 145 | 69% | 1% | 0% | 30% | 87% |
| Government and national IDs | 72 | 0% | 75% | 7% | 18% | 90% |
| Financial and payment | 45 | 9% | 51% | 11% | 29% | 98% |
| Healthcare and PHI | 40 | 15% | 10% | 2% | 72% | 92% |
| Technical and asset | 42 | 38% | 29% | 7% | 26% | 88% |
The healthcare row totals 99% after rounding.
Government and national IDs are the sharpest case. Across those 72 annotations, PrivacyFilter correctly classified 0%, generalized 75%, misclassified 7%, and did not detect 18%. These are passports, national IDs, tax identifiers, and driving licenses, the exact category where residency rules, retention schedules, and cross-border transfer restrictions differ by type. Redaction was working. Classification was not available to act on.
Healthcare behaves differently again: 72% of those 40 annotations were not detected at all. We cover that pattern in how to de-identify PHI in AI workflows.
Where we did not do well
Protecto correctly classified 90% of the 344 annotations, which means 10% were not correctly classified. Some specific misses are worth naming. One US Social Security number came back classified as a Canadian SIN, which is a precision failure inside the right family and would still send a value into the wrong jurisdiction-specific rule. On the edge case and adversarial prompt category, 43 annotations, Protecto correctly classified 79%. On character-spaced values, 6 annotations, 67%. No classification layer is finished, including ours.
How to test this before you buy
Detection rate is the easiest metric to produce and the least useful one to buy on. Three tests separate PII redaction from classification quickly.
Put three identifier types in one sentence. A driving license number, a bank account number, and a passport number in a single prompt. Check whether you get three distinct types back or one generic tag applied three times.
Ask for output on your own text. Production text is messy, code-switched, partially masked, and shaped by whatever form or scanner produced it. Clean demo strings hide most of the difference.
Ask for the breakdown by data type, not the aggregate. An aggregate score can carry a category sitting at zero. Ours did, on the PrivacyFilter side, for government IDs.
What this comparison does not establish
The run measured detection, classification, and the transformation the API returned. It did not evaluate architecture, extensibility, Protecto Vault capability, tenancy, access controls, deployment options, or operational integrations. Those are system-level properties and nothing here speaks to them. It also did not measure false positives, latency, throughput, or cost.
How we ran this comparison
Protecto designed the prompt set, ran both APIs, wrote the scoring rules, and scored the results. The comparison was not independently validated and no third party reviewed it. Both products were called through their public APIs under default configuration, on the same input set, in a single run on 29 July 2026. The 150-prompt set deliberately includes difficult and adversarial cases, so these rates describe this test set rather than universal production performance. False positives were outside the scope, so none of these figures are precision or F-score measurements. The full prompt set, the expected annotations, the scoring rules, and both products’ raw API responses are available on request from sales@protecto.ai.
FAQ
Is PII redaction the same as PII masking?
They overlap. Redaction usually means removing or blacking out a value. Masking usually means replacing it with a substitute that keeps some structure. Both are transformations. Neither one tells a downstream system what the value was.
Can I run a policy engine on redaction output alone?
Only for uniform policies. If every sensitive value gets the same treatment regardless of type, redaction is sufficient. As soon as retention, residency, or role-based access differs by data type, the policy engine needs the type.
Does a generalized label mean the tool failed?
No. A generalized label means the value was found and the returned type was too broad for a system to select a specific rule on. That is a different outcome from a miss, and in some pipelines it is recoverable with an application-side step. It is not the same as an actionable classification.
What is the difference between generalized and misclassified?
Generalized means the type returned was too broad. Misclassified means the type returned contradicts what the value is. We recorded 14 specific misclassifications in this run and cover them in why wrong labels are the AI data leakage risk.
If you want to run this test against your own production text, schedule a walkthrough or request the prompt set and scoring rules from sales@protecto.ai. More at www.protecto.ai.
