Teams usually plan to de-identify PHI as a detection problem. Find the identifiers, mask or remove them, pass the rest to the model. For medical record numbers, beneficiary numbers, and Social Security numbers, that plan mostly works. Those values have a shape a pattern matcher can learn. Most of the protected content in a clinical AI workflow does not have a shape at all.
Short answer. To de-identify PHI in an AI workflow you need two capabilities, not one. Pattern matching handles structured identifiers such as MRNs and beneficiary numbers. Semantic classification handles clinical language, because a diagnosis, a procedure, or a blood type carries no fixed format to match against. A layer built on only the first has a category-shaped blind spot.
We ran a comparison between OpenAI PrivacyFilter and Protecto in July 2026 and scored the clinical cases separately. The results below are from that run, which we designed and scored ourselves. The methodology section states what it does and does not establish.
What de-identification actually requires of a classification layer
HIPAA provides two de-identification methods. Both require organizations to understand what information is present; accurate classification can support that process. This is a description of the rule, not legal advice.
Safe Harbor requires removing 18 specified categories of identifiers relating to the individual and to relatives, employers, and household members, and requires that the covered entity has no actual knowledge that the remaining information could identify the person. The list is category-specific. Health plan beneficiary numbers, account numbers, medical record numbers, and biometric identifiers are separate categories. A layer that returns one generic tag for all of them has not told you which category you satisfied.
Expert Determination requires a person with appropriate knowledge and experience applying generally accepted statistical and scientific principles to conclude that the re-identification risk is very small, and to document the methods and results. That determination depends on knowing what is in the data, including the clinical detail, because rare conditions and unusual procedures carry re-identification risk in combination with quasi-identifiers.
One point is worth stating precisely: a diagnosis is not one of the 18 Safe Harbor identifier categories. It still matters to the classification layer, because it carries re-identification risk under an expert determination, and because most organizations have their own rules about what clinical detail may reach a third-party model.
Clinical language has nothing to match against
“Patient was prescribed metformin for type 2 diabetes” is a sentence. Recognizing it as protected health information requires understanding what the sentence means. There is no character pattern, no checksum, no field width.
We scored 21 expected annotations covering clinical concepts. Protecto correctly classified 19 of the 21 while PrivacyFilter correctly classified none of the 21 healthcare-related annotations in this test.
| Concept | Expected | Protecto | PrivacyFilter |
|---|---|---|---|
| Medical condition | 6 | 5 | 0 |
| Medical procedure | 4 | 4 | 0 |
| Blood type | 4 | 4 | 0 |
| Medication | 3 | 3 | 0 |
| Health beneficiary number | 3 | 2 | 0 |
| Biometric data | 1 | 1 | 0 |
| Total | 21 | 19 | 0 |
Privacy Filter does not claim clinical-language classification, so its 0-of-21 result should be read as a difference in product scope, not accuracy. For enterprises processing clinical content, however, it remains a meaningful coverage gap.
The wider healthcare picture from the same run
The clinical set is 21 annotations. Two larger cuts of the same 344-annotation scorecard point the same way, and each has its own denominator.
Grouped by PII family, the healthcare and PHI row covers 40 annotations. PrivacyFilter correctly classified 15%, generalized 10%, misclassified 2%, and did not detect 72%. Protecto correctly classified 92%. The row totals 99% after rounding.
That 72% is the number to sit with. In the government ID and financial families, PrivacyFilter’s dominant failure was generalization: it found the value and returned a type too broad to act on, at 75% and 51% of those rows. In healthcare, the dominant failure was that no span, label, or output came back at all.
Grouped by enterprise domain, the healthcare and clinical group covers 72 annotations. PrivacyFilter correctly classified 33%. Protecto correctly classified 93%. That group includes structured healthcare identifiers alongside narrative content, which is why it sits well above the 15% in the family view.
Our document and form-style prompts, which include discharge summaries and claims, carried 82 annotations across 9 prompts. PrivacyFilter correctly classified 40% of those 82 and detected 68% of those 82. Protecto correctly classified 94%.
We also recorded one misclassification in this territory: a medical code returned as PRIVATE_ADDRESS. One annotation, and a clean illustration of why a specific wrong type is harder to catch than a missing one. That pattern is covered in why wrong labels are the AI data leakage risk.
Why this compounds in RAG and clinical agents
A single miss in a one-time document review is a defect. The same miss in a retrieval pipeline repeats. The discharge summary that was not de-identified correctly gets chunked, embedded, retrieved, and passed into a model on every relevant query, and it stays in the index until someone re-runs the pipeline. Agent memory behaves the same way. The further into the pipeline a value travels before it is classified, the more copies of the mistake exist.
Where Protecto fell short
Protecto correctly classified 19 of the 21 healthcare-related annotations spanning clinical concepts and health identifiers. The two errors were one medical condition and one health beneficiary number. Health plan beneficiary numbers are listed as a Safe Harbor identifier category, making that miss relevant to HIPAA de-identification.
Elsewhere in the run, Protecto did not detect one phone number written in Arabic-Indic numerals. On the edge case and adversarial prompt category, 43 annotations, Protecto correctly classified 79%.
If you are running a POC in a regulated environment, these are the numbers to ask any vendor for, including us.
What to test in a healthcare POC
Test narrative text, not a field list. Feed real discharge summary structure: a paragraph containing a condition, a medication, a procedure, and a beneficiary number in running prose. Score each expected value separately.
Ask for the breakdown by data type. An aggregate accuracy figure can carry a category sitting at zero. Ours does, on the PrivacyFilter side, for clinical language.
Check which identifier category comes back. Safe Harbor is category-specific. Ask whether the tool distinguishes a health plan beneficiary number from a medical record number from an account number, or returns one generic tag for all three.
Test what survives the transformation. In our run, separate spans allowed masking of a street address and an identifier while retaining city and postcode, which a workflow may still need for residency, regional entitlement, or care analytics. A merged span is not unsafe in itself. It reduces the policy granularity available and can remove context the application still needs.
Test consistency across a long document. The same patient identifier appearing eight times in one record should map to the same replacement each time, or downstream joins break. In our run, replacements were consistent on all but three of the 347 measured replacements across the 150 prompts. That 347 is a count of replacements and is unrelated to the 344 expected annotations.
Test your own messy inputs. OCR artifacts, transcription errors, and partial masking are normal in healthcare data. Across our 9 OCR-like annotations, Protecto correctly classified 89%. PrivacyFilter detected 22% and correctly classified 0%. These edge-case groupings are analytical overlays on the same prompt set and should not be summed with each other.
What this comparison does not establish
The run measured detection, classification, and the transformation the API returned. It did not evaluate architecture, extensibility, Protecto Vault capability, tenancy, access controls, deployment model, or operational integrations, so nothing here supports a claim about how either product deploys in a regulated environment. It also did not measure false positives, latency, throughput, or cost. Nothing in this post is a determination that a given workflow meets a HIPAA de-identification standard. That belongs to your privacy office and, for expert determination, to a qualified expert.
How we ran this comparison
Protecto designed the prompt set, ran both APIs, wrote the scoring rules, and scored the results. The comparison was not independently validated and no third party reviewed it. Both products were called through their public APIs under default configuration, on the same input set, in a single run on 29 July 2026. The 150-prompt set deliberately includes difficult and adversarial cases, so these rates describe this test set rather than universal production performance.
False positives were outside the scope, so none of these figures are precision or F-score measurements. All prompts and values used in the comparison and in this post are synthetic. The full prompt set, the expected annotations, the scoring rules, and both products’ raw API responses are available on request from sales@protecto.ai.
FAQ
Is masking enough to de-identify PHI for an AI workflow?
Masking is the transformation. De-identification also requires knowing which identifier category each value belongs to, because Safe Harbor is written by category. A mask applied to an unclassified value tells you nothing about which obligation it satisfied.
Does clinical text need to be removed to de-identify PHI?
Diagnoses and procedures are not one of the specifically named Safe Harbor identifier categories. They do affect re-identification risk under expert determination. Your privacy office decides the standard. The classification layer decides whether you can implement it.
Why did a pattern-based tool score zero on clinical language?
Conditions, medications and procedures often lack fixed character formats. A blood type, a medication name, and a procedure name have no fixed character structure, checksum, or field width. Detecting them requires classifying meaning rather than matching format.
Can I combine pattern matching and semantic classification myself?
Yes, and teams do. The build cost is rarely the first version. It is maintaining type coverage as identifier formats change, keeping replacements consistent across long documents, and handling multilingual and obfuscated inputs, which is where our own results still show gaps.
The full analysis is in the pillar post, OpenAI PrivacyFilter vs Protecto: what 150 prompts and 344 annotations told us about classification, and the general classification argument is in PII redaction vs. PII classification. To walk through how Protecto handles structured PHI and clinical narrative text on your own records, schedule a walkthrough or contact sales@protecto.ai. More at www.protecto.ai.
