Model-agnostic PII detection with LLMs

Fine-tuning a model on real-world text creates a personally identifiable information (PII) detection problem. Training corpora are full of PII: names, home addresses, email and phone numbers, national-ID and social-security numbers, bank accounts, dates of birth.

What changed

A model trained on uncleaned text can memorize that data and later reproduce it, leaking a real person’s details through a prompt that was never meant to surface them. In this post, we describe a configurable, model-agnostic detector built on large language models (LLMs), walk through its implementation, benchmark it against an off-the-shelf tool, and show how to run it on your own data.

Price and bundle value

Sample code: The detector described in this post ships as the pii-detector package, available in the sample-llm-pii-detection repository . Every code snippet that follows is drawn from that package, and the Running the detector end to end section walks through installing and running it on your own data.

AI features that change the value

Large language models reframe the problem. An LLM reads its instructions at inference time, so the entities to detect, the output format, and the deployment backend all become configuration rather than code. One detector can target a new entity type by editing a prompt instead of retraining, run on a managed API or inside your own virtual private cloud (VPC), and reason about context across eight languages without a translation step. The rest of this post describes such a detector, walks through the engineering behind it, and shows how it measures up against existing tools.

Who should pay attention

The detector treats the language model as a configurable, swappable component. You wrap the input text in instructions that define the PII entities to detect and the expected output. The model then returns a structured list of detected entities. Two design choices make it model-agnostic:.

Patrick Tech Media take

Customization comes from two independent components. The first is the model, which sets accuracy, latency, and cost: you choose a frontier Amazon Bedrock model or a small open model on a single GPU. The second is the entity set, which defines what counts as PII. To extend it, you add a domain-specific identifier or drop one that you don’t need. Changing the entity set is a one-line edit to the instructions, with no retraining and no redeployment.

Source notes

AWS ML Blog