LLM Anonymization on the TAB Dataset

1 Background & Motivation

Text anonymization is a critical task in NLP and privacy engineering. The goal is to remove or replace personally identifiable information (PII) from text while preserving the document's meaning and utility.

Recent research has shown that Large Language Models (LLMs) like GPT-4 are surprisingly effective at this task; sometimes outperforming traditional rule-based and NER-based approaches.

2 The Original Project

The base project implements the paper "Large Language Models are Advanced Anonymizers" (Staab et al., ICLR 2025). It works on Reddit comments and performs attribute-level anonymization.

Original Pipeline

What It Does

3 The TAB Dataset

The Text Anonymization Benchmark (TAB) is an open-source corpus developed by Pilán et al. (2022) for evaluating text anonymization systems. It consists of 1,268 English-language court cases from the European Court of Human Rights (ECHR).

Dataset Structure

Entity Types Annotated

Annotation Labels

4 How We Adapted the Pipeline

The original project does attribute-level anonymization on short Reddit comments. TAB requires entity-level anonymization on long legal documents. Here is how we bridged the gap:

Key Differences

New Pipeline

What We Reused vs. What We Built New

5 Architecture & Code Structure

Files Created

Key Component: Data Loader

The TAB dataset uses a specific JSON format where annotations are nested under annotator keys:

// TAB JSON structure
{
  "doc_id": "001-61807",
  "text": "PROCEDURE\nThe case originated...",
  "annotations": {
    "annotator1": {
      "entity_mentions": [
        {
          "entity_type": "CODE",
          "span_text": "36110/97",
          "identifier_type": "DIRECT",
          "start_offset": 54,
          "end_offset": 62
        }
      ]
    }
  }
}

Our parser extracts all entity mentions from this nested structure:

def parse_document(doc_json):
    annotations = []
    for ann_key, ann_data in doc_json["annotations"].items():
        # annotations → annotatorN → entity_mentions → [list]
        entity_mentions = ann_data.get("entity_mentions", [])
        for mention in entity_mentions:
            annotations.append(EntityMention(
                entity_type=mention["entity_type"],
                span_text=mention["span_text"],
                identifier_type=mention["identifier_type"],
                ...
            ))
    return TABDocument(doc_id=..., text=..., annotations=annotations)

Key Component: Document Chunking

Court documents average ~5,000 characters . It is too long for a single LLM prompt. We split at paragraph boundaries into chunks of ≤3,500 characters, tracking which annotations belong to each chunk:

def chunk_document(doc, max_chars=3500):
    if len(doc.text) <= max_chars:
        return [(doc.text, doc.annotations)]  # Fits in one chunk

    chunks = []
    for paragraph in doc.text.split("\n"):
        if current_chunk_too_long:
            chunks.append((current_chunk, entities_in_range))
            start_new_chunk()
    return chunks

6 Prompt Engineering (3 Levels)

Following the original paper's approach, we designed three prompt levels of increasing sophistication, mirroring the original Level 1 (naive), Level 2 (intermediate), and Level 3 (CoT + expert) structure. (in src/tab/tab_anonymize.py)

Level 1 — Naive

Level 2 — Intermediate

Level 3 — Advanced (Chain-of-Thought)

Prompt Structure (Level 3)

System Prompt:
  You are an expert legal document anonymizer...
  Replacement categories: [PERSON], [LOC], [ORG], [DATETIME]...
  Rules: Replace DIRECT always, QUASI when risky...

User Prompt:
  Header: Below is an excerpt from an ECHR case...
  Document excerpt:
    <actual text chunk>
  Identified entities:
    - "Mr Galip Yalman" (PERSON, DIRECT)
    - "36110/97" (CODE, DIRECT)
    - "Ankara" (LOC, QUASI)
  Footer: First explain, then write # and anonymized text.

7 Evaluation Metrics

We evaluate two aspects: privacy protection (did we mask the right things?) and text utility (is the text still useful?).

Privacy Metrics

Utility Metrics

Evaluation Logic

def evaluate_entity_detection(doc, anonymized_text):
    for entity in doc.entities_to_mask:
        # If the original span text STILL appears → missed
        if entity.span_text in anonymized_text:
            overall["missed"] += 1
        else:
            overall["masked"] += 1  # Successfully anonymized!

    recall = masked / total  # Higher = better privacy

8 Worked Example

Here's what anonymization looks like on a real TAB document excerpt:

9 How to Run

Prerequisites

# Only two dependencies needed
pip install openai==0.28.1
pip install pyyaml  # only if using --config flag

Step 1: View Dataset Statistics (No API Key Needed)

cd llm-anonymization
python run_tab.py --stats_only --split test

Step 2: Run Anonymization

# Anonymize 5 documents with GPT-4o
python run_tab.py --model gpt-4o --split test --max_docs 5

# Compare prompt levels
python run_tab.py --model gpt-4o --prompt_level 1 --max_docs 10
python run_tab.py --model gpt-4o --prompt_level 3 --max_docs 10

# Use a config file
python run_tab.py --config configs/anonymization/tab.yaml

Step 3: Evaluate Results

python run_tab.py --evaluate --results_path anonymized_results/tab/results.jsonl --split test

Output Files

10 Summary

Key Technical Decisions

Potential Extensions

• Compare multiple models (GPT-4o vs GPT-3.5 vs open-source LLMs)
• Add BLEU/ROUGE scores for richer utility evaluation
• Implement iterative anonymization (multiple passes like the original pipeline)
• Add the official TAB evaluation script for direct comparison with published baselines