TLDR
Episode 2 of PAPER TRAIL introduces the 16-script processing pipeline that converted 2.1 million raw government documents into structured, searchable data — extracting 1.18 million entities at the time of filming, revealing a 1.55:1 organization-to-person ratio, and estimating via Chao1 that over 468,000 entities remained unseen (PAPER TRAIL Project, 2026a).
What the Episode Covers
EP02 walks through the computational backbone of the project: a directed acyclic graph of 16 Python scripts organized into 8 processing stages. Stage 1 ingests raw metadata into PostgreSQL. Stage 2 converts PDFs and images to searchable markdown via OCR. Stage 3 runs named entity recognition using spaCy's en_core_web_lg model across 8 parallel workers. Later stages deduplicate entities, build co-occurrence graphs, compute temporal density curves, and reprocess degraded documents through a vision-language model on a consumer GPU (PAPER TRAIL Project, 2026b).
At the time of filming, the pipeline had processed 40.8% of the corpus — 857,000 documents — extracting 1.18 million entities linked by 29.5 million relationship pairs. The entity composition showed a striking inversion: 57% organizations, 37% persons, 6% locations, yielding a 1.55:1 organization-to-person ratio that reflects Epstein's reliance on corporate structures (PAPER TRAIL Project, 2026a).
The Blind Spots
EP02 identifies two critical gaps in the processing state. Data Sets 9 and 11 — containing 863,000 email documents — sat at 0% NER coverage, the largest unprocessed sub-corpus. The episode flags these as "the richest veins unmined," a gap later closed when NER completed across all email documents by EP10 (PAPER TRAIL Project, 2026c).
The second blind spot is structural: 197,945 entities appeared exactly once in the entire corpus — singletons representing 24.1% of all extracted entities. Some are OCR errors. Some are aliases. Some are real individuals mentioned in a single document. The pipeline cannot distinguish between these categories without additional context, making the singleton population a persistent source of uncertainty (PAPER TRAIL Project, 2026a).
The Chao1 Estimate
The episode's most consequential number is not what the pipeline found but what it estimated was missing. The Chao1 species richness estimator — originally developed in ecology to estimate the total number of species from partial samples — projected 1,290,141 total entities from 821,633 observed, meaning 63.7% of the entity population had been surfaced and 468,000 remained unseen (Chao, 1984; PAPER TRAIL Project, 2026d).
The three-step formula is presented visually: count singletons (f1 = 197,945), count doubletons (f2 = 41,816), compute the estimate. The result reframes the entire project: every finding derived from the corpus is drawn from roughly two-thirds of the available evidence. The remaining third could confirm, contradict, or transform any conclusion.
Why This Episode Matters
EP02 establishes the evidentiary foundation for every episode that follows. The pipeline is not a black box — every script, every parameter, every threshold is named and explained. The processing gaps are disclosed before the findings are presented. And the Chao1 estimate sets the project's epistemic ceiling: no finding from this corpus can claim completeness. The pipeline knows what it has found. It also knows, approximately, how much it has not.
References
Chao, A. (1984). Nonparametric estimation of the number of classes in a population. Scandinavian Journal of Statistics, 11(4), 265–270.
PAPER TRAIL Project. (2026a). EP02 slide content: Entity composition, singleton analysis, Chao1 estimation [Presentation]. communications/ep02_slides/
PAPER TRAIL Project. (2026b). Processing pipeline: 16 scripts in 8 stages [Computer software]. app/scripts/
PAPER TRAIL Project. (2026c). NER processing status: DS9 and DS11 email completion [Data set]. app/scripts/04_extract_entities.py
PAPER TRAIL Project. (2026d). Chao1 species richness estimation [Data set]. _exports/validation/chao1_summary.json
This research is sponsored by Subthesis.