Technical Deep Dives
The Many Rodgers: How Entity Resolution Handles Scanning Chaos
TLDR A software tool that uses statistics to decide which database records refer to the same person (called Splink) merged eight or more scanning variants of...
#data quality
5 investigations
Data Quality
The Singleton Crisis: 197,945 Entities That Appear Once
TLDR Nearly one in four entities in the corpus (197,945 out of 821,633) appears in only one document. An entity that appears in only one document — called a...
Data Quality
Garbage Cluster Dissolution: When Entity Resolution Connects Noise to Noise
TLDR A cleanup script (Script 19b) dissolved 8 entity groups containing 3,576 records that had been incorrectly merged because unreadable text that the...
Observations
OBS-5/6: When Scanning Software Hallucinates Countries From Blank Forms
TLDR Two observations were retracted after visual inspection revealed that scanning software (OCR, or optical character recognition — software that reads text...
Machine Intelligence
OCR Hallucinations
TLDR OCR engines (software that converts images of text into searchable characters) produce phantom entities from blank form labels and repeated document...