๐Ÿ“š General & Other

The Second Deduplication Pass Everyone Forgets

A list deduplicated once still contains duplicates. Exact matching removed the identical rows and left every variant that differs by a capital letter.

A list deduplicated once still contains duplicates. Exact matching removed the identical rows and left every variant that differs by a capital letter.

The layered approach

Run exact deduplication, then case-insensitive, then sort words within lines and deduplicate again. Each pass catches a different class of repeat, and each finds fewer than the last.

What the passes typically remove

Exact matching takes the largest share. Case-insensitive frequently removes another ten to twenty per cent on a multi-source list. Word-order normalisation catches a smaller but genuinely distinct set.

Report the final number

The count after all three passes is the real size of the opportunity. Reporting the figure after one pass overstates it, and that overstatement gets built into plans and budgets that later have to be revised downward.

Try it: Remove Duplicates (Case-Insensitive) on SeoWolf's Notepad.