Case-Insensitive Comparison Needs Folding, Not Lowercasing

sort -fu deduplicates ignoring case. For ASCII that is sufficient; beyond it, lowercasing both sides is not the correct operation.

Case folding versus lowercasing

Unicode defines case folding specifically for caseless comparison, and it differs from lowercasing in edge cases. German ß folds to ss, so STRASSE and straße compare equal under folding and unequal under naive lowercasing.

The Turkish i, again

Lowercasing under a Turkish locale maps I to ı, so INDEX and index do not match. Case folding is locale-independent by design and avoids this entirely — which is why it exists as a separate operation.

Keep the original, key on the folded form

Store the folded value as the comparison key and retain the original for display. Replacing the data with its folded form loses information permanently, and folded text is not always presentable — it is a comparison artefact, not a formatting choice.

Try it: Remove Duplicates (Case-Insensitive) on SeoWolf's Notepad.