๐Ÿ“š General & Other

Intersecting Many Lists Without Quadratic Cost

gives the intersection of two sorted files. Beyond two, the approach matters.

comm -12 a.txt b.txt gives the intersection of two sorted files. Beyond two, the approach matters.

Reduce pairwise, smallest first

Intersecting n sets is a fold: intersect the first two, then that result with the third, and so on. Starting with the smallest sets keeps the intermediate result small, and since an intersection can only shrink, every subsequent comparison gets cheaper.

Counting occurrences is often simpler

For "appears in at least k of n lists", concatenate all lists with duplicates removed within each, then count occurrences: sort | uniq -c | awk '$1>=k'. One pass, no repeated pairwise work, and it answers the more useful question directly โ€” the strict intersection is just the case where k equals n.

Hash-based intersection

Where inputs are unsorted, load the smallest into a hash set and stream the others against it. Memory scales with the smallest input rather than the largest, which is the opposite of the naive ordering and frequently the difference between fitting in memory and not.

Try it: List Intersection (Common Lines) on SeoWolf's Notepad.