πŸ“š General & Other

Finding the URL Patterns Hidden Behind Product IDs

A crawl of an ecommerce site returns forty thousand URLs, every one carrying a unique product ID. The individual URLs are meaningless; the patterns behind them are the site architecture.

A crawl of an ecommerce site returns forty thousand URLs, every one carrying a unique product ID. The individual URLs are meaningless; the patterns behind them are the site architecture.

Reducing to structure

Strip the digits and deduplicate. Forty thousand URLs collapse to perhaps twelve distinct patterns β€” the templates the site is actually built from.

What the patterns tell you

Twelve templates means twelve places where a fix applies to thousands of pages. It also exposes patterns nobody knew existed: an old URL structure still generating pages, or a duplicate route serving the same products under two paths.

Count pages per pattern

Group the original URLs by their stripped form and count. A pattern with thirty thousand pages and one with forty are very different problems, and the counts decide which template gets attention first.

Try it: Strip Digits from Text on SeoWolf's Notepad.