Price monitoring answers one question. Ecommerce intelligence answers the rest of them: what products exist in a category, who sells them, how they're described and positioned, what customers say about them, which sellers are gaining ground, what's in stock, what's been discontinued, and how all of that differs by market.
It's the broadest of the commerce data disciplines and the one where the collection is easiest and the analysis is hardest. Getting the pages is a solved problem. Turning millions of listings across a dozen retailers into a coherent picture of a category — where the same product appears under six names, sellers come and go, and half the signal is in what's absent — is where the actual work lives.
This guide covers what ecommerce data can tell you, how to structure collection around a catalogue rather than a page list, the entity resolution problem that determines whether any of it is comparable, review and sentiment analysis, and how to build a programme that produces decisions rather than dashboards.
What Ecommerce Data Actually Tells You
Being precise about this prevents most of the analytical mistakes downstream.
What it observes well:
- Assortment. What products exist in a category, who carries them, and how ranges differ between retailers and markets.
- Pricing and promotion. Covered in depth elsewhere, and one input among several here.
- Availability. Stock status, delivery estimates, and how these move over time.
- Product presentation. Titles, descriptions, images, attributes, and how sellers position the same item differently.
- Seller landscape. Who's selling on marketplaces, their ratings, their catalogue breadth, and how that changes.
- Customer expression. Review text, ratings, and question-and-answer content.
- Merchandising behaviour. What gets featured, bundled, cross-sold, and promoted.
- Change over time. New products appearing, old ones disappearing, ranges expanding or contracting.
What it does not observe:
- Sales volume. Review counts and rankings are proxies with unknown and inconsistent conversion rates. They are not sales.
- Margin. You see prices, never costs.
- Inventory depth. "In stock" rarely tells you how many.
- Wholesale and B2B transactions, which are negotiated rather than listed.
- Customer identity or motivation. You see what reviewers wrote, not who bought or why.
- Anything offline. For many categories this is most of the market.
The discipline that matters: be explicit about which of these you're measuring. A great deal of weak ecommerce analysis consists of treating review velocity as sales data, and the resulting charts look authoritative while meaning something other than their labels claim.
Structuring Collection Around a Catalogue
The architectural difference between ecommerce intelligence and general scraping: you're maintaining a picture of a changing catalogue, not fetching a fixed list of pages.
Discovery is a first-class workload. New products appear constantly. A collection system that only refreshes a known URL list will slowly diverge from reality, and the divergence is invisible from inside the data. Run category traversal, sitemap parsing, search-based discovery, and seller catalogue enumeration as ongoing processes, not one-time setup.
Track the universe, not just your tracked set. Knowing that a category contains four thousand products of which you monitor three hundred is different from monitoring three hundred and assuming that's the category. Coverage ratio is a metric worth maintaining.
Detect disappearance explicitly. Products get delisted, sellers exit, and ranges shrink. Absence is signal, and it only exists if you were tracking presence. A pipeline that just fetches what's there will never tell you what left.
Separate discovery from extraction. Finding what exists and collecting its details are different workloads with different frequencies, failure modes, and cost profiles. Coupling them means a discovery problem looks like an extraction problem.
Version the catalogue. A snapshot of what existed on a date, so you can answer "what changed" rather than only "what is."
Handle variants deliberately. A product with twelve size and colour combinations may be one page or twelve, depending on the retailer. Deciding your unit of observation — product, variant, or listing — before building determines whether cross-retailer comparison is possible.
> Tip: The most common structural mistake is treating ecommerce collection as a URL list refresh. It's a catalogue synchronisation problem, and the difference shows up as slowly worsening coverage that nobody notices for months.
Discovery Strategies
Since catalogue coverage determines what your analysis can see, discovery deserves as much design attention as extraction.
Category traversal. Walking a retailer's category hierarchy and paginating each leaf. The most systematic approach, and the one most likely to hit pagination limits — many sites cap results at a few hundred per category regardless of how many products exist. Where that happens, subdivide by filters until each result set falls under the cap.
Faceted navigation. Using filters — brand, price band, attribute — to partition a category into slices small enough to fully enumerate. Effective for large categories and the standard workaround for pagination limits. Watch for combinatorial explosion.
Sitemap parsing. Many retailers publish product sitemaps, sometimes with last-modified timestamps. Free, authoritative coverage information, and routinely ignored.
Search-based discovery. Querying site search for brand names, model numbers, and category terms. Catches products that category navigation buries.
Seller catalogue enumeration. On marketplaces, walking a specific seller's full listing set. The right approach when tracking particular competitors rather than a whole category.
Identifier enumeration. Where product URLs contain sequential or predictable identifiers. Efficient when it works and worth checking for.
Cross-retailer discovery. A product found at one retailer can be searched for at others. This is how you build assortment gap analysis, and it depends on your entity resolution working.
Recommendation and cross-sell traversal. Following "related products" links surfaces items the category structure hides. Noisy but occasionally the only route to certain listings.
Coverage measurement: whichever combination you use, maintain a coverage estimate. Compare product counts against any figure the site itself displays, check whether newly discovered products are appearing at a plausible rate, and periodically verify a sample of category listings by hand. A discovery process that quietly stops finding new products looks identical to a category that stopped growing.
Entity Resolution: The Central Problem
The same product appears across retailers as different strings, different pack configurations, different bundles, and different identifiers. Everything comparative depends on matching them correctly, and this consumes more engineering effort than the collection itself.
Use identifiers where they exist. Manufacturer part numbers, barcodes, model numbers, and standard product identifiers. Reliable when present, frequently absent, and occasionally wrong.
Normalise aggressively before matching. Case, punctuation, whitespace, abbreviations, unit representation, and pack notation account for a large share of near-misses.
Extract structured attributes rather than matching titles. Brand, model, capacity, size, colour, material, and variant. Attribute-level matching is substantially more reliable than string similarity on titles, which are marketing copy rather than identifiers.
Score confidence rather than deciding binary. Record the score so downstream analysis can filter by it, and so you can measure how much of your comparison rests on shaky matches.
Review the ambiguous band manually. High-confidence matches and clear non-matches need no attention. Errors concentrate in between.
Decide variant policy explicitly. Same product in a different colour? Different pack size? A bundle with an accessory? Refurbished? Regional edition? These are commercial decisions that should be documented, not emergent properties of a matching algorithm.
Maintain a persistent registry. Re-matching between runs produces inconsistency that appears as fake changes in your time series.
Measure match rate over time. A falling rate is an early signal that a source changed its title formatting or attribute structure, and it degrades quietly.
Extraction: What to Capture From a Product Page
A product page contains far more than price, and collecting it in one pass is much cheaper than going back later.
Identity fields: title, brand, manufacturer part number, barcode or standard identifier, model, and the retailer's own SKU.
Commercial fields: list price, selling price, currency, tax treatment, shipping cost and conditions, quantity basis, promotional labels, and any quantity-break pricing.
Availability fields: stock status, delivery estimate, fulfilment method, and store-level availability where shown.
Attribute fields: the structured specification table, which is where the data most useful for matching and comparison lives, and which many pipelines skip because it's variable in shape.
Presentation fields: description text, bullet points, image count and URLs, video presence, and rich content blocks. These support presentation benchmarking and are rarely collected.
Social proof fields: rating, review count, question count, and badges such as bestseller labels.
Seller fields: seller identity, rating, fulfilment type, and buy-box status on marketplaces.
Categorisation: the retailer's own breadcrumb path, which is useful for cross-retailer category mapping and is frequently more informative than their navigation.
Collection metadata: timestamp, market, exit location, extraction method, and page URL.
Prefer structured sources. Most retail product pages embed JSON-LD or microdata containing identity, price, availability, and rating in machine-readable form. This is dramatically more stable than CSS selectors and should be your first extraction attempt. Internal JSON endpoints are the second, and DOM parsing the fallback.
Capture the specification table properly. It arrives as variable key-value pairs rather than a fixed schema, so store it as structured data rather than forcing it into predefined columns. It's the richest matching signal available and the most commonly discarded.
Review and Sentiment Data
Reviews are the richest qualitative source in ecommerce data and the most frequently misused.
What review data supports:
- Complaint theme extraction. What specifically goes wrong with a product, in customers' words. Frequently the most actionable output of an entire programme.
- Feature gap identification. What customers wish a product did, expressed unprompted.
- Comparative mentions. Reviews that name competitors reveal the consideration set.
- Quality trend detection. Sentiment shifting over time on a product often precedes broader problems.
- Velocity as an attention proxy. New reviews per unit time correlates loosely with sales activity.
The biases you must account for:
- Self-selection. Reviewers skew toward extremes. The satisfied middle rarely writes.
- Incentivised and solicited reviews distort ratings upward.
- Fake reviews exist at meaningful rates in some categories.
- Recency and volume effects. Newer products have fewer reviews and noisier averages.
- Platform demographics. Each marketplace's reviewers are a specific population.
- Language and market. Review culture differs substantially by country.
Practical handling:
- Normalise by volume so high-selling products don't dominate purely through review count
- Segment by rating band. Three-star reviews are typically the most informative; one- and five-star are the most polarised
- Prefer themes over scores. "Battery life" appearing in 30% of negative reviews is more actionable than an average rating of 3.8
- Track change rather than level. A complaint theme appearing or disappearing tells you something happened
- Treat velocity as a proxy and label it as one
- Handle personal data appropriately. Reviews contain names and sometimes more, and that's personal data regardless of being publicly posted
Who Uses This and What They Need
The same infrastructure serves quite different purposes, and knowing which you are shapes what to prioritise.
Brands and manufacturers need to know where their products are sold, by whom, at what price, and how they're presented. Unauthorised sellers, incorrect product information, poor imagery, and missing variants on retailer sites all cost revenue directly. Coverage breadth across many retailers matters more than depth on any one.
Retailers need competitive assortment and pricing intelligence, plus visibility into what's selling in their categories. Depth on a defined competitor set matters more than breadth.
Marketplace sellers need buy-box dynamics, competitor listing changes, and rapid detection of new entrants on their products. Very high frequency, narrow scope, latency-sensitive.
Private label and product development teams need review-derived complaint themes across a category to find where the market is failing customers. Text depth matters far more than price precision.
Category managers and buyers need assortment gaps, new product detection, and supplier landscape. Moderate frequency, broad coverage.
Investors and analysts need category-level structure, concentration, and trend signals. Low frequency, very broad, tolerant of approximation.
Distributors and wholesalers need to monitor both upstream supplier catalogues and downstream retail presence, which is an unusually wide collection footprint.
The useful question for any of these: what decision does this inform, and at what latency? A category manager reviewing quarterly needs different infrastructure from a marketplace seller responding to buy-box changes within minutes, even though the pages being fetched are identical.
Handling Change Detection
Ecommerce intelligence is mostly about change, and detecting it reliably needs deliberate design.
Store observations immutably. Never overwrite a current-state record. The history is the product, and it cannot be recollected.
Diff at the field level. Knowing that a product changed is less useful than knowing its price fell, its description was rewritten, or it went out of stock. Field-level diffing makes changes actionable.
Distinguish meaningful from cosmetic change. Description text changes constantly for trivial reasons. Hashing normalised content after stripping volatile elements avoids drowning in noise.
Detect appearance and disappearance separately from modification. These are three different event types with three different meanings, and collapsing them makes analysis harder.
Handle temporary disappearance carefully. A product missing from one collection run may be delisted or may be a failed fetch. Require confirmation across multiple runs before recording a delisting, or your data will be full of products that resurrect.
Track the rate of change per source. A retailer whose entire catalogue appears to change overnight has probably restructured their site, not their range.
Timestamp precisely and consistently. Change attribution depends on knowing when you observed, not when you processed.
Working With Review Text at Scale
Review text is the highest-value and least-structured part of an ecommerce dataset, and extracting signal from it needs its own approach.
Theme extraction rather than sentiment scoring. An average sentiment of 0.62 tells you nothing actionable. "Battery life appears in 34% of negative reviews for this product and 8% for the category" tells you where to invest. Extract the topics customers actually raise, then measure their prevalence.
Build a category-specific theme taxonomy. Generic sentiment tooling produces generic output. The complaint themes that matter in kitchen appliances are not the ones that matter in outdoor gear, and a taxonomy grounded in what your category's reviewers actually discuss produces far sharper analysis.
Normalise by volume and by product age. Raw theme counts track review volume, which tracks sales and time on market. Proportions are comparable; counts are not.
Segment by rating band. Three-star reviews are usually the most informative — detailed, balanced, and specific. One-star reviews are frequently about delivery or a defective unit rather than the product. Five-star reviews are often uninformative.
Track theme prevalence over time. A complaint appearing or vanishing indicates something changed — a manufacturing revision, a supplier switch, a firmware update. This is one of the few genuine leading indicators available in public data.
Compare across the category, not just within a product. A complaint present at similar rates across every competitor is a category characteristic. One that's elevated on a single product is a defect and an opportunity.
Handle verified-purchase flags where available, since they carry meaningfully different reliability.
Extract comparative mentions. Reviews that name other products reveal the actual consideration set, which is frequently not the competitive set the business assumes.
Respect the personal data dimension. Review text contains names, sometimes locations, occasionally more. Storage, retention, and access controls should reflect that this is personal data regardless of having been publicly posted.
Marketplace-Specific Complications
Marketplaces are the richest ecommerce data source and the most structurally awkward.
Multiple sellers per listing. One product page may host dozens of sellers at different prices and conditions. Decide explicitly what you're tracking — buy-box winner, lowest price, a specific seller, or the full distribution — and hold that definition constant.
Buy-box dynamics. Which seller wins the default purchase option changes frequently and by algorithm. Tracking it over time reveals competitive dynamics that static observation misses entirely.
Seller identity is unstable. Sellers rebrand, exit, and re-enter. Tracking by seller name alone produces false continuity and false disappearance.
Listings versus products. The same product may exist as several separate listings, some duplicate, some for variants, some created by different sellers. Deduplicating at the product level is necessary and non-trivial.
Ranking and search results are personalised and algorithmic. A category's "top products" is a marketplace's ranking output, not a market structure. Treating it as the latter is a common error.
Sponsored placements are interleaved with organic results and are frequently not clearly distinguished in the markup. Failing to separate them contaminates any analysis of what's genuinely popular.
Regional storefronts differ substantially. The same marketplace in two countries may have different assortment, different sellers, and different pricing, which makes geographic collection a requirement rather than an option.
Aggressive bot detection. Marketplaces are among the most likely targets to require residential addresses, and among the most likely to serve degraded content rather than an outright block.
Detecting Silent Failures
Ecommerce pipelines fail plausibly. A parser that stops finding the specification table doesn't error — it records an empty attribute set, and your match rate quietly declines.
Track extraction rates per field, per source. This is the single most valuable monitor available. If image count extraction drops from 97% to 12% on one retailer, that retailer changed their markup. Aggregate success rates will not show this.
Monitor coverage ratio. Products discovered versus products expected. A discovery process that stops working looks exactly like a category that stopped growing, and only a coverage metric distinguishes them.
Watch match rate over time. A falling entity resolution rate usually means a source changed its title or attribute formatting.
Validate value plausibility. Prices within category bands, review counts non-decreasing, rating values in range, image counts non-zero. Assert at extraction rather than discovering downstream.
Gate implausible aggregate movement. An entire retailer's catalogue appearing to change overnight is a site restructure or a parser break, not a business event.
Check response sizes. Challenge pages, redirects to generic category pages, and degraded content are all much smaller than a real product page, and all can parse as "nothing found."
Require confirmation for disappearance. A single missed fetch should not record a delisting. Two or three consecutive absences should.
Distinguish absence states. Out of stock, delisted, page unreachable, field not present, and not yet collected are five distinct things.
Sample manually every month. Pull a handful of records and check them against the live pages by hand. Automated validation catches what you thought to check for. This catches everything else, and on a dataset this wide there is always something else.
Proxy and Collection Requirements
Ecommerce intelligence has one of the more demanding profiles in web data work.
Expect to need residential addresses on major retailers and marketplaces. These are the targets most likely to block hosting ranges, and among the most likely to serve subtly different content rather than a clean refusal.
Geographic targeting is a correctness requirement for any multi-market work. Assortment, pricing, availability, and even seller composition vary by market, and collecting from one location produces a picture of that location.
Sticky sessions matter. Retail sites hold market, currency, and delivery destination in session state. An address change mid-sequence resets that context and produces inconsistent records. Hold the session for the duration of a product or category sequence.
Bandwidth is substantial. Retail pages are heavy — images, scripts, tracking, recommendation widgets. If you're rendering with a headless browser on metered plans, this becomes the dominant cost quickly. Check for internal JSON endpoints first; most modern retail sites have them, and they're both smaller and more stable to parse.
Segment by workload. Category discovery, product page collection, review collection, and availability checking have different frequencies and different requirements. Routing all of them through the most expensive tier is straightforward waste.
Test per source. Not every retailer runs aggressive detection. Five hundred requests through datacenter addresses per significant source, status distribution recorded, splits your sources into a cheap majority and an expensive minority.
Scaling the Collection
Ecommerce intelligence generates more volume than most web data work, and the growth is multiplicative rather than linear.
The multiplication problem. Products multiplied by retailers multiplied by markets multiplied by collection frequency. Two thousand products across six retailers in four markets, checked daily, is 48,000 fetches a day before you touch reviews or discovery.
Segment by frequency rather than collecting everything at one rate:
- Price and availability on priority products, high frequency
- Price and availability on the long tail, daily or weekly
- Product attributes and descriptions, weekly or monthly, since they rarely change
- Reviews, incrementally — only fetch new ones rather than recollecting the full set
- Discovery, on its own schedule, since finding new products doesn't need the same cadence as tracking known ones
Fetch incrementally wherever possible. Reviews in particular accumulate, so collecting only those newer than your last observation avoids refetching thousands of records to find a handful of new ones.
Prefer endpoints to rendered pages. The bandwidth difference on image-heavy retail sites is an order of magnitude, and internal JSON endpoints are usually more stable to parse as well.
Parallelise across retailers, not within them. Aggregate throughput and per-site politeness aren't in conflict. Cap per-retailer concurrency at a respectful level and run many retailers simultaneously.
Use the cheap tier where testing permits. The per-source split between permissive and demanding sources is usually lopsided in your favour, and routing everything through residential because the marketplaces need it is a common and expensive default.
Track cost per retailer and per data type. It makes frequency decisions evidence-based, and it occasionally reveals that one source consumes a disproportionate share for marginal value.
Watch storage growth. Immutable observations plus raw response retention at this volume adds up. Partition by date, tier hot and cold storage, and make retention a deliberate policy rather than an accident.
Analysis That Produces Decisions
Assortment gap analysis. Products competitors carry that you don't, and vice versa. Usually the highest-value output and one of the simplest to compute once matching works.
Category structure and concentration. How many distinct products, how many sellers, how concentrated the top of the category is, and how that shifts over time.
New product detection. What appeared this month, from whom, at what price point. A leading indicator of competitor strategy that precedes any announcement.
Delisting and range contraction. What disappeared. Harder to detect and frequently more informative, since companies announce launches and stay quiet about withdrawals.
Availability patterns. Persistent stockouts indicate supply problems or demand exceeding forecast, both of which are commercially useful to know about a competitor.
Presentation benchmarking. How competitors title, describe, photograph, and attribute the same products. Directly actionable for your own listings.
Review-derived product intelligence. Complaint themes across a category reveal where the entire market is failing customers, which is where product opportunity lives.
Seller landscape movement. New entrants, exits, and shifts in catalogue breadth on marketplaces.
Cross-market comparison. How assortment and positioning differ between countries, which informs both expansion decisions and an understanding of what a competitor prioritises where. Products launched in one market before another are a particularly reliable signal of where a company is testing.
Private label detection. Products carried by only one retailer, with no manufacturer identifier and a retailer-owned brand, are usually private label. Tracking their share of a category reveals how aggressively a retailer is competing with the brands it stocks.
Common Mistakes
Treating collection as URL refreshing. Ecommerce is catalogue synchronisation. A pipeline that only refreshes known URLs diverges from reality silently.
Not tracking disappearance. Delistings and range contractions are signal, and they only exist if you were tracking presence.
Skipping the specification table. The richest matching and comparison signal on the page, discarded because it doesn't fit a fixed schema.
Conflating listings, products, and variants. Decide the unit of observation before building, or cross-retailer comparison becomes impossible.
Treating marketplace rankings as market structure. Category top-sellers are an algorithm's output, personalised and sponsored-contaminated.
Not separating sponsored placements. Contaminates every conclusion about what's genuinely popular.
Treating review velocity as sales. A proxy with unknown conversion, routinely presented as a measurement.
Collecting from one market. Assortment, availability, and seller composition all vary geographically.
Ignoring seller identity instability. Sellers rebrand and re-enter, producing false continuity and false exit in your data.
One collection schedule for everything. Prices move fast, descriptions move slowly, reviews accumulate. A single frequency is either wasteful or insufficient.
Under-investing in entity resolution. Everything comparative depends on it, and it's the part most often treated as a build-phase task rather than ongoing work.
Not measuring coverage. A discovery process that stops finding new products looks exactly like a category that stopped growing.
A Realistic Build Sequence
Phase one — define the unit of observation. Product, variant, or listing. This determines whether cross-retailer comparison is possible and is expensive to change later.
Phase two — source assessment and proxy requirement testing. Which retailers, which markets, and what each requires. Five hundred datacenter requests per source, status distribution recorded.
Phase three — discovery design. Category traversal, faceted partitioning, sitemaps, and search. Establish a coverage measurement before you rely on it.
Phase four — extraction with structured-data preference. JSON-LD first, internal endpoints second, DOM last. Validate at extraction and verify a sample manually.
Phase five — entity resolution. Confidence-scored matching, persistent registry, manual review of the ambiguous band. Before scaling, since retrofitting is painful.
Phase six — storage with observations immutable. One record per observation, absence states explicit, raw responses retained, collection metadata on everything.
Phase seven — silent failure detection. Extraction rates per field per source, coverage ratio monitoring, implausible-change gating.
Phase eight — analysis layers. Assortment gaps, change detection, review theme extraction, seller landscape.
Phases one and five are the ones compressed under pressure, and they're the two that determine whether the dataset supports comparison or merely accumulates.
Legal and Ethical Considerations
- Collecting publicly displayed product and price information is normal commercial practice and generally unproblematic in itself.
- Terms of service on retail sites and marketplaces frequently prohibit automated collection. Contractual and commercial risk rather than criminal in most jurisdictions, but real.
- Competition law considerations apply to systematic competitor price and assortment monitoring, particularly in concentrated markets and particularly where pricing responses are automated. Worth a conversation with counsel.
- Review data contains personal data. Names, sometimes locations, occasionally more. Data protection law applies regardless of public posting, and retention policies should reflect that.
- Don't republish scraped content. Reviews, descriptions, and images are the retailer's or the reviewer's, and using them as inputs to analysis is different from reproducing them.
- Rate-limit as a matter of conduct. Degrading a competitor's site is an unforced error.
- Be accurate about what proxies measure when the data informs commercial decisions. Treating review velocity as sales data has led to real mistakes.
Not legal advice, and jurisdictions differ. Commercial programmes at scale warrant a lawyer's review, particularly around competition law and personal data in review content.
Frequently Asked Questions
Can I estimate competitor sales from this data?
Not directly. Review velocity, ranking movement, and stockout frequency are proxies with unknown conversion rates. Build an estimate if you must, label it clearly as an estimate, and triangulate across several signals rather than trusting one.
Do I need residential proxies?
For major retailers and marketplaces, usually yes. For smaller sites and brand-direct stores, often not. Test per source rather than assuming either way.
How do I handle products that won't match across retailers?
Accept a match rate below 100%, record confidence, and analyse the matched subset while reporting coverage. Unmatched products are also signal — they may be exclusives or private label.
How often should I collect?
Different per workload. Prices and availability move fast; assortment and descriptions move slowly; reviews accumulate steadily. One schedule for everything is either wasteful or insufficient.
Should I collect reviews or just ratings?
Text is where the value is. Ratings alone give you an average that hides the specific complaints that would actually change a product decision.
How do I handle marketplace listings with many sellers?
Decide what you're tracking, document it, and hold it constant. Switching between buy-box and lowest-price mid-series produces movement that isn't real.
What about sponsored versus organic placements?
Separate them. Failing to distinguish sponsored placements contaminates any conclusion about what's genuinely popular in a category.
Should I collect competitor images?
Collect the URLs and metadata — count, dimensions, whether video is present — for presentation benchmarking. Downloading and storing the images themselves adds substantial bandwidth and storage cost and raises copyright questions, so do it only if you have a specific use for the pixels rather than the metadata.
How do I map categories across retailers?
Retailer taxonomies rarely align. Use breadcrumb paths as raw input, then build your own category mapping keyed to your products rather than trying to reconcile the retailers' hierarchies to each other. It's less elegant and it actually works.
How do I know my coverage is complete?
You usually don't, and claiming otherwise costs credibility. Track discovery coverage explicitly, report it, and treat a falling coverage ratio as a defect rather than an inconvenience.
Getting the Collection Layer Right
Ecommerce intelligence puts unusual demands on collection: heavy pages, aggressive bot detection on major retailers, and geographic accuracy as a correctness requirement rather than a refinement. ProxyScrape's residential proxies provide the country and city-level targeting that multi-market assortment and availability work needs, with sticky sessions long enough to hold a retailer's market and delivery context through a category sequence. For the substantial portion of sources that testing shows don't check IP classification — brand-direct stores, smaller retailers, review aggregators — their datacenter plans with unlimited bandwidth handle image-heavy retail pages at high volume far more cheaply, and their ecommerce intelligence documentation covers the setup side.
→ Compare proxy options for catalogue and marketplace collection
Ecommerce intelligence rewards patience with the unglamorous parts. The teams that get real value from it are the ones that treated it as catalogue synchronisation rather than URL refreshing, invested properly in entity resolution before scaling, tracked what disappeared as carefully as what appeared, and labelled their proxies as proxies when the data reached a decision-maker. The pages are easy to get. Knowing what you're looking at, and that it's comparable to the thing you're comparing it against, is the entire discipline.