eBay processes millions of active listings across thousands of categories. Prices shift mid-day and search rankings rearrange hourly. Competitors adjust strategies in response to demand signals that most sellers never see.
Professional sellers - those managing hundreds or thousands of SKUs - face a structural disadvantage. The sellers who react fastest to pricing shifts, competitive entries, and demand signals capture disproportionate margin. Everyone else operates on instinct, spreadsheets, and data that aged out before the morning was over.
The difficulty is access to data since eBay’s public APIs offer limited coverage. The platform’s anti-bot infrastructure actively blocks external data collection. Listing formats vary wildly in structure and quality, making cross-seller comparison impossible without excessive normalisation. And the signals that matter most - real-time rank positioning, competitor pricing velocity, inventory movement - require continuous monitoring at a scale no manual process sustains.
We built Indications to close that gap. The platform extracts market data at scale, structures and normalises it using AI-driven parsing, and delivers action items through dashboards, alerts, and API access - all while remaining invisible to eBay’s detection systems.
Three connected problems
The engagement began with a seller community that had data needs outpacing their tools. Our discovery work identified three interconnected challenges. These were:
- Detection and access control. eBay monitors for automated access patterns. Predictable request intervals, consistent IP addresses, and uniform browser fingerprints trigger detection systems within hours. Building a pipeline that operates continuously at high volume without tripping these defences is an engineering problem before it is a data problem.
- Data heterogeneity across listings. The same product sold by five different sellers appears with five different title formats, attribute structures, and pricing conventions. Extracting comparable data from this variation requires parsing and normalisation logic that accounts for the inconsistency at scale.
- Coverage and freshness requirements. Professional sellers need three distinct data views: search rankings (competitive positioning), product-level detail (pricing, stock, condition), and storefront-level intelligence (competitor portfolio composition and sales velocity). Each view demands its own collection strategy, and the data loses value within hours.
We designed the platform as three connected layers, each addressing one of these challenges while feeding into the next.
Click to expand Layer 1: stealth data acquisition
We built the scraping infrastructure to operate across three collection tiers - search, product, and storefront - each with its own crawling logic, frequency, and evasion requirements. Session-aware IP rotation cycles through blended proxy pools combining ISP and residential addresses. Our rotation patterns are randomised to prevent the predictable cycling that detection systems flag. The blend ratio adjusts dynamically based on detection risk signals from each session.
Browser fingerprint management varies user agents, viewport dimensions, interaction timing, scroll behaviour, and JavaScript execution characteristics across sessions. Each scraping session presents a technically distinct fingerprint that matches organic browsing patterns. Automated anti-captcha handling uses ML-based solvers integrated directly into the crawl pipeline. When eBay presents a captcha challenge, the solver resolves it in-flow without breaking session continuity or introducing latency spikes that would signal automation.
Airflow orchestration coordinates the entire crawl schedule - managing job sequencing, retry logic on failed requests, and pipeline state from raw HTML through normalised output. The orchestrator maintains awareness of rate limits, detection thresholds, and source-specific crawl windows.
This led to a continuous, high-volume data collection operating around the clock without triggering eBay’s detection infrastructure. We maintained uninterrupted access throughout the engagement and subsequent production operation.
Layer 2: AI-driven parsing and normalisation
Raw eBay listing data is structurally inconsistent. Two listings for the same product may describe it with different attribute names, different unit conventions, and entirely different title structures. We built the normalisation layer to handle this variation programmatically.
Attribute alignment maps inconsistent listing attributes into a unified schema. Our models identify equivalent attributes across different naming conventions - “color” versus “colour,” “sz” versus “size,” free-text descriptions versus structured fields - and produce a canonical record that supports genuine cross-seller comparison.
Cross-seller product mapping links listings from different sellers that represent the same underlying product, even when titles, images, and attribute structures differ substantially. This mapping enables apples-to-apples competitive analysis - the core requirement our sellers had been attempting to do manually.
SKU velocity detection identifies fast-moving products by tracking inventory changes and listing turnover rates across poll cycles. The system surfaces which products are selling, how quickly, and whether the velocity is accelerating or decelerating.
Layer 3: seller intelligence tools
The intelligence layer translates normalised data into operational decisions. We designed each tool around the specific workflows our seller users described during discovery.
Pricing alerts fire when competitors adjust prices on tracked products, when search rank positions shift beyond configurable thresholds, or when new competitors enter a monitored category. Alert delivery is near real-time - the lag between a competitor’s price change and the seller’s notification is measured in minutes.
SKU performance scoring evaluates each product in a seller’s portfolio against live market benchmarks. The score incorporates pricing competitiveness, rank trajectory, estimated margin relative to category averages, and sales velocity percentile. Underperforming SKUs surface automatically.
Category trend detection identifies emerging demand patterns and early-mover opportunities by tracking search volume, new listing rates, and price movement across category segments. These signals become visible in our system before they appear in eBay’s public trending data.
Portfolio optimisation recommendations synthesise pricing, ranking, velocity, and margin data into specific, actionable suggestions - which SKUs to reprice, where to increase inventory, which categories show entry opportunity, and which products to consider discontinuing.
Data flow - from eBay listing to seller decision
The full pipeline - from a listing change on eBay to an action item or alert in a seller’s dashboard - completes in minutes. Understanding how each stage hands off to the next is critical to understanding the system’s responsiveness.
Technology decisions
Every technology choice reflects a specific tradeoff we evaluated during architecture design.
| Layer | Technology | Why We Chose It |
|---|---|---|
| Scraping Engine | Python, Scrapy | Concurrency model and middleware flexibility handle multi-tier crawling at this volume |
| Orchestration | Airflow | DAG-based scheduling with retry logic and pipeline state management across hundreds of crawl jobs |
| Primary Database | PostgreSQL | Relational complexity of listings, rankings, seller profiles, and time-series pricing data |
| Caching | Redis | Sub-second reads for dashboard queries and real-time alert evaluation against incoming data |
| Anti-Bot Stack | ISP/residential proxies, fingerprint spoofing, ML captcha solving | Layered evasion required for sustained, high-volume access |
| Frontend | React, Webflow | React for interactive dashboard components; Webflow for marketing pages and onboarding flows |
| Design | Figma | Component-based design system for information-dense dashboard interfaces |
User experience
We served two user profiles with different interaction requirements. These are:
- Dashboard users access responsive, mobile-friendly interfaces for monitoring pricing, rank movements, competitor activity, and portfolio health. We designed the information architecture in Figma around scan-ability - professional sellers check dashboards frequently throughout the day and need pattern recognition at a glance. Dense data presented without visual clutter.
- API users access the same underlying data programmatically. Larger sellers who operate custom pricing engines, inventory management systems, or analytics tooling integrate Indications data directly into their workflows through embedded API endpoints.
Results we delivered
- Time-to-insight compression. Market intelligence that previously required weeks of manual effort - checking competitor listings individually, tracking price changes in spreadsheets, estimating category trends from anecdotal observation - now arrives in minutes through automated pipelines. Our sellers reported shifting from reactive to proactive competitive behaviour within the first month of adoption.
- Pricing and SKU optimisation. Sellers using the platform reported double-digit ROI improvements driven by data-informed pricing decisions and portfolio adjustments. The competitive visibility we provided enabled faster reaction to market shifts and identification of margin opportunities that manual monitoring consistently missed.
- 90-day trend intelligence. Trend reports by product and seller provide the historical context required for strategic decisions - seasonal patterns, competitor pricing behaviour over time, and category lifecycle trajectories. These reports became a core input to our sellers’ quarterly planning processes.
- Operational confidence. Sellers operate with a continuous, structured view of their competition. Pricing decisions, inventory allocation, and category entry decisions are grounded in current data rather than assumptions or outdated snapshots.
| Outcome | Detail |
|---|---|
| Time-to-insight | Weeks to minutes |
| Seller ROI improvement | Double-digit, driven by pricing and portfolio decisions |
| Trend reporting | 90-day historical analysis by product, seller, and category |
| Data freshness | Continuous polling with near real-time alert delivery |
| Detection incidents | Zero - sustained uninterrupted access throughout operation |
What this work represents
Indications shows what becomes possible when data acquisition engineering, AI-driven normalisation, and domain-specific intelligence design come together as a single integrated platform. The sellers we serve operate in a market where speed of insight determines margin. We gave them that speed.
The same architectural pattern - stealth-grade data acquisition, heterogeneous source normalisation, and operationalised intelligence - applies wherever valuable market data is fragmented, inconsistently structured, and access-controlled. Competitive intelligence, pricing optimisation, and market monitoring across e-commerce, logistics, and financial services all present structurally similar problems.
We build such systems. If the data your business needs is scattered, messy, and hard to reach - that is where our engineering starts.
We built Indications on Algorithmic’s data infrastructure and integrations practice - high-throughput ingestion, orchestration, and cross-system normalisation - paired with a web application and API layer designed for information-dense professional workflows.
