Sage AI Labs runs a shopping product built on product data from a large number of online retailers, with its own checkout and order handling. My work covers the backend behind that: getting retailer data in and keeping it fresh, an AI harness that writes much of that extraction code, image processing, scheduling, live stock and shipping checks, and the internal tools the operations team relies on.

Retailer onboarding and the data pipeline

Every store structures its catalogue differently, so each retailer needs its own product data extraction script. I’ve onboarded 55+ retailers by hand, end to end: writing the script, validating the data it returns, and plugging it into pipelines that process more than 100K products a day.

Every extraction run is also exported to Google Cloud Storage as CSV, and product records keep price history, so run frequency can be set per retailer from real change data instead of a fixed hourly schedule.

The AI harness that writes extraction scripts

Writing a script by hand takes 1 to 72 hours depending on the store. I’m building an LLM-based harness that does it in about 30 minutes to 2 hours, on 4 retailers in parallel. So far 165+ of its scripts, for Shopify and non-Shopify stores, have been merged into the data pipeline.

A run goes through three stages:

• Discovery. For non-Shopify stores, the harness captures the site’s browser network traffic to find the APIs behind product listings and pages, then checks that those requests can be replayed and paginated. Cookies and auth headers are redacted.
• Verification. After a coding agent writes the script, 15 sample products are checked against the live store. Confirmed mismatches go back to the agent for up to two repair attempts.
• Approval. A dry run over the full catalogue comes next, then a person signs off. The approval is tied to that exact verified script and output.

LLM spend has limits, with token and cost logging per call. Run state is checkpointed at each phase change, and deploys are immutable staged releases that a daemon swaps in, rolling back on a failed health check.

Image processing

Product images go through a serverless pipeline on Modal and Cloudflare R2 that handles over 1 million images a day. When I moved image storage from AWS S3 to R2, I added multiple resolutions, compression, BlurHash placeholders, asynchronous processing and Sentry monitoring.

The pipeline used to ask R2 whether each image already existed before processing it. I replaced those calls with a database cache of known images, which cut R2 Class A operations by about half (an estimate, not an exact measurement) and made runs faster and lighter on compute.

A database-driven scheduler

Scrapes used to run as 200+ separate Modal cron jobs, so every schedule change needed a deploy. I designed and built a scheduler that keeps schedules in the database. A dispatcher runs every 5 minutes and claims due jobs atomically, so nothing runs twice. Jobs get retry backoff, timeouts and a Slack heartbeat. Operations now edit schedules from the ops console.

Live inventory and shipping

I built a FastAPI live-inventory service that reuses each retailer’s extraction code to fetch current size availability and per-size price, covering 58 non-Shopify retailers. In the Node.js/TypeScript checkout backend it runs with a 5-second timeout, a cached list of supported retailers and fail-open handling, so a slow check never blocks a purchase.

For shipping, a daily pipeline collects cart quotes from 156 Shopify stores. For 159 other retailers it finds each U.S. shipping-policy page through search, and an LLM extracts the rules into a strict JSON schema, run twice to flag disagreements. A shipping-cost API returns live Shopify quotes and falls back to the database estimate. Both run in production.

Keeping extraction reliable

• Bot protection. When sites with strict bot protection started blocking extraction, I got it working again by rotating browser sessions before they hit block limits and fixing a broken challenge handler.
• Proxy waterfall. I extended our shared proxy-fallback layer to Scrapy and added Selenium and Camoufox routes. Requests start without a proxy and step up cheapest-first, replacing about 450 lines of static per-spider proxy lists.
• A file-descriptor leak. Live checks once went down for every retailer. I traced it to uvloop races on concurrent Playwright/Camoufox browser launches and browsers shared across threads. Pinning each retailer to one thread took open descriptors from 650–750 and climbing to a steady 130.
• Alerts. I fixed false “zero products” and “50% drop” Slack alerts caused by stale date windows and an incremental SQL model overwriting past counts with zeros.

Affiliate and commission work

A daily Airflow pipeline I built ingests affiliate revenue from six affiliate networks into PostgreSQL, with normalized statuses, idempotent upserts, raw payloads for audit and Slack failure alerts.

I also added affiliate link generation for two more networks, moved affiliate mappings into Amplitude remote config so they change without a deploy, and fixed commission reporting in the brand-operations dashboard: approved versus pending commission, a sales-weighted rate, and pagination that skipped same-day transactions.

Internal tools and order management

I independently maintain two inherited internal tools, including the order-management system a 5–8 person operations team uses. There I added Stripe refunds, cancellation-reason statuses, and “Need Attention” and “Refunded” views with tags and notes. I also fixed lost deal selections in the merchandising dashboard with a single transaction and SELECT ... FOR UPDATE row locks, and moved the AI order-placement bot’s per-retailer routing from static config into the database. Day to day, I ship through CI/CD and handle production issues across 10+ scrapers and services.

Built with: Python, FastAPI, Scrapy, Playwright, Selenium, Camoufox, Modal, Cloudflare R2, AWS S3, PostgreSQL, Airflow, Google Cloud Storage, Node.js, TypeScript, Shopify, Stripe, Amplitude, Sentry and Slack.