catalog-extract/SKILL.md · md · download
---
name: catalog-extract
description: |
Build and run a recurring competitor catalog tracker with plain Nimble Extract:
every product (SKU, price, specs), every store location, and third-party
retailer carriage for a brand's DTC site, written to append-only CSVs with a
snapshot_date so the history is the file. No Extract Templates, no Web Search
Agents: one `client.extract` fetch path, site-owned discovery, JSON-first
parsing, and a truncation guard on every fetch.
Triggers: "scrape a brand catalog", "track competitor prices", "competitor
catalog", "catalog tracker", "monitor a brand's products and prices over time",
"store locator scrape", "which retailers carry", "add a new brand to the
tracker", "build a scraper with Extract", "Citizen use case", "watch competitor
intel", "price refresh", "delisted SKUs", "new SKUs since".
Not for one-off page reads (use nimble-web-expert), open-ended research or
entity discovery without URLs (Web Search Agents), or building reusable
Extract Templates.
allowed-tools:
- Bash
- Read
- Write
- Edit
- Glob
- Grep
- Task
- AskUserQuestion
metadata:
version: "1.0.0"
author: Jack Ingoldsby
origin: Citizen Watch Company of America competitor-intel handoff (2026-08)
---
# Catalog Extract
**Goal:** for each brand site the user names, produce three append-only CSVs (products, stores, retailer listings) that can be re-run nightly and diffed over time. Everything goes through Nimble Extract (`nimble_python` `client.extract`, or `nimble extract run` for probes). Nothing else fetches.
User request: $ARGUMENTS
Scripts live in `scripts/` next to this file. Set `SK` to that directory. `NIMBLE_API_KEY` must be in the environment or a `.env` in the working tree.
```bash
SK="$(dirname "$(find ~/.claude /Users -path '*catalog-extract/SKILL.md' 2>/dev/null | head -1)")/scripts"
python3 -c "import nimble_python, bs4" || pip install nimble_python beautifulsoup4
```
## Non-negotiables (read before writing anything)
1. **`status == "success"` proves nothing about completeness.** Nimble returns cleanly truncated bodies (closing tags intact) when a render races: 840 KB `products.json` came back at 15-60 KB; PDPs came back `<head>` only. A truncated page parses into a row that *looks* complete with every spec silently empty. Every fetch therefore carries a `wait` action (8 s for big JSON, 4-5 s for JS PDPs) **and** a sentinel or parse check, and retries. `nx.fetch(..., sentinel=..., expect_json=...)` does this; never bypass it.
2. **Discovery comes from the site, never a hardcoded list.** Sitemap, `products.json`, grid endpoint, or paginated listing. A static list silently misses additions.
3. **JSON before DOM.** JSON-LD `Product`, `__NEXT_DATA__`, inline state objects, or the page's own XHR endpoint. CSS selectors rot.
4. **Filtered is not failed.** A strap or a bag is `FILTERED` (never counts against the failure rate). "Tried and could not parse" is a failure and is never written as a hollow row.
5. **Never fabricate.** A locator that returns nothing parseable writes 0 rows and a WARNING with the raw dump path. Store counts under an N-nearest cap are reported as a floor.
6. **Never guess in recon.** Every claim carries the URL it was seen on and a real example value; anything unverified is labelled `UNVERIFIED`.
7. **Read-only, public catalog pages, human pace.** No logins, no checkout, no CAPTCHA work-arounds. Public front-end tokens (an Algolia search key) are read live from the page at runtime, never pasted into docs or commands.
8. **Append-only + `snapshot_date`.** One block of rows per run. Never rewrite history; the active catalog is derived (see `references/schema.md`).
## Which workflow to load
Scope first (layers, brands, locale: `workflows/recon.md` Step 0), then read only the workflow file(s) the request needs; each holds the full procedure, commands and measurements for its layer, and ends with the per-layer report to give.
| Request | Load |
|---|---|
| New brand, "is this site scrapable", `UNVERIFIED` recon, a site that started failing | `workflows/recon.md` |
| Products, prices, specs, "add a brand", spec map, hook, nightly / weekly runs, "new SKUs since", "delisted SKUs", "price refresh" | `workflows/products.md` |
| Store locator, dealers, doors, "how many stores" | `workflows/stores.md` |
| "Which retailers carry", retailer prices, carriage matching (no retailer scripts exist yet) | `workflows/retailers.md` |
References: `references/schema.md` (columns, active-catalog window, SKU lifecycle), `references/site-recipes.md` (worked brand and retailer recipes), `references/extract-field-notes.md` (measured Nimble behaviour).
## Where to look when something is off
- Specs empty on rows that have a price -> truncation. Raise `--wait`, tighten `--sentinel` to the spec block, re-run `--mode full --limit 5`.
- `products.json` or a JSON API "unparseable" -> truncation; `fetch_json` already waits 8 s and retries 3x. Lower `limit=` does **not** help.
- `400 browser_actions must be a non-empty array` -> you sent an empty list; `nx.fetch` omits the key when there is nothing to run.
- `cookies=` rejected -> you passed `get_cookies` output verbatim; use `nx.clean_cookie`.
- Session-pinned fetches hang 8-12 min then 500 -> do not use `session=`; carry cookies instead.
- A JSON endpoint through a browser driver arrives as `<pre>{...}</pre>`, HTML-escaped; `nx.json_from_html` unwraps it.
- Sweeps of thousands of URLs: the Extract **batch** endpoint queues server-side (~500 URLs/min vs ~0.5/s from a local pool); set `driver` to `vx8` or above explicitly, `vx6`/unset can sit pending forever. See `references/extract-field-notes.md`.