How the Extract skill works
Name a brand site; the skill tells Claude how to build and run the tracker on Extract
The skill packages the production pipeline Citizen runs nightly for ten brands and eleven retailers into a workflow Claude follows for any new brand. One Nimble Extract fetch path, site-owned discovery, JSON-first parsing, a truncation guard on every fetch, and the same CSV schema the analysis layer already reads.
Recon
One site fetch and one product page in both render modes. The report names the discovery source (sitemap, Shopify products.json, grid endpoint), whether the page is server-rendered, where the data lives (JSON-LD, __NEXT_DATA__, inline state) and the spec labels the page prints.
python3 $SK/recon.py https://www.brand.com --pdp https://www.brand.com/<one product>
Discovery
Every product URL from a site-owned source, never a hardcoded list. Sitemap or index, Shopify products.json with prices and variants, or a paginated listing. Output is urls.csv with sku, url and source.
python3 $SK/discover.py --sitemap <url> --match <regex> --sku-regex <regex> --out urls.csv
Spec map
Three real product pages (quartz, automatic, top of range) are saved and their labels listed. The user or Claude maps printed labels to canonical columns; unmapped labels still land in extra_attributes so nothing published is dropped.
python3 $SK/parse_product.py page.html --labels
Smoke test, then the run
Five rows first. Empty columns that should be full mean the spec map or the sentinel is wrong. Then the full run: threads, per-row flush, filtered versus failed accounting, failure-rate and volume-drop tripwires.
python3 $SK/run_catalog.py --brand tissot --urls urls.csv --sentinel "Country of manufacture" --spec-map specs/tissot.json --out output/brands/tissot.csv --mode full
Stores
The cheapest locator mechanism that works: the site's own JSON API, an in-browser fetch for APIs that need headers or a POST body, a metro grid for locators capped at N-nearest, or a server-rendered country page. A locator that returns nothing writes zero rows and a warning.
python3 $SK/stores.py --brand hamilton --browser-api <api> --page <locator page> --headers '{"Accept":"application/json"}' --records items --map '{...}' --out output/stores/hamilton.csvRetailer carriage
A retailer brand page is a listing with a different schema: one row per colourway with the retailer id, the manufacturer reference and a match back to the DTC SKU by exact reference first.
Report
Per layer: source used, render mode, rows written, failures and why, filtered count, billed requests, every UNVERIFIED item, and NOT PUBLISHED for fields the site does not carry.
Why plain Extract
Built for a nightly job, not a one-off scrape
Truncation guard on every fetch
A success status does not prove the body is complete. Every fetch waits, checks a sentinel or parses the JSON, retries, and counts a failure rather than writing a hollow row.
Discovery from the site
Sitemaps, products.json and grid endpoints enumerate the whole catalog each run, so additions are caught, not only drops.
JSON before DOM
JSON-LD, __NEXT_DATA__, inline state and the page's own XHR endpoints come first. CSS selectors are the fallback, because they rot.
Filtered is not failed
A strap or a bag is filtered on purpose and never counts toward the failure rate. Tried-and-could-not-parse does.
The history is the file
Append-only CSVs with a snapshot_date; the active catalog and the SKU lifecycle (first seen, last seen, delisted, price moves) are derived, never scraped.
Nothing fabricated
Unverified locators write zero rows. Counts under an N-nearest cap are labelled as a floor. Recon never guesses.
Nimble documentation
Read more
How the skill decides
Routing, render mode and the rules that shaped the code
Each decision comes from a measured production behaviour. The references carry the numbers; this is the short form.
| The user wants | The skill does | Example |
|---|---|---|
| A brand site to track over time | Runs the full workflow: recon, discovery, spec map, smoke test, run, stores, report | "Add Tudor to the tracker" |
| A price refresh on a known brand | run_catalog.py --mode full with the existing urls.csv and spec map | "Refresh Tissot prices" |
| Only new SKUs since last run | run_catalog.py --mode new skips every SKU already in the CSV | the nightly top-up |
| Store locations | JSON GET, in-browser fetch, metro grid or SSR country page, in that order of cost | "Where does Hamilton sell in the US" |
| Which retailers carry the brand | Listing discovery on the retailer brand page, one row per colourway, exact-reference match to the DTC SKU | "Is Movado at Macy's and at what price" |
| One page, one time | Not this skill: nimble-web-expert | "What does this page say" |
| Entities without URLs, open research | Not this skill: a Web Search Agent | "Find every watch boutique in Texas" |
Render mode is decided by parsing, not by bytes
Fetch the product page both ways
render=False is the raw server HTML and costs least; render=True runs the page in a browser with a wait action.
Parse both
Name, price, JSON-LD Product, __NEXT_DATA__ and the count of spec labels decide which fetch is complete.
Prefer raw when it carries the data
Hamilton: raw 542 KB with JSON-LD, price and 35 labels, so render=False. Tissot moved from a 5.9 KB shell in July to a complete raw page in October; recon is re-run before every rebuild because render needs drift.
Escalate only when both fail
A longer wait, the vx12-pro driver, a cookie warm-up, a referrer. A site that blocks is reported, not worked around.
Non-negotiables
- status == "success" proves nothing about completeness. Every fetch carries a wait action and a sentinel or a parse check, and retries. Measured: an 840 KB products.json delivered at 15 to 60 KB; product pages delivered as <head> only on 8 of 8 cold fetches.
- The sentinel must sit in the spec block, never in <head>. JSON-LD is at byte 18k of a 410k Tissot page and the spec block at byte 266k; a body cut between them passes a JSON-LD sentinel with a name, a price and no specs. One of seven fetches did exactly that on 19 October 2026.
- Discovery comes from the site. A static list silently misses additions; volume tripwires only catch drops.
- Filtered is not failed. Only tried-and-could-not-parse counts toward the 50% failure limit that exits non-zero.
- A catalog contraction beyond 15% is logged as VOLUME DROP, not failed, unless --fail-on-drop. A real contraction is valid data; failing nightly on it teaches people to ignore failures.
- Never fabricate. A locator that returns nothing parseable writes zero rows and dumps the raw body. Public front-end tokens are read live from the page, never pasted into docs.
- Read-only public catalog pages at human pace: no logins, no checkout, no CAPTCHA work-arounds.
- Append-only with snapshot_date. The active catalog is the newest substantial snapshot window; "latest row per SKU across all history" resurrects delisted products at stale prices.
Where to look when something is off
Specs empty on rows that have a price
Truncation. Raise the wait, move the sentinel to the spec block, re-run five rows.
A JSON endpoint is unparseable
Truncation. fetch_json already waits 8 s and retries three times; a smaller limit does not help.
400 browser_actions must be a non-empty array
An empty list was sent; nx.fetch omits the key when there is nothing to run.
cookies= rejected
get_cookies output was passed verbatim; nx.clean_cookie keeps only the accepted fields.
A session-pinned fetch hangs 8 to 12 minutes then 500s
Do not use session=; carry cookies instead.
Thousands of URLs
The Extract batch endpoint queues server-side at about 500 URLs a minute; set the driver to vx8 or above explicitly.
Validated examples
Three brands, five mechanisms, 19 October 2026
Every command below was run from a clean directory against the live sites on 19 October 2026. The files are the actual outputs.
tissotwatches.com, sitemap discovery, product loop
Tissot products
Discovery from the product sitemap, then the product loop three ways: a full run of five pages with a hook, a new-mode run that skips what is known, and a deliberately wrong sentinel to prove failures stay out of the file.
Discover
python3 $SK/discover.py --sitemap "https://www.tissotwatches.com/en-us/sitemap-custom-sitemap_0-product-en_US.xml" --match '/en-us/T\d+\.html$' --sku-regex '/en-us/(T\d+)\.html' --out .nimble/tissot/urls.csv
569 URLs, every one with a SKU, 1 billed request
Full run, five pages, hook
python3 $SK/run_catalog.py --brand tissot --urls .nimble/tissot/urls.csv --wait 3000 --sentinel 'Country of manufacture' --spec-map specs/tissot.json --hook hooks/tissot.py --out output/brands/tissot.csv --limit 5 --mode full
5 of 5 rows, 6 billed requests: one page came back as a 13.5 KB shell, the guard caught it and the retry succeeded
New mode
python3 $SK/run_catalog.py ... --limit 2 --mode new
skipped the 5 known SKUs, scraped the next 2; one page passed a JSON-LD sentinel without its spec block and the hook turned it into a counted failure instead of a hollow row
Wrong sentinel on purpose
python3 $SK/run_catalog.py ... --sentinel THIS_STRING_IS_NOT_ON_THE_PAGE --limit 2 --tries 1
2 counted failures, 0 rows written, exit 1
Result files
| File | What it holds |
|---|---|
| tissot_products_sample.csv | The seven rows written across the runs |
| tissot_recon.md | Recon report: raw versus rendered, spec labels |
| tissot_spec_map.json | The spec map built from the labels |
| tissot_hook.py | The hook: model line from the name, accessories filtered, no movement type is a failure |
| tissot_run_full.log | Log of the full run, with the retry |
| tissot_run_new.log | Log of the new-mode run |
| tissot_run_bad_sentinel.log | Log of the wrong-sentinel run |
Sample: first 7 rows of tissot_products_sample.csv
| sku | name | collection | category | price_value | price_currency | stock_status | movement_type | case_material | case_diameter_mm | water_resistance | dial_color |
|---|---|---|---|---|---|---|---|---|---|---|---|
| T1424621605200 | Tissot Telemeter 1938 42mm | Telemeter 1938 | Classic Traditional | 2400.0 | USD | InStock | Swiss automatic | 316L stainless steel | 42.0 | Water-resistant up to a pressure of 3 bar (30 m / 100 ft) | Black |
| T1214204705105 | Tissot T-Touch Connect Solar Jungfraubahn 47.5mm | T-Touch Connect Solar Jungfraubahn | Sport Connect | 1250.0 | USD | OutOfStock | Swiss quartz solar tactile | Titanium | 47.5 | Water-resistant up to a pressure of 10 bar (100 m / 330 ft) | |
| T0062071109600 | Tissot Le Locle 29mm | Le Locle | Classic Traditional | 950.0 | USD | InStock | Swiss automatic | 316L stainless steel | 29.0 | Water-resistant up to a pressure of 3 bar (30 m / 100 ft) | Green |
| T1374071104103 | Tissot PRX 40mm | PRX | Classic Contemporary | 850.0 | USD | InStock | Swiss automatic | 316L stainless steel | 39.5 | Water-resistant up to a pressure of 10 bar (100 m / 330 ft) | Blue |
| T0064071103300 | Tissot Le Locle 39.3mm | Le Locle | Classic Traditional | 795.0 | USD | InStock | Swiss automatic | 316L stainless steel | 39.3 | Water-resistant up to a pressure of 3 bar (30 m / 100 ft) | Silver |
| T1504171105100 | Tissot PR 100 Tour de France 40mm | PR 100 Tour de France | Classic Contemporary | 515.0 | USD | OutOfStock | Quartz chronograph PowerDrive | 316L stainless steel | 40.0 | Water-resistant up to a pressure of 10 bar (100 m / 330 ft) | Black |
| T8624101904200 | Tissot Savonnette 48.5mm | Savonnette | Classic Traditional | 425.0 | USD | InStock | Quartz three hands | 316L stainless steel | 48.5 | Blue gradient |
hamiltonwatch.com, Magento, in-browser XHR
Hamilton recon and stores
Recon recommends the cheap render mode from parsing, and the store locator is the site's Magento REST endpoint, which answers XML to a browser navigation and JSON only to an XHR with an Accept header, so the call is made from inside the loaded locator page.
Recon
python3 $SK/recon.py https://www.hamiltonwatch.com --pdp https://www.hamiltonwatch.com/en-us/h32475140-jazzmaster-auto.html
raw 542 KB and rendered 504 KB both carry JSON-LD, the price 1125 and 35 spec labels; recommendation render=False; 13 billed requests
Stores
python3 $SK/stores.py --brand hamilton --browser-api 'https://www.hamiltonwatch.com/en-us/rest/V1/siprmisstorelocator/store-list/coord/39.8283/-98.5795/9999' --page https://www.hamiltonwatch.com/en-us/store-locator/united-states/ --headers '{"Accept":"application/json"}' --records items --country-filter US --map '{"store_name":"name","store_type":"store_type","address":"address","city":"city","state_region":"region","country_code":"country_code","latitude":"latitude","longitude":"longitude","store_url":"url"}' --out output/stores/hamilton.csv2,953 records worldwide, 358 US rows written, 1 billed request
Result files
| File | What it holds |
|---|---|
| hamilton_recon.md | Recon report |
| hamilton_stores.csv | 358 US doors |
| hamilton_stores.log | Run log |
Sample: first 10 rows of hamilton_stores.csv
| store_name | store_type | address | city | state_region | latitude | longitude | store_url |
|---|---|---|---|---|---|---|---|
| Watch Works | IR | 601 N Ridge Rd Suite 109 | Wichita | Kansas | 37.692433 | -97.426591 | https://www.hamiltonwatch.com/en-us/store-loca… |
| Borsheim Jewelry Co | IR | 120 Regency Pkwy | Omaha | Nebraska | 41.259412 | -96.071623 | https://www.hamiltonwatch.com/en-us/store-loca… |
| Dillards | IR | 11601 W 95th St | Overland Park | Kansas | 38.952456 | -94.720311 | https://www.hamiltonwatch.com/en-us/store-loca… |
| Karat's Jewelers | IR | 12260 Blue Valley Parkway | Overland Park | Kansas | 38.8832963 | -94.6856854 | https://www.hamiltonwatch.com/en-us/store-loca… |
| Meierotto Jewelers | IR | 1900 diamond parkway | North Kansas City | Missouri | 39.18755460000001 | -94.5286294 | https://www.hamiltonwatch.com/en-us/store-loca… |
| Drenon Jewelers | IR | 3727 Bolger Rd | Independence | Missouri | 39.077301 | -94.39120800000001 | https://www.hamiltonwatch.com/en-us/store-loca… |
| Rogers & Hollands | IR | 5000 W Empire Place | Sioux Falls | South Dakota | 43.511223 | -96.7759929 | https://www.hamiltonwatch.com/en-us/store-loca… |
| Dillards | IR | 4001 W 41st St | Sioux Falls | South Dakota | 43.5138795 | -96.7748051 | https://www.hamiltonwatch.com/en-us/store-loca… |
| Christopher's Fine Jewelry | IR | 160 Jordan Creek Pkwy Suite 100 | West Des Moines | Iowa | 41.5694198 | -93.8112978 | https://www.hamiltonwatch.com/en-us/store-loca… |
| Espigares | IR | 6550 E 51st St | Tulsa | Oklahoma | 36.0898746 | -95.90386740000001 | https://www.hamiltonwatch.com/en-us/store-loca… |
grand-seiko.com, public store JSON API
Grand Seiko stores
The site's own store API answers a plain GET with a country-wide radius. The JSON arrives wrapped in an HTML <pre> shell, which the helper unwraps, and nested fields are mapped with dotted paths.
Stores
python3 $SK/stores.py --brand grand_seiko --json-api 'https://www.grand-seiko.com/api/v2/stores?...&country=<US id>&unit=99999' --records stores --const '{"country_code":"US"}' --map '{"store_name":"name","store_type":"store_type.0.store_type_name","address":"address","city":"city.0.name","state_region":"state.state_name","postal_code":"zip_code","phone":"phone","latitude":"latitude","longitude":"longitude","store_url":"store_url"}' --out output/stores/grand_seiko.csv100 US rows, 1 billed request
Result files
| File | What it holds |
|---|---|
| grand_seiko_stores.csv | 100 US doors |
| grand_seiko_stores.log | Run log |
Sample: first 10 rows of grand_seiko_stores.csv
| store_name | store_type | address | city | state_region | postal_code | phone | latitude | longitude |
|---|---|---|---|---|---|---|---|---|
| Grand Seiko Boutique Miami | Grand Seiko Boutique | The Moore – Sun Room, 4040 NE 2nd Avenue | Miami | Florida | 33137 | 786-360-6869 | 25.8133419 | -80.1928298 |
| Grand Seiko Boutique Ala Moana, Honolulu | Grand Seiko Boutique | 1450 Ala Moana Blvd., Space 2008 | Honolulu | Hawaii | 96814 | 808-427-5676 | 21.291462701016926 | -157.8443862133348 |
| Grand Seiko Boutique Beverly Hills | Grand Seiko Boutique | 439 North Rodeo Drive | Beverly Hills | California | 90210 | 310-724-8170 | 34.069906918852915 | -118.40370733245548 |
| Grand Seiko Boutique Boston, Copley | Grand Seiko Boutique | 100 Huntington Ave. Suite B023C | Boston | Massachusetts | 02116 | 617-207-2778 | 42.347283479357564 | -71.07782315241694 |
| Grand Seiko Boutique Kalakaua Avenue, Honolulu | Grand Seiko Boutique | 2300 Kalakaua Avenue, Space 101 | Honolulu | Hawaii | 96815 | 808-830-4995 | 21.27850053665757 | -157.82657295944256 |
| Grand Seiko Boutique Lenox Square, Atlanta | Grand Seiko Boutique | 3393 Peachtree Rd NE, Unit 3090C | Atlanta | Georgia | 30326 | 470-408-3803 | 33.84692751738703 | -84.36370578786924 |
| Grand Seiko Boutique New York, SoHo | Grand Seiko Boutique | 154 Spring Street | New York | New York | 10012 | 646-693-0893 | 40.724224 | -74.001672 |
| Grand Seiko Boutique Topanga | Grand Seiko Boutique | 6600 Topanga Canyon Blvd, Suite 1082C | Canoga Park | California | 91303 | 818-839-8267 | 34.18901815004638 | -118.60341357372853 |
| Grand Seiko Flagship Boutique New York | Grand Seiko Boutique | 540 Madison Avenue | New York | New York | 10022 | 212-355-3718 | 40.7609424 | -73.9737512 |
| Bachendorf's | Grand Seiko Salon | 8400 Preston Road | Dallas | Texas | 75225 | 214-692-8400 | 32.864667388157514 | -96.80380533881183 |
The skill files
Download the skill
The skill is a folder of plain text and Python: one SKILL.md that Claude reads, seven scripts, three references and an example hook. Copy the folder into a repo's .claude/skills/ (or ~/.claude/skills/ for every project), install two packages, set NIMBLE_API_KEY, and ask Claude in plain words.
Install
cp -r catalog-extract <repo>/.claude/skills/pip install nimble_python beautifulsoup4export NIMBLE_API_KEY=... # or a .env in the working tree
The skill triggers on phrases like "track competitor prices", "add a brand to the tracker", "store locator scrape" and "which retailers carry".
| File | What it holds | |
|---|---|---|
| SKILL.mddownload | The workflow Claude follows: recon, discovery, spec map, smoke test, run, stores, retailers, report, and the eight rules | 271 lines |
| README.mddownload | Folder map, install, what was verified | 31 lines |
| scripts/nx.pydownload | Nimble Extract helper: fetch with the truncation guard, JSON unwrap, in-browser fetch for POST and XHR APIs, cookie warm-up, billed-request counter. Also a CLI probe | 373 lines |
| scripts/recon.pydownload | Site and product-page reconnaissance: sitemaps, Shopify, platform hints, raw versus rendered, spec labels | 206 lines |
| scripts/discover.pydownload | Sitemap, Shopify products.json or paginated listing to urls.csv | 189 lines |
| scripts/parse_product.pydownload | JSON-LD, __NEXT_DATA__, inline state and spec harvest to a canonical row, with a spec map | 434 lines |
| scripts/run_catalog.pydownload | The product loop: threads, sentinel guard, hook, filtered versus failed, append-only CSV, failure-rate and volume-drop tripwires | 227 lines |
| scripts/stores.pydownload | Store locators: JSON GET, in-browser POST or XHR, metro-grid sweep, field map to the store schema | 199 lines |
| scripts/csvio.pydownload | The three schemas and the per-row-flushing, header-guarded CSV writer | 154 lines |
| hooks/example_hook.pydownload | Where site-specific logic goes: prepare, fetch and parse | 66 lines |
| references/extract-field-notes.mddownload | Measured Nimble behaviour: truncation, sessions, cookies, <pre> JSON, batch endpoint | 113 lines |
| references/site-recipes.mddownload | Worked recipes for the ten brands and eleven retailers, platform fingerprints, store mechanisms | 91 lines |
| references/schema.mddownload | Canonical columns, the active-catalog window rule, SKU lifecycle | 90 lines |
| references/us_metro_grid.csvdownload | 28 metro points for locators capped at N-nearest | 29 lines |
| requirements.txtdownload | nimble_python and beautifulsoup4 | 2 lines |
| catalog-extract.zip | The whole skill, ready to drop into .claude/skills/ |
Nimble documentation