catalog-extract/references/extract-field-notes.md · md · download
# Nimble Extract field notes
Measured behaviour from months of nightly production runs (10 brands, 11
retailers, ~4,900 products, ~9,400 stores). Each item shaped a line of code in
`scripts/nx.py`.
## 1. Silent body truncation (the significant one)
`status == "success"` with the body cut off mid-document. The truncation is
clean (closing `</pre>` / `</html>` intact), so status proves nothing.
| Measured 2026-07-27 | true size | delivered |
|---|---|---|
| Timex `products.json?limit=250` | 840 KB | 15-60 KB, failed to parse |
| Fossil PDPs, 8 of 8 cold fetches | ~600 KB | `<head>` only, ~10 KB |
| Shopify `limit=50` | ~170 KB | truncated at 79 KB |
Smaller pages are **not** safer: it behaves like a render race, not a size
cap. Dangerous rather than flaky because a truncated page keeps whatever
loads early (Fossil's `dataLayer` sits in `<head>`), so a row parses with a
name and price and every spec silently empty.
Mitigation, applied to every fetch that can be large or JS-hydrated:
1. Always pass a `wait` browser action: 8 s for big JSON, 4-5 s for PDPs.
2. Validate by **parsing the payload or a DOM sentinel**, never by status.
3. Retry on guard failure; return a counted failure rather than a hollow row.
Impact of adding the guard on TAG Heuer: `case_material` nulls 35 -> 1,
movement nulls 35 -> 1, unpriced 25 -> 12, `case_diameter` nulls 15 -> 0.
Exactly one product was genuinely spec-less. The "35 no-spec pieces" had
been an API artifact treated as site truth for weeks.
## 2. Sticky sessions hang (since 2026-07-29)
`session={"id": ..., "prefetch_userbrowser": true}` renders began hanging
8-12 minutes and then returning HTTP 500 "can't download the query response".
Identical sessionless calls returned in seconds.
Replacement for Akamai-protected edges (Signet: Kay / Jared / Zales, which
serve an empty app shell to a cold client): fetch the homepage with a
`get_cookies` browser action, sanitise the jar, pass `cookies=` on every grid
fetch. Faster too: warm 12 s + grid 32 s vs ~3 min on a good session day.
`nx.warm_cookies(host)` does this. Nimble's own `get_cookies` output fails
Nimble's `cookies=` validation verbatim (`sameSite` capitalised, `expires`
in a rejected shape); `nx.clean_cookie` keeps name/value/domain/path/
sameSite/secure only.
## 3. Smaller API friction
- **Empty `browser_actions` is rejected** (HTTP 400 "must be a non-empty
array"). Omit the key when there is nothing to run.
- **JSON endpoints arrive HTML-wrapped**: `<html>...<pre>{json}</pre></html>`,
HTML-escaped. `nx.json_from_html` unwraps and unescapes, falling back to the
whole body.
- **Magento REST returns XML to a browser navigation** regardless of `Accept`;
an in-browser XHR with `Accept: application/json` gets JSON. Use
`nx.browser_fetch(locator_page, [{"url": api, "headers": {"Accept": "application/json"}}])`.
- **In-browser `fetch` results** come back at `resp.data.fetch` as
`{url, status, body}`; status 200 + parseable body is the only success.
- **List-shaped `parsing`** (WSA results) emits a Pydantic serializer warning
on every call. Harmless; `nx` silences it by message prefix.
- **`driver`**: `vx10-pro` is the production default for both render modes.
`vx12-pro` is the next escalation when a site serves a shell.
## 4. Where Extract beat a Web Search Agent
Store locators started as WSAs and moved to Extract against the site's own
store API: cheaper, faster, exact. The Grand Seiko store WSA returned 0 rows
for every country code; the public `/api/v2/stores` GET returns 102 doors in
one call. Only Longines and Tissot stayed on a WSA (no reachable API).
Fossil: the site's own `Stores-FindStores` controller is a legacy dead end
that always answers `stores: []` (three wasted attempts). The live locator is
a Where2GetIt / Brandify widget in a cross-origin iframe, so its request never
appears in a network panel scoped to fossil.com. One POST to
`hosted.where2getit.com/fossil/rest/locatorsearch` with the widget's public
appkey (read from the widget page), radius 5000 from the centre of the lower
48 and limit 10000, returns all 1,528 US doors in a single ~15 MB response.
The 100-mile cap is a UI dropdown limit only.
## 5. Batch endpoint for sweeps
For thousands of PDPs, `POST https://sdk.nimbleway.com/v1/extract/batch`
with `{"shared_inputs": {...extract params...}, "inputs": [{"url": ...}, ...]}`
queues server-side: ~500 URLs/min vs ~0.5/s from a local thread pool
(5,677 PDPs in 10.9 min, 0 failures, 400 URLs per batch, 4 in flight).
Poll `/v1/batches/<id>/progress`; download each task from
`/v1/tasks/<id>/results`. Set `driver` explicitly to `vx8` or above:
`vx6` / unset can sit in `pending` forever. Results still need the same
sentinel / parse validation as realtime fetches. `nimble extract batch
--input ... --shared-inputs ...` exposes the same endpoint from the CLI (1.5.0).
## 6. CLI equivalents (for probes, not loops)
```bash
nimble extract run --url URL --render true --format html --country US --driver vx10-pro \
--browser-action '{"wait":{"duration":4000}}' --request-timeout 120000 > page.json
python3 -c "import json;d=json.load(open('page.json'));print(d['status'],len(d['data']['html']))"
```
Response shape: `{url, task_id, status, data: {html, browser_actions|fetch,
cookies}, metadata, status_code, warnings}`.
## 7. Open items raised with Nimble (2026-08)
1. A completeness signal on the response (expected vs delivered length, or
`truncated: true`) would replace DOM sentinels. Largest source of silent
data corruption.
2. Sticky sessions unusable since 2026-07-29.
3. `get_cookies` -> `cookies=` round-trip should be lossless.
4. A JSON passthrough format for raw endpoints would remove the `<pre>` step.