· Product updates · Donatas
Elements Mapper reliability, row containers, and queue improvements
We fixed Elements Mapper add/remove sync, added row-container XPaths for list scraping, queued preview refetch with loading UI, and improved Pages enable/disable plus scrape queue visibility and worker shutdown.
We shipped a batch of improvements aimed at trust in the Elements Mapper, clearer control over list scraping, and more honest queue visibility on the Pages screen.
Elements Mapper: mapping that sticks
The mapper could lose your work right after you clicked to add or remove a field. The preview iframe and localStorage were syncing over in-progress marks before the server save finished.
- Server-first on load — stale mark keys are cleared before the preview boots; the UI re-seeds from the database.
- No ghost sync — form XPath lists are not pushed back into localStorage on iframe storage events.
- Safer live updates — Echo broadcasts during an in-flight edit are ignored so the UI does not roll back mid-click.
- Clearer click behavior — map, remove, and layout groups behave more predictably for list vs carousel HTML.
Tip: Click an element in the preview to map it. Click a highlight again to remove. Hold Alt (Option on Mac) to exclude a specific instance from preview and scraped data.
Row containers (map anchor XPaths)
Listing pages often mix product tiles, carousels, and sponsored blocks. Field XPaths alone are not always enough — the scraper needs to know which repeating blocks are rows.
On Page URL & schedule in the mapper you can set Row containers: one XPath per line, optional. Leave empty to auto-detect from mapped fields; detected paths are shown for reference. Values are stored on the page and sent on the next scrape so field selectors run inside each container, not across the whole document.
This works together with layout groups (multiple XPath variants per field name) and Alt+click exclusions.
Preview refetch without blocking
Refetch preview now runs on the dedicated elements queue instead of blocking the HTTP request. The UI shows a loading state, polls until the preview is ready, and surfaces errors if the refetch fails. Duplicate refetches for the same page are guarded.
Embed preview HTML also uses stylesheet inlining so saved pages look closer to the live site inside the iframe.
Pages list: enable/disable and live traffic
On Pages you can enable or disable a page without opening the editor. Re-enabling resets failure counts so a recovered page is not stuck in a penalty state.
Scrape lane traffic (queued/running, lane bars) is more accurate: Redis FIFO dequeue was corrected, stale entries are reconciled, and updates combine Echo with polling ordered by snapshot time.
Queue workers for local dev and deploys
- Chunked scrape/app queue wrappers stop cleanly on SIGTERM instead of restarting
queue:workforever afterdocker compose stop. make queue-kill(and scoped variants) force-stops stuck workers when SIGTERM is not enough.- Preview refetch and other new paths use named feature queues (e.g.
elements), notdefault.
What to try
- Open Elements for a page — map fields, set row containers if auto-detect is not enough, use Refetch preview for fresh HTML.
- On Pages, toggle enabled state and watch lane traffic while scrapes run.
- For local queues:
make queue-work-appand your usual scrape workers; usemake queue-killif workers hang after a stop.
| Area | What changed |
|---|---|
| Elements Mapper | Stable add/remove; server-first state; better exclusions and layouts |
| Row containers | Explicit or auto-detected anchor XPaths |
| Preview refetch | Queued, non-blocking, loading + error UI |
| Pages table | Enable/disable; more accurate queue/lane traffic |
| Queues / ops | Clean worker shutdown; make queue-kill for stuck workers |