Challenge
The sources used different page structures, loaded data dynamically, and applied anti-bot controls. A one-off script would fail silently and leave the downstream dataset incomplete.
Approach
We built browser-driven extraction with source-specific selectors, retry paths, validation, and normalized output. The workflow separated navigation, parsing, and delivery so failures could be isolated instead of rerunning the entire job.
What changed
- More than 100 sites could be processed through one observable workflow.
- Validation caught incomplete records before they reached the client dataset.
- Source changes could be repaired without rewriting the whole pipeline.