#crawling · 8sync News

100% Automated: How I Built a Self-Healing Price Scraper for 163 Shipping Providers

A solo engineer describes building a fully automated, self-healing price scraper for 163 European shipping providers using an agentic task graph. The system runs updater scripts per provider, detects failures, and dispatches AI agents (Claude Code CLI, OpenAI Codex CLI, Cursor) to diagnose broken scrapers, rewrite scripts, and verify prices against live carrier websites. Key design patterns include: a layered architecture separating the calculator, updater scripts, and agentic loop; pluggable LLM routing with fallbacks and cooldown handling for rate limits; deterministic pre-checks before invoking LLMs; and a two-agent verifier pattern where a second agent reviews the first's output cold. Feedback from failed verifications is written into the next agent's prompt as structured context. The post concludes with practical takeaways for engineers building agentic pipelines, including single-responsibility agents, independent verifiers, and model-to-task matching.

The emergence of the web data infrastructure layer for AI

As AI systems move beyond static training data, enterprises face a growing need for real-time web data infrastructure. Traditional model training on fixed snapshots is insufficient for use cases like dynamic pricing, market tracking, and reducing hallucinations. A new infrastructure layer — capable of emulating human browsing at massive scale, navigating anti-bot protections, and delivering structured data with low latency — is emerging to fill this gap. Bright Data's CEO argues that AI intelligence without a live knowledge layer is practically useless, and that 97% of AI organizations depend on real-time web data yet 90% feel constrained by access restrictions. Compliance with GDPR and CCPA is addressed through consent-based networks and public-data-only policies. This sponsored content promotes Bright Data's web data platform as a solution.

Raspberry Pi Locator Website To Shut Down In July

Rpilocator.com, a popular site tracking Raspberry Pi stock and pricing across online retailers, is shutting down in July 2026. The main reason is that the site's indexing bot has been blocked by most shopping sites, likely due to anti-scraping measures aimed at AI bots. The site still serves around 11,000 monthly users but the effort to maintain workarounds became unsustainable. The closure also reflects a broader shift in the single-board computer landscape: the chip shortage era that made such a tracker essential has passed, replaced by concerns over rising memory prices pushing Raspberry Pi boards into luxury price territory.

PyCoder’s Weekly

PyCoder's Weekly Issue #740 curates Python news and tutorials including a deep dive into the Pluggy plugin system used by pytest and tox, a guide on implementing interfaces with ABCs and Protocols, a tutorial on building Scrapy extensions, and coverage of Python 3.14 garbage collection changes. Also featured: a comparison of Python task queue libraries (Celery, Dramatiq, FastStream, Taskiq, Repid) for 2026, security concerns around AI code completions in PyCharm, PSF board election dates, multiple PEPs marked Final for Python 3.15 beta, and a roundup of security topics from PyCon US 2026. Projects highlighted include NVIDIA's Warp GPU simulation library, python-socketio, and Facebook Research's Hydra configuration framework.