Universal Web Scraper — AI Discovery & Crawling Platform

Problem

Construction project data sits on thousands of publisher websites with no shared structure. Finding and validating new sources by hand doesn't scale. The hackfest brief was to build something that could discover and crawl them automatically in about 98 hours.

Approach

• Built the tiered crawler layer with a Strategy pattern, escalating from plain HTTP requests up to headless Chromium for JavaScript-heavy sites.
• Covered that layer with 113 tests across strategy selection, escalation, and extraction.
• Wired the discovery, crawling, and analytics modules together end to end.
• Added xlsx export and applied ConstructConnect brand theming across the UI.
• Used FastAPI and PostgreSQL to stream crawl progress live into a React and TypeScript front end.
• Used the Anthropic Claude API for source discovery and content extraction.

Outcome

A working platform built from scratch in under 100 hours, and the team finished as a top-16 finalist out of 48. Good practice at owning a full stack on a hard deadline.