Clubworks Data Platform.
A distributed scraping platform that ingests 30GB/month across 12 locations, 2 providers, and 6 timeslots — in under 3 minutes per cycle, down from 45.
- Founding Engineer @ Clubworks (Contract)
- 2025 — 2026
- Data Pipeline
Working is not the same as scalable.
Clubworks needed a way to systematically scrape and index location-based pricing data from multiple providers. I built the entire system solo — from scrapers to task orchestration to the frontend dashboard. Two major iterations: the first worked, the second is production-grade.
V1 worked, but it hit a ceiling
Jobs ran sequentially through BullMQ, Selenium-based scrapers couldn't scale, and a full scrape took 45 minutes. Individual scraper concurrency and multi-scraper parallelism became bottlenecks at production volume.
V2 decoupled everything.
Celery + Celery Beat for scheduling, Playwright for browser automation, FlareSolverr for complex cookie management. Added circuit breakers, dynamic rate limiting, cookie jar to reduce proxy costs. Backend consumes Redis Streams with parallel batch processing. The architecture now scales.
one at a time
20 concurrent Celery workers
Scrapes 12 locations × 2 providers × 6 timeslots in under 3 minutes, down from 45 (15×)
Celery Beat schedules daily/weekly/monthly/yearly scrapes.
In under 3 minutes per cycle, down from 45.
- 12 x 2 x 6
- < 3 min
- ~30 GB / mo
- 15×
- Scrapes 12 locations × 2 providers × 6 timeslots in under 3 minutes, down from 45 (15×)
- Extracts ~30GB/month of pricing data, stored in NeonDB with snowflake-style aggregation
- 20 concurrent Celery workers with tuned threading for Playwright + FlareSolverr sessions
- Redis TTL caching (3 min) for hot data, eliminating frontend latency spikes
V1 taught me that 'working' isn't the same as 'scalable.' It worked, but when we needed more concurrency, the architecture couldn't support it. V2 was about building room to grow. Celery made all the difference.