Web Scraping Platform - Phase 1

Large-scale web scraping platform with Python workers, MongoDB storage, and Redis job queue.

Features

• Multiple scraper types: HTTP (HTML) and API (JSON) scrapers
• Flexible extraction: CSS selectors, XPath, and JSONPath support
• Rate limiting: Per-domain rate limiting via Redis sliding window
• Retry logic: Exponential backoff with configurable retry attempts
• Pipeline processing: Composable steps for cleaning, transforming, and validating data
• Change detection: Monitor periodic scrapes for content changes
• MongoDB storage: Async storage with automatic TTL for old results
• Redis queue: Distributed job queue with dead letter queue for failures

📚 Documentation

New to the platform? Start with the Tutorial (15 minutes)

Complete documentation:

• Tutorial - Step-by-step getting started guide
• Overview - Complete technical documentation
• Documentation Index - Navigation and search guide

Quick Start

# 1. Install dependencies
uv pip install -e ".[dev]"

# 2. Start services
docker compose up -d mongodb redis

# 3. Verify setup
python scripts/verify_setup.py

# 4. Enqueue a job (new terminal)
python scripts/enqueue_job.py \
  --url "https://quotes.toscrape.com" \
  --type http \
  --selectors '{"quotes": {"selector": "span.text", "type": "css", "multiple": true}}'

# 5. Start worker (new terminal)
python scripts/run_worker.py

# 6. View results at http://localhost:8082 (Mongo Express)

See Tutorial for detailed walkthrough.

Architecture

┌──────────────────┐     ┌──────────────────┐
│      Redis       │────▶│  Python Workers  │
│  (Job Queue)     │◀────│  (Scraper Pool)  │
└────────┬─────────┘     └────────┬─────────┘
         │                         │
         │                         ▼
         │                 ┌──────────────┐
         └────────────────▶│   MongoDB    │
                           │   (Storage)  │
                           └──────────────┘

Setup

Prerequisites

• Python 3.12+
• Docker and Docker Compose (for local dev services)

Installation

1. Clone and navigate to directory:

   cd scraper

2. Install dependencies (using uv, pip, or poetry):

   # Using uv (recommended - fastest)
   uv pip install -e ".[dev]"
   
   # Or using pip
   pip install -e ".[dev]"

3. Set up environment variables:

   cp .env.example .env
   # Edit .env with your configuration

4. Start local services:

   docker compose up -d mongodb redis

   Optional web UIs:

   • Redis Commander: http://localhost:8081
   • Mongo Express: http://localhost:8082 (admin/admin)

Usage

1. Enqueue a Scraping Job

# Scrape HTML page
python scripts/enqueue_job.py \
  --url "https://quotes.toscrape.com" \
  --type http \
  --selectors '{
    "quotes": {"selector": "span.text", "type": "css", "multiple": true},
    "authors": {"selector": "small.author", "type": "css", "multiple": true}
  }'

# Scrape JSON API
python scripts/enqueue_job.py \
  --url "https://api.example.com/data" \
  --type api \
  --selectors '{
    "items": {"selector": "data.items", "type": "jsonpath"},
    "total": {"selector": "meta.total", "type": "jsonpath"}
  }' \
  --headers '{"Authorization": "Bearer YOUR_TOKEN"}'

2. Start a Worker

python scripts/run_worker.py

# With debug logging
python scripts/run_worker.py --log-level DEBUG

3. Check Results

Query MongoDB directly or use Mongo Express (http://localhost:8082):

// In MongoDB shell or Mongo Express
db.results.find().sort({scraped_at: -1}).limit(10)

Configuration

Edit .env to configure:

• MongoDB: MONGO_URI, MONGO_DB_NAME
• Redis: REDIS_URL, REDIS_JOB_QUEUE
• Worker: WORKER_CONCURRENCY, WORKER_MAX_RETRIES
• Rate Limiting: RATE_LIMIT_DEFAULT_REQUESTS, RATE_LIMIT_DEFAULT_WINDOW_SECONDS
• HTTP: HTTP_TIMEOUT_SECONDS, HTTP_USER_AGENT
• Storage: RESULT_TTL_DAYS, STORE_RAW_HTML

Extractor Configuration

CSS Selectors (HTTP scraper)

{
  "fields": {
    "title": {
      "selector": "h1.title",
      "type": "css",
      "attribute": "text"
    },
    "links": {
      "selector": "a.product-link",
      "type": "css",
      "attribute": "href",
      "multiple": true
    }
  }
}

XPath (HTTP scraper)

{
  "fields": {
    "price": {
      "selector": "//span[@class='price']/text()",
      "type": "xpath"
    }
  }
}

JSONPath (API scraper)

{
  "fields": {
    "items": {
      "selector": "data.items[*]",
      "type": "jsonpath"
    },
    "next_page": {
      "selector": "links.next",
      "type": "jsonpath"
    }
  }
}

Testing

# Run all tests
pytest

# Run with coverage
pytest --cov=scraper --cov-report=html

# Run specific test file
pytest tests/test_http_scraper.py

Development

Project Structure

scraper/
├── scraper/
│   ├── models/          # Job and Result schemas
│   ├── scrapers/        # HTTP and API scrapers
│   ├── extractors/      # CSS, XPath, JSONPath extractors
│   ├── config.py        # Configuration management
│   ├── db.py            # MongoDB connection
│   ├── worker.py        # Job consumer
│   ├── pipeline.py      # Post-scrape processing
│   └── rate_limiter.py  # Redis rate limiter
├── scripts/             # CLI utilities
└── tests/               # Test suite

Adding a New Extractor

1. Create scraper/extractors/my_extractor.py
2. Inherit from BaseExtractor
3. Implement extract(content, config) method
4. Register in scraper/extractors/__init__.py
5. Add tests in tests/test_extractors.py

Troubleshooting

Worker not processing jobs:

• Check Redis connection: redis-cli ping
• Verify job queue has items: redis-cli LLEN scraper:jobs
• Check worker logs for errors

MongoDB connection failed:

• Ensure MongoDB is running: docker compose ps
• Verify MONGO_URI in .env

Rate limit too strict:

• Adjust RATE_LIMIT_DEFAULT_REQUESTS and RATE_LIMIT_DEFAULT_WINDOW_SECONDS in .env
• Check rate limit stats in Redis: redis-cli ZRANGE rate_limit:example.com 0 -1 WITHSCORES

Next Phases

• Phase 2: Elixir orchestrator with Phoenix API and job scheduler
• Phase 3: Domma dashboard for job management and monitoring
• Phase 4: Proxy rotation, headless browsers, distributed deployment

License

MIT