solocrawl

Environment

• SOLOCRAWL_LOG_FILE

  Optional log file path (also logs to stderr)

• SOLOCRAWL_LOG_LEVEL

  Log level: DEBUG, INFO, WARNING, ERROR

• SOLOCRAWL_PROXY_LIST

  Comma-separated proxy URLs

• SOLOCRAWL_PROXY_MODE

  Proxy mode: list (rotate a pool) or endpoint (single rotating endpoint)

• SOLOCRAWL_USER_AGENT

  Override HTTP User-Agent for API requests

• SOLOCRAWL_MAX_RETRIES

  Retries on network errors / rate limits

• SOLOCRAWL_SEARXNG_URL

  Base URL of a self-hosted SearXNG instance (enables the searxng provider)

• SOLOCRAWL_PROXY_ENABLED

  Enable optional proxy layer

• SOLOCRAWL_PROXY_ENDPOINT

  Single rotating proxy endpoint

• SOLOCRAWL_PROXY_PASSWORDsecret

  Proxy auth password

• SOLOCRAWL_PROXY_USERNAME

  Proxy auth username

• SOLOCRAWL_RESPECT_ROBOTS

  Honour robots.txt on scrape (fail-open); set false to skip

• SOLOCRAWL_BROWSER_ALLOWED

  Allow Playwright fallback when installed

• SOLOCRAWL_MAX_CONCURRENCY

  Global fetch concurrency limit

• SOLOCRAWL_TIMEOUT_SECONDS

  Per-request timeout in seconds

• SOLOCRAWL_ENABLE_PROVIDERS

  Comma-separated opt-in provider names

• SOLOCRAWL_PER_DOMAIN_LIMIT

  Per-domain concurrency limit

• SOLOCRAWL_CACHE_TTL_SECONDS

  In-memory fetch cache TTL in seconds (0 = disabled)

• SOLOCRAWL_MAX_RESPONSE_BYTES

  Cap on fetched response body size (10 MiB)

• SOLOCRAWL_ALLOW_INTERNAL_URLS

  Allow scraping localhost/private IPs (dev only)