crawl4ai

craw4ai

https://github.com/unclecode/crawl4ai/blob/main/Dockerfile

https://github.com/unclecode/crawl4ai

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.

 

Why developers pick Crawl4AI

• LLM ready output, smart Markdown with headings, tables, code, citation hints
• Fast in practice, async browser pool, caching, minimal hops
• Full control, sessions, proxies, cookies, user scripts, hooks
• Adaptive intelligence, learns site patterns, explores only what matters
• Deploy anywhere, zero keys, CLI and Docker, cloud friendly

 

📝 Markdown Generation 📊 Structured Data Extraction 🌐 Browser Integration 🔎 Crawling & Scraping 🚀 Deployment 🎯 Additional Features

 

 

https://github.com/fanqingsong/craw4ai-fastapi

https://github.com/fanqingsong/Visualized-Craw4Ai

https://github.com/fanqingsong/webscraping-craw4ai

https://github.com/coleam00/mcp-crawl4ai-rag

 

https://docs.crawl4ai.com/core/docker-deployment/#prerequisites

3. Run the Container

• Basic run:

  docker run -d \
    -p 11235:11235 \
    --name crawl4ai \
    --shm-size=1g \
    unclecode/crawl4ai:latest
  

•  

• With LLM support:

  # Make sure .llm.env is in the current directory
  docker run -d \
    -p 11235:11235 \
    --name crawl4ai \
    --env-file .llm.env \
    --shm-size=1g \
    unclecode/crawl4ai:latest