crawl4ai

craw4ai

https://github.com/unclecode/crawl4ai/blob/main/Dockerfile

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.

 

Why developers pick Crawl4AI

• LLM ready output, smart Markdown with headings, tables, code, citation hints
• Fast in practice, async browser pool, caching, minimal hops
• Full control, sessions, proxies, cookies, user scripts, hooks
• Adaptive intelligence, learns site patterns, explores only what matters
• Deploy anywhere, zero keys, CLI and Docker, cloud friendly

 

📝 Markdown Generation 📊 Structured Data Extraction 🌐 Browser Integration 🔎 Crawling & Scraping 🚀 Deployment 🎯 Additional Features

 

 

https://github.com/fanqingsong/craw4ai-fastapi

https://github.com/fanqingsong/Visualized-Craw4Ai

https://github.com/fanqingsong/webscraping-craw4ai

https://github.com/coleam00/mcp-crawl4ai-rag