Web Scraping

D4Vinci / Scrapling

A Python web scraping framework covering the full workflow from a single request to a full-scale crawl: stealth browsers with Cloudflare bypass, adaptive CSS/XPath parsing, and Scrapy-like concurrent spiders with proxy rotation and MCP support for AI agents.

puppeteer / puppeteer

A Node.js library providing a high-level API to control Chrome and Firefox via DevTools Protocol or WebDriver BiDi. Supports headless mode by default, ideal for web scraping, automated testing, screenshot capture, PDF generation, and browser automation workflows.

unclecode / crawl4ai

Open-source web crawler optimized for LLMs, converting web content into clean Markdown for AI applications. Features async processing, browser automation, and structured data extraction.

NanmiCoder / MediaCrawler

A multi-platform social media crawler supporting Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. Built on Playwright with CDP mode, it offers keyword search, post detail and nested comment extraction, creator profile crawling, IP proxy rotation, login state caching, comment word cloud generation, WebUI, and multiple storage backends including CSV, JSON, Excel, SQLite, and MySQL.

scrapy / scrapy

Scrapy is a powerful Python framework for web crawling and scraping, providing a complete toolkit for extracting structured data from websites efficiently and at scale.

cheeriojs / cheerio

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

gocolly / colly

Elegant Scraper and Crawler Framework for Golang

Y2Z / monolith

⬛️ GUI/CLI tool and library for saving complete web pages as a single HTML file