Web Scraping

puppeteer / puppeteer

Puppeteer is a JavaScript library providing a high-level API to control Chrome or Firefox via DevTools Protocol or WebDriver BiDi. It runs headless by default and is widely used for web scraping, testing, and automation.

unclecode / crawl4ai

Open-source web crawler optimized for LLMs, converting web content into clean Markdown for AI applications. Features async processing, browser automation, and structured data extraction.

scrapy / scrapy

Scrapy is a powerful Python framework for web crawling and scraping, providing a complete toolkit for extracting structured data from websites efficiently and at scale.

NanmiCoder / MediaCrawler

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫

D4Vinci / Scrapling

Scrapling is an adaptive Python web scraping framework that handles everything from single requests to full-scale crawls. Its smart parser automatically relocates elements after website changes, built-in fetchers bypass anti-bot systems like Cloudflare, and the spider framework supports concurrent crawling with pause/resume, proxy rotation, and AI integration via MCP server.

cheeriojs / cheerio

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

gocolly / colly

Elegant Scraper and Crawler Framework for Golang

Y2Z / monolith

⬛️ CLI tool and library for saving complete web pages as a single HTML file