Web Scraping

puppeteer / puppeteer

A Node.js library providing a high-level API to control Chrome and Firefox via DevTools Protocol or WebDriver BiDi. Supports headless mode by default, ideal for web scraping, automated testing, screenshot capture, PDF generation, and browser automation workflows.

unclecode / crawl4ai

Open-source web crawler optimized for LLMs, converting web content into clean Markdown for AI applications. Features async processing, browser automation, and structured data extraction.

D4Vinci / Scrapling

Scrapling is an adaptive Python web scraping framework that handles everything from single requests to full-scale crawls. Its smart parser automatically relocates elements after website changes, built-in fetchers bypass anti-bot systems like Cloudflare, and the spider framework supports concurrent crawling with pause/resume, proxy rotation, and AI integration via MCP server.

scrapy / scrapy

Scrapy is a powerful Python framework for web crawling and scraping, providing a complete toolkit for extracting structured data from websites efficiently and at scale.

NanmiCoder / MediaCrawler

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫

cheeriojs / cheerio

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

gocolly / colly

Elegant Scraper and Crawler Framework for Golang

Y2Z / monolith

⬛️ CLI tool and library for saving complete web pages as a single HTML file