Repository navigation
web-scraping
- Website
- Wikipedia
Scrapy, a fast high-level web crawling & scraping framework for Python.
The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
🔥 Open Source No Code Web Data Extraction Platform. Turn Websites To APIs & Spreadsheets With No-Code Robots In Minutes 🔥
Python APIs for web automation, testing, and bypassing bot-detection.
Lighter web automation with Python
List of libraries, tools and APIs for web scraping and data processing.
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
A Chrome DevTools Protocol driver for web automation and scraping.
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
Snoop — инструмент разведки на основе открытых данных (OSINT world)
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
🕷️ An undetectable, powerful, flexible, high-performance Python library that makes Web Scraping simple and easy again!
Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.
Web Scraping Framework
Learn Python for the next 30 (or so) Days.