proxy.golang.org "web-crawler" keyword
Top 6.0% on proxy.golang.org
41 versions - Latest release: 10 days ago - 84,750 stars on GitHub
github.com/unclecode/crawl4ai v0.9.4 💰
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Mar...41 versions - Latest release: 10 days ago - 84,750 stars on GitHub
Top 5.6% on proxy.golang.org
479 versions - Latest release: 10 days ago - 31,532 stars on GitHub
github.com/scrapegraphai/scrapegraph-ai v2.3.0+incompatible 💰
Python scraper based on AI479 versions - Latest release: 10 days ago - 31,532 stars on GitHub
Top 5.6% on proxy.golang.org
479 versions - Latest release: 10 days ago - 31,532 stars on GitHub
github.com/ScrapeGraphAI/Scrapegraph-ai v2.3.0+incompatible 💰
Python scraper based on AI479 versions - Latest release: 10 days ago - 31,532 stars on GitHub
Top 5.0% on proxy.golang.org
518 versions - Latest release: 5 months ago - 188,327 stars on GitHub
github.com/firecrawl/firecrawl v3.0.0+incompatible
Supercharge your AI agents with data from the web and beyond. Building the library for superintel...518 versions - Latest release: 5 months ago - 188,327 stars on GitHub
Top 5.0% on proxy.golang.org
27 versions - Latest release: 3 days ago - 188,327 stars on GitHub
github.com/firecrawl/firecrawl/apps/go-sdk v1.16.2
Package firecrawl provides a Go client for the Firecrawl v2 web scraping API.27 versions - Latest release: 3 days ago - 188,327 stars on GitHub
Top 4.9% on proxy.golang.org
17 versions - Latest release: about 1 month ago - 7,550 stars on GitHub
github.com/firecrawl/firecrawl-mcp-server v3.24.1+incompatible
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any...17 versions - Latest release: about 1 month ago - 7,550 stars on GitHub
Top 4.7% on proxy.golang.org
11 versions - Latest release: 3 months ago - 5,446 stars on GitHub
github.com/KnockOutEZ/wigolo v0.2.1 💰
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No...11 versions - Latest release: 3 months ago - 5,446 stars on GitHub
Top 4.7% on proxy.golang.org
11 versions - Latest release: 3 months ago - 5,446 stars on GitHub
github.com/knockoutez/wigolo v0.2.1 💰
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No...11 versions - Latest release: 3 months ago - 5,446 stars on GitHub
Top 5.6% on proxy.golang.org
833 versions - Latest release: 3 months ago - 2,752 stars on GitHub
github.com/spider-rs/spider v2.52.2+incompatible
Foundational low latency web data collecting in Rust833 versions - Latest release: 3 months ago - 2,752 stars on GitHub
Top 8.2% on proxy.golang.org
11 versions - Latest release: over 6 years ago - 11,356 stars on GitHub
github.com/ssssssss-team/spider-flow v0.5.0
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。11 versions - Latest release: over 6 years ago - 11,356 stars on GitHub
Top 4.8% on proxy.golang.org
649 versions - Latest release: 6 days ago - 25,979 stars on GitHub
github.com/apify/crawlee v3.18.2+incompatible
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In ...649 versions - Latest release: 6 days ago - 25,979 stars on GitHub
Top 5.2% on proxy.golang.org
12 versions - Latest release: 5 months ago - 101 stars on GitHub
github.com/Sriram-PR/doc-scraper v1.4.3
Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingest...12 versions - Latest release: 5 months ago - 101 stars on GitHub
Top 5.2% on proxy.golang.org
12 versions - Latest release: 5 months ago - 101 stars on GitHub
github.com/sriram-pr/doc-scraper v1.4.3
Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingest...12 versions - Latest release: 5 months ago - 101 stars on GitHub
Top 4.6% on proxy.golang.org
9 versions - Latest release: 29 days ago - 101 stars on GitHub
github.com/Sriram-PR/doc-scraper/v2 v2.9.2
Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingest...9 versions - Latest release: 29 days ago - 101 stars on GitHub
Top 4.9% on proxy.golang.org
28 versions - Latest release: 25 days ago - 429 stars on GitHub
github.com/internetarchive/Zeno/v2 v2.0.27
State-of-the-art web crawler 🔱28 versions - Latest release: 25 days ago - 429 stars on GitHub
Top 4.9% on proxy.golang.org
28 versions - Latest release: 25 days ago - 429 stars on GitHub
github.com/internetarchive/zeno/v2 v2.0.27
State-of-the-art web crawler 🔱28 versions - Latest release: 25 days ago - 429 stars on GitHub
github.com/internetarchive/Zeno v1.0.85
State-of-the-art web crawler 🔱29 versions - Latest release: over 1 year ago - 429 stars on GitHub
Top 9.9% on proxy.golang.org
29 versions - Latest release: over 1 year ago - 429 stars on GitHub
github.com/internetarchive/zeno v1.0.85
State-of-the-art web crawler 🔱29 versions - Latest release: over 1 year ago - 429 stars on GitHub
Top 4.6% on proxy.golang.org
39 versions - Latest release: 22 days ago - 1 dependent package - 2 dependent repositories - 212 stars on GitHub
github.com/gosom/scrapemate v1.4.0 💰
Golang Crawling and scraping framework39 versions - Latest release: 22 days ago - 1 dependent package - 2 dependent repositories - 212 stars on GitHub
Top 5.7% on proxy.golang.org
1 version - Latest release: over 6 years ago - 0 stars on GitHub
github.com/iAziz786/wego v0.0.0-20200411180604-5a89dbf421b2
A web crawler to keep you updated about things you want to keep track of1 version - Latest release: over 6 years ago - 0 stars on GitHub
Top 8.8% on proxy.golang.org
103 versions - Latest release: 5 months ago - 1 dependent repositories - 372 stars on GitHub
github.com/tech-engine/goscrapy v0.27.17 💰
GoScrapy: High perfomance webscraping framework in Go, inspired by Python's Scrapy.103 versions - Latest release: 5 months ago - 1 dependent repositories - 372 stars on GitHub
Top 1.0% on proxy.golang.org
5 versions - Latest release: 4 months ago - 0 stars on GitHub
github.com/enot-style/go-sitemap-fetcher v1.1.0
Fast, streaming sitemap fetcher for Go with nested sitemap indexes, gzip, robots.txt support, URL...5 versions - Latest release: 4 months ago - 0 stars on GitHub
Top 5.8% on proxy.golang.org
16 versions - Latest release: about 1 year ago - 7,538 stars on GitHub
github.com/mendableai/firecrawl-mcp-server v3.2.1+incompatible
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any...16 versions - Latest release: about 1 year ago - 7,538 stars on GitHub
Top 5.8% on proxy.golang.org
16 versions - Latest release: about 1 year ago - 7,538 stars on GitHub
github.com/vrknetha/mcp-server-firecrawl v3.2.1+incompatible
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any...16 versions - Latest release: about 1 year ago - 7,538 stars on GitHub
Top 5.9% on proxy.golang.org
Latest release: about 2 months ago - 0 stars on GitHub
github.com/mmycin/npminfo
A CLI tool to check the downloads of an NPM package written in GoLatest release: about 2 months ago - 0 stars on GitHub
Top 4.5% on proxy.golang.org
2 versions - Latest release: about 1 month ago - 1,092 stars on GitHub
github.com/scrapfly/scrapfly-scrapers v0.0.0-20260902140739-6d87191c13f0 removed
Scalable Python web scraping scripts for +40 popular domains2 versions - Latest release: about 1 month ago - 1,092 stars on GitHub
Top 5.9% on proxy.golang.org
1 version - Latest release: over 1 year ago - 1 stars on GitHub
github.com/heinrichb/scrapey-cli v0.0.0-20250218200722-9ff2d1f4e8f2
Scrapey CLI is a lightweight, modular command-line tool built in Go for web crawling and scraping...1 version - Latest release: over 1 year ago - 1 stars on GitHub
Top 4.8% on proxy.golang.org
84 versions - Latest release: 8 days ago - 180 stars on GitHub
github.com/xberg-io/crawlberg/packages/go v1.8.0
Package crawlberg provides Go bindings for the crawlberg library.84 versions - Latest release: 8 days ago - 180 stars on GitHub
Top 4.8% on proxy.golang.org
135 versions - Latest release: 8 days ago - 180 stars on GitHub
github.com/xberg-io/crawlberg v1.8.0
High-performance web crawling engine with bindings for 11 languages135 versions - Latest release: 8 days ago - 180 stars on GitHub
Top 6.0% on proxy.golang.org
Latest release: 3 months ago - 0 stars on GitHub
github.com/Huy-Ngo/cov-news
A web crawler that takes content from https://ncov.moh.gov.vn/dong-thoi-gianLatest release: 3 months ago - 0 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 3 days ago - 12,276 stars on GitHub
github.com/crawlab-team/crawlab/trace
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 3 days ago - 12,276 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 3 days ago - 12,276 stars on GitHub
github.com/crawlab-team/crawlab/fs
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 3 days ago - 12,276 stars on GitHub
Top 6.7% on proxy.golang.org
1 version - Latest release: almost 2 years ago - 12,276 stars on GitHub
github.com/crawlab-team/crawlab/db v0.0.0-20241009023509-0485310def8b
Distributed web crawler admin platform for spiders management regardless of languages and framewo...1 version - Latest release: almost 2 years ago - 12,276 stars on GitHub
Top 6.7% on proxy.golang.org
1 version - Latest release: almost 2 years ago - 12,276 stars on GitHub
github.com/crawlab-team/crawlab/core v0.0.0-20241009023509-0485310def8b
Distributed web crawler admin platform for spiders management regardless of languages and framewo...1 version - Latest release: almost 2 years ago - 12,276 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 3 days ago - 12,276 stars on GitHub
github.com/crawlab-team/crawlab/template-parser
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 3 days ago - 12,276 stars on GitHub
Top 6.2% on proxy.golang.org
1 version - Latest release: almost 2 years ago - 12,276 stars on GitHub
github.com/crawlab-team/crawlab/cli v0.0.0-20241120074730-139d7de3ad17
Distributed web crawler admin platform for spiders management regardless of languages and framewo...1 version - Latest release: almost 2 years ago - 12,276 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 3 days ago - 12,276 stars on GitHub
github.com/crawlab-team/crawlab/grpc
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 3 days ago - 12,276 stars on GitHub
Top 8.2% on proxy.golang.org
33 versions - Latest release: about 3 years ago - 12,276 stars on GitHub
github.com/crawlab-team/crawlab v0.6.3
Distributed web crawler admin platform for spiders management regardless of languages and framewo...33 versions - Latest release: about 3 years ago - 12,276 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 3 days ago - 12,276 stars on GitHub
github.com/crawlab-team/crawlab/vcs
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 3 days ago - 12,276 stars on GitHub
Top 5.0% on proxy.golang.org
11 versions - Latest release: 8 months ago - 378 stars on GitHub
github.com/lewisdonovan/google-news-scraper v2.7.1+incompatible 💰
Lightweight scraper for Google News11 versions - Latest release: 8 months ago - 378 stars on GitHub
Top 4.9% on proxy.golang.org
97 versions - Latest release: 5 days ago - 2 stars on GitHub
github.com/mysleekdesigns/crawlforge-mcp v6.15.0+incompatible 💰
31 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, re...97 versions - Latest release: 5 days ago - 2 stars on GitHub
Top 6.4% on proxy.golang.org
Latest release: 5 days ago - 2 stars on GitHub
github.com/psidex/nomad
An experimental web crawler to visualise & map the connections between domainsLatest release: 5 days ago - 2 stars on GitHub
Top 6.5% on proxy.golang.org
83 versions - Latest release: 5 months ago - 186,582 stars on GitHub
github.com/mendableai/firecrawl v3.0.0+incompatible
🔥 Supercharge your AI agents with data from the web and beyond. A web data API to search, scrape,...83 versions - Latest release: 5 months ago - 186,582 stars on GitHub
Top 4.5% on proxy.golang.org
68 versions - Latest release: 7 days ago - 103 stars on GitHub
github.com/Goldziher/basemind v0.27.4 💰
Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, d...68 versions - Latest release: 7 days ago - 103 stars on GitHub
Top 4.5% on proxy.golang.org
68 versions - Latest release: 7 days ago - 102 stars on GitHub
github.com/goldziher/basemind v0.27.4 💰
Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, d...68 versions - Latest release: 7 days ago - 102 stars on GitHub
Top 4.9% on proxy.golang.org
2 versions - Latest release: 5 months ago - 8 stars on GitHub
github.com/geiserx/web-mirror v1.0.1 💰
Mirror any webpage to a local server for offline access2 versions - Latest release: 5 months ago - 8 stars on GitHub
Top 4.9% on proxy.golang.org
1 version - Latest release: 7 months ago - 6 stars on GitHub
github.com/geiserx/media-download v1.0.0 💰
Downloads all media files from any web page into a folder schema1 version - Latest release: 7 months ago - 6 stars on GitHub
Top 6.7% on proxy.golang.org
141 versions - Latest release: about 2 months ago - 1,148 stars on GitHub
github.com/webrecorder/browsertrix-crawler v1.14.3 💰
Run a high-fidelity browser-based web archiving crawler in a single Docker container141 versions - Latest release: about 2 months ago - 1,148 stars on GitHub
Top 6.7% on proxy.golang.org
645 versions - Latest release: 4 months ago - 25,903 stars on GitHub
github.com/apifytech/apify-js v3.17.0+incompatible
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In ...645 versions - Latest release: 4 months ago - 25,903 stars on GitHub
Top 5.3% on proxy.golang.org
1 version - Latest release: almost 5 years ago - 181 stars on GitHub
github.com/saeeddhqan/evine v0.0.0-20211015222239-46051ded7145
Interactive CLI Web Crawler1 version - Latest release: almost 5 years ago - 181 stars on GitHub
Top 4.8% on proxy.golang.org
38 versions - Latest release: about 1 month ago - 7 stars on GitHub
github.com/kinorai/omnifeed v0.29.0
LLM-friendly web crawler & scraper with a dedicated Reddit engine, built on Crawl4AI — Open WebUI...38 versions - Latest release: about 1 month ago - 7 stars on GitHub
Top 5.7% on proxy.golang.org
112 versions - Latest release: 8 days ago - 1,629 stars on GitHub
github.com/gildas-lormeau/single-file-cli v2.16.0+incompatible
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on Single...112 versions - Latest release: 8 days ago - 1,629 stars on GitHub
Top 7.2% on proxy.golang.org
Latest release: 7 days ago - 5 stars on GitHub
github.com/fuchsia-programming/scrapers
Scraper ReplicantsLatest release: 7 days ago - 5 stars on GitHub
Top 5.0% on proxy.golang.org
17 versions - Latest release: 27 days ago - 5 stars on GitHub
github.com/scrapfly/scrapfly-cli v0.4.0
Agentic CLI for the Scrapfly platform: scrape, extract, crawl, and drive a cloud browser over CDP...17 versions - Latest release: 27 days ago - 5 stars on GitHub
Top 5.2% on proxy.golang.org
19 versions - Latest release: 27 days ago - 14 stars on GitHub
github.com/scrapfly/scrapfly-mcp v1.3.4
Official Scrapfly MCP server for Cursor, Claude Desktop, and any MCP-compatible client. Enterpris...19 versions - Latest release: 27 days ago - 14 stars on GitHub
Top 5.2% on proxy.golang.org
11 versions - Latest release: 28 days ago - 5 stars on GitHub
github.com/scrapfly/go-scrapfly v1.6.0
Package scrapfly provides a Go SDK for the Scrapfly.io web scraping API.11 versions - Latest release: 28 days ago - 5 stars on GitHub
Top 4.9% on proxy.golang.org
29 versions - Latest release: 27 days ago - 23 stars on GitHub
github.com/scrapfly/typescript-scrapfly v0.12.0
Official TypeScript/JavaScript SDK for the Scrapfly platform: web scraping, screenshots, AI extra...29 versions - Latest release: 27 days ago - 23 stars on GitHub
Top 5.0% on proxy.golang.org
63 versions - Latest release: 21 days ago - 2,362 stars on GitHub
github.com/0xmassi/webclaw v0.6.23 💰
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all f...63 versions - Latest release: 21 days ago - 2,362 stars on GitHub
Top 5.0% on proxy.golang.org
63 versions - Latest release: 21 days ago - 2,362 stars on GitHub
github.com/0xMassi/webclaw v0.6.23 💰
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all f...63 versions - Latest release: 21 days ago - 2,362 stars on GitHub
Top 4.9% on proxy.golang.org
15 versions - Latest release: about 2 months ago - 38 stars on GitHub
github.com/geiserx/wayback-archive v1.4.6 💰
Download complete websites from the Wayback Machine with full asset preservation for offline viewing15 versions - Latest release: about 2 months ago - 38 stars on GitHub
Top 4.9% on proxy.golang.org
15 versions - Latest release: about 2 months ago - 38 stars on GitHub
github.com/GeiserX/Wayback-Archive v1.4.6 💰
Download complete websites from the Wayback Machine with full asset preservation for offline viewing15 versions - Latest release: about 2 months ago - 38 stars on GitHub
Top 7.4% on proxy.golang.org
Latest release: 3 months ago - 3 stars on GitHub
github.com/robcharlwood/spider-man
Your friendly neighbourhood web crawlerLatest release: 3 months ago - 3 stars on GitHub
Top 7.5% on proxy.golang.org
1 version - Latest release: about 7 years ago - 0 stars on GitHub
github.com/gellel/steamer v0.0.0-20190913012513-22ec79b1836c
Go application. Crawls the Steam store and collects Game records based on search criteria.1 version - Latest release: about 7 years ago - 0 stars on GitHub
Top 7.6% on proxy.golang.org
Latest release: 3 months ago - 0 stars on GitHub
github.com/oguzhand95/go-web-crawler-starter
Starter project for web crawler with controller per siteLatest release: 3 months ago - 0 stars on GitHub
Top 4.2% on proxy.golang.org
2 versions - Latest release: about 1 year ago - 134 stars on GitHub
github.com/pinkpixel-dev/web-scout-mcp v1.5.2
An MCP server providing web search and content extraction capabilities. Integrates DuckDuckGo sea...2 versions - Latest release: about 1 year ago - 134 stars on GitHub
Top 7.6% on proxy.golang.org
Latest release: almost 2 years ago - 0 stars on GitHub
github.com/theognis1002/go-rotten-tomatoes
rottentomatoes.com web crawler written in goLatest release: almost 2 years ago - 0 stars on GitHub
github.com/timtosi/mcrawler v0.0.0-20190807141622-9ce7f06ab0a8
[Go] - Web Crawler with composable pipeline.1 version - Latest release: about 7 years ago - 3 stars on GitHub
Top 4.8% on proxy.golang.org
4 versions - Latest release: over 7 years ago - 0 stars on GitHub
github.com/jpallari/lukki v0.1.2
Website tester4 versions - Latest release: over 7 years ago - 0 stars on GitHub
Top 7.8% on proxy.golang.org
Latest release: 3 months ago - 0 stars on GitHub
github.com/iamtonmoy0/crawler
web crawlerLatest release: 3 months ago - 0 stars on GitHub
github.com/teohrt/abrasion v0.0.0-20191001034743-c6bfbce89da2
Collect emails through web scraping1 version - Latest release: about 7 years ago - 0 stars on GitHub
Top 9.3% on proxy.golang.org
6 versions - Latest release: almost 4 years ago - 11 stars on GitHub
github.com/its-my-data/doubak v0.4.5
新版本请见 https://github.com/doubak | 豆备 (Doubak) - 豆瓣 (Douban.com) 命令行版备份工具，备份完整页面镜像，Golang实现。6 versions - Latest release: almost 4 years ago - 11 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 7 years ago - 0 stars on GitHub
github.com/iaziz786/go-web-crawler v0.0.0-20190206074040-a589353c8195
A web crawler in Golang1 version - Latest release: over 7 years ago - 0 stars on GitHub
Top 5.5% on proxy.golang.org
15 versions - Latest release: over 5 years ago - 97 stars on GitHub
github.com/creekorful/bathyscaphe v1.0.0
Fast, highly configurable, cloud native dark web crawler.15 versions - Latest release: over 5 years ago - 97 stars on GitHub
Top 4.4% on proxy.golang.org
62 versions - Latest release: over 2 years ago - 41 stars on GitHub
github.com/juroOravec/crawlee-one v2.0.3+incompatible 💰
Production-ready web scraping in a single function call. Built on Crawlee.62 versions - Latest release: over 2 years ago - 41 stars on GitHub
Top 8.6% on proxy.golang.org
Latest release: 12 days ago - 0 stars on GitHub
github.com/apkuznetsov/computer-networks-2019
Web Crawler collects and saves links from a given page, storing them in the memory for you to acc...Latest release: 12 days ago - 0 stars on GitHub
Top 9.0% on proxy.golang.org
1 version - Latest release: about 3 years ago - 0 stars on GitHub
github.com/apkuznetsov/go v0.0.0-20230806070220-9fb6084b3249
Web Crawler collects and saves links from a given page, storing them in the memory for you to acc...1 version - Latest release: about 3 years ago - 0 stars on GitHub
Top 9.9% on proxy.golang.org
1 version - Latest release: about 3 years ago - 1 stars on GitHub
github.com/drw3b/goshooter v0.0.0-20230731165456-a67922abd929
GoShooter is a command-line interface (CLI) tool, written in Go, designed to take screenshots of...1 version - Latest release: about 3 years ago - 1 stars on GitHub
Top 8.6% on proxy.golang.org
Latest release: 3 months ago - 0 stars on GitHub
github.com/R00tendo/GoldDigger
GolDigger web crawlerLatest release: 3 months ago - 0 stars on GitHub
Top 4.4% on proxy.golang.org
78 versions - Latest release: 19 days ago - 9,528 stars on GitHub
github.com/apify/crawlee-python v1.10.1
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extr...78 versions - Latest release: 19 days ago - 9,528 stars on GitHub
Top 5.6% on proxy.golang.org
Latest release: 16 days ago - 1 stars on GitHub
github.com/DrW3b/goshooter
GoShooter is a command-line interface (CLI) tool, written in Go, designed to take screenshots of...Latest release: 16 days ago - 1 stars on GitHub
Top 8.6% on proxy.golang.org
4 versions - Latest release: over 6 years ago - 0 stars on GitHub
github.com/oAGoulart/roedor v0.1.3 💰
A modular web crawler Go module.4 versions - Latest release: over 6 years ago - 0 stars on GitHub
github.com/cyclone-github/spider v1.0.0
Spider - web crawler and local wordlist processor to generate frequency sorted wordlist / ngrams7 versions - Latest release: 5 months ago - 28 stars on GitHub
Top 9.1% on proxy.golang.org
Latest release: 3 months ago - 0 stars on GitHub
github.com/mlmhl/gcrawler
Package gcrawler provides a very simple and flexible web crawler that supports three running modes:Latest release: 3 months ago - 0 stars on GitHub
Top 4.6% on proxy.golang.org
1 version - Latest release: about 1 month ago - 3 stars on GitHub
github.com/commonhuman-lab/chcrawl v0.0.0-20260903161429-b6d62fd30ab9
High-performance, correctness-first web crawler and reconnaissance engine written in Go.1 version - Latest release: about 1 month ago - 3 stars on GitHub
github.com/JarvisPrestidge/sitemap v0.0.0-20170326150601-a0b4d21ea3e1
Simple command line program to crawl a single domain implemented in Go .1 version - Latest release: over 9 years ago - 1 stars on GitHub
Top 6.7% on proxy.golang.org
16 versions - Latest release: over 1 year ago - 836 stars on GitHub
github.com/postmodern/spidr v0.7.2 💰
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or...16 versions - Latest release: over 1 year ago - 836 stars on GitHub
github.com/agrawalnaman/web-crawler-using-Golang v0.0.0-20200510005849-404a325006f4
web crawler which fetches URLs and outputs crawl results to console as the crawl proceeds1 version - Latest release: over 6 years ago - 0 stars on GitHub
Top 5.4% on proxy.golang.org
Latest release: about 2 months ago - 6 stars on GitHub
github.com/gustycube/spyder
Distributed DNS + HTTP crawler with queueing, deduplication, tracing, and ingest batching.Latest release: about 2 months ago - 6 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: about 8 years ago - 1 stars on GitHub
github.com/psatler/go-web-crawler v0.0.0-20181002024630-af05757a8be5
A simple web crawler written in Golang1 version - Latest release: about 8 years ago - 1 stars on GitHub
Top 8.2% on proxy.golang.org
5 versions - Latest release: over 6 years ago - 0 stars on GitHub
github.com/oagoulart/roedor v0.1.3 💰
A modular web crawler Go module.5 versions - Latest release: over 6 years ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 7 years ago - 0 stars on GitHub
github.com/iAziz786/go-web-crawler v0.0.0-20190206074040-a589353c8195
A web crawler in Golang1 version - Latest release: over 7 years ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 7 years ago - 6 stars on GitHub
github.com/integralist/go-web-crawler v0.0.0-20190301093913-65544983c478
A web crawler built in the Go programming language1 version - Latest release: over 7 years ago - 6 stars on GitHub
Top 7.8% on proxy.golang.org
1 version - Latest release: almost 4 years ago - 4 stars on GitHub
github.com/star-inc/butterfly v0.0.0-20221108064423-2ef8d7227940
The World-Wide-Web crawler for Apache Solr.1 version - Latest release: almost 4 years ago - 4 stars on GitHub
Top 5.7% on proxy.golang.org
1 version - Latest release: about 5 years ago - 79 stars on GitHub
github.com/redcode-labs/UnChain v0.0.0-20210721214645-e0bc12722569
A tool to find redirection chains in multiple URLs1 version - Latest release: about 5 years ago - 79 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 6 years ago - 0 stars on GitHub
github.com/agrawalnaman/web-crawler-using-golang v0.0.0-20200510005849-404a325006f4
web crawler which fetches URLs and outputs crawl results to console as the crawl proceeds1 version - Latest release: over 6 years ago - 0 stars on GitHub
Top 8.6% on proxy.golang.org
1 version - Latest release: about 7 years ago - 0 stars on GitHub
github.com/benmatselby/crawl v0.0.0-20190828205130-d35440badfad
🕷 A web crawler to build a sitemap1 version - Latest release: about 7 years ago - 0 stars on GitHub
Top 5.4% on proxy.golang.org
1 version - Latest release: over 4 years ago - 2 dependent repositories - 281 stars on GitHub
github.com/yields/ant v0.0.0-20220301182912-6c7a55c5708c
Package ant implements a web crawler.1 version - Latest release: over 4 years ago - 2 dependent repositories - 281 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 9 years ago - 0 stars on GitHub
github.com/panux/crawler-benchmark v0.0.0-20170608024305-5ce7344df2bc
A tool used to compare ARM and Intel server systems for General Purpose Computing (GPC)1 version - Latest release: over 9 years ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 9 years ago - 1 stars on GitHub
github.com/dineshsprabu/concurrent-web-crawler v0.0.0-20170505055956-d21717df7b11
Concurrent Web Crawler is a light weight configurable crawler with concurrecy enabled.1 version - Latest release: over 9 years ago - 1 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 9 years ago - 0 stars on GitHub
github.com/georgynet/crawler v0.0.0-20170224084818-026000c7ddb9
Web Crawler1 version - Latest release: over 9 years ago - 0 stars on GitHub
Related Keywords
golang
43
web-scraping
42
crawler
42
go
32
spider
23
scraper
19
mcp
18
scraping
16
mcp-server
14
llm
13
crawling
13
data-extraction
13
cli
13
webcrawler
13
web-scraper
12
docker
12
markdown
11
scrapy
11
python
11
webspider
10
platform
9
claude
9
crawlab
9
rag
9
crawling-tasks
9
spiders-management
9
ai-agents
9
scrapyd-ui
9
ai
9
rust
8
firecrawl-alternative
8
model-context-protocol
8
web-search
8
web-spider
8
ai-scraping
7
html-to-markdown
7
playwright
7
automation
7
typescript
7
webscraping
6
web-crawling
6
self-hosted
6
anti-detect
6
ai-agent
5
archiving
5
web-data
5
content-extraction
5
stealth-browser
5
nodejs
5
ruby
5
web-data-extraction
5
proxy
5
web-archiving
5
ai-crawler
4
developer-tools
4
ai-search
4
zeno
4
golang-application
4
headless-chrome
4
anti-bot
4
antidetect-browser
4
bot-detection
4
browser-automation
4
php
4
headless-browser
4
multithreading
4
web
4
scrapfly
4
apify
4
screenshot
4
cache-warmer
3
javascript-rendering
3
data-preparation
3
html
3
firecrawl-ai
3
firecrawl
3
data-collection
3
batch-processing
3
digital-preservation
3
elasticsearch
3
reconnaissance
3
docker-compose
3
documentation
3
headless
3
javascript
3
golang-cli
3
wayback-machine
3
link-checker
3
performance
3
saas
3
cache
3
wasm
3
website-backup
3
shopify
3
website-downloader
3
java
3
ffi
3
elixir
3
csharp
3
webflow
3