proxy.golang.org "web-crawler" keyword
View the packages on the proxy.golang.org package registry that are tagged with the "web-crawler" keyword.
Top 7.8% on proxy.golang.org
Latest release: 17 minutes ago - 0 stars on GitHub
github.com/iamtonmoy0/crawler
web crawlerLatest release: 17 minutes ago - 0 stars on GitHub
Top 5.6% on proxy.golang.org
429 versions - Latest release: about 2 months ago - 21,328 stars on GitHub
github.com/scrapegraphai/scrapegraph-ai v1.62.0 💰
Python scraper based on AI429 versions - Latest release: about 2 months ago - 21,328 stars on GitHub
Top 5.6% on proxy.golang.org
429 versions - Latest release: about 2 months ago - 21,328 stars on GitHub
github.com/ScrapeGraphAI/Scrapegraph-ai v1.62.0 💰
Python scraper based on AI429 versions - Latest release: about 2 months ago - 21,328 stars on GitHub
github.com/cyclone-github/spider v0.9.0
Spider - web crawler and local wordlist processor to generate frequency sorted wordlist / ngrams5 versions - Latest release: 5 months ago - 24 stars on GitHub
Top 6.5% on proxy.golang.org
4 versions - Latest release: about 7 years ago - 310 stars on GitHub
github.com/infinilabs/crawler v0.11.0 💰
🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)4 versions - Latest release: about 7 years ago - 310 stars on GitHub
Top 8.6% on proxy.golang.org
4 versions - Latest release: over 5 years ago - 1 stars on GitHub
github.com/oAGoulart/roedor v0.1.3 💰
A modular web crawler Go module.4 versions - Latest release: over 5 years ago - 1 stars on GitHub
Top 8.2% on proxy.golang.org
4 versions - Latest release: over 6 years ago - 0 stars on GitHub
github.com/Lepovirta/lukki v0.1.2
Website tester4 versions - Latest release: over 6 years ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
11 versions - Latest release: over 5 years ago - 7,729 stars on GitHub
github.com/ssssssss-team/spider-flow v0.5.0
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。11 versions - Latest release: over 5 years ago - 7,729 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 6 years ago - 0 stars on GitHub
github.com/iaziz786/go-web-crawler v0.0.0-20190206074040-a589353c8195
A web crawler in Golang1 version - Latest release: over 6 years ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
33 versions - Latest release: about 2 years ago - 11,928 stars on GitHub
github.com/crawlab-team/crawlab v0.6.3
Distributed web crawler admin platform for spiders management regardless of languages and framewo...33 versions - Latest release: about 2 years ago - 11,928 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 29 days ago - 11,928 stars on GitHub
github.com/crawlab-team/crawlab/template-parser
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 29 days ago - 11,928 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 2 months ago - 11,928 stars on GitHub
github.com/crawlab-team/crawlab/core
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 2 months ago - 11,928 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 17 days ago - 11,928 stars on GitHub
github.com/crawlab-team/crawlab/fs
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 17 days ago - 11,928 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 9 days ago - 11,928 stars on GitHub
github.com/crawlab-team/crawlab/db
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 9 days ago - 11,928 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 2 months ago - 11,928 stars on GitHub
github.com/crawlab-team/crawlab/trace
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 2 months ago - 11,928 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 2 months ago - 11,928 stars on GitHub
github.com/crawlab-team/crawlab/vcs
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 2 months ago - 11,928 stars on GitHub
Top 6.7% on proxy.golang.org
Latest release: 5 days ago - 11,928 stars on GitHub
github.com/crawlab-team/crawlab/grpc
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: 5 days ago - 11,928 stars on GitHub
Top 6.2% on proxy.golang.org
Latest release: about 2 months ago - 11,928 stars on GitHub
github.com/crawlab-team/crawlab/cli
Distributed web crawler admin platform for spiders management regardless of languages and framewo...Latest release: about 2 months ago - 11,928 stars on GitHub
Top 6.7% on proxy.golang.org
641 versions - Latest release: 6 days ago - 11,973 stars on GitHub
github.com/apifytech/apify-js v3.15.1+incompatible
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In ...641 versions - Latest release: 6 days ago - 11,973 stars on GitHub
Top 5.0% on proxy.golang.org
4 versions - Latest release: about 7 years ago - 294 stars on GitHub
github.com/infinitbyte/gopa v0.11.0 💰
GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn4 versions - Latest release: about 7 years ago - 294 stars on GitHub
github.com/internetarchive/Zeno v1.0.85
Authors:29 versions - Latest release: 7 months ago - 332 stars on GitHub
Top 9.9% on proxy.golang.org
29 versions - Latest release: 7 months ago - 328 stars on GitHub
github.com/internetarchive/zeno v1.0.85
State-of-the-art web crawler 🔱29 versions - Latest release: 7 months ago - 328 stars on GitHub
Top 4.8% on proxy.golang.org
624 versions - Latest release: 11 months ago - 19,547 stars on GitHub
github.com/apify/crawlee v3.12.0+incompatible
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In ...624 versions - Latest release: 11 months ago - 19,547 stars on GitHub
Top 7.5% on proxy.golang.org
Latest release: 7 days ago - 0 stars on GitHub
github.com/gellel/steamer
Go application. Crawls the Steam store and collects Game records based on search criteria.Latest release: 7 days ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
4 versions - Latest release: over 5 years ago - 1 stars on GitHub
github.com/oagoulart/roedor v0.1.3 💰
A modular web crawler Go module.4 versions - Latest release: over 5 years ago - 1 stars on GitHub
Top 9.3% on proxy.golang.org
6 versions - Latest release: almost 3 years ago - 9 stars on GitHub
github.com/its-my-data/doubak v0.4.5
豆备 (Doubak) - 豆瓣 (Douban.com) 命令行版备份工具，备份完整页面镜像，Golang实现。6 versions - Latest release: almost 3 years ago - 9 stars on GitHub
Top 5.8% on proxy.golang.org
15 versions - Latest release: 15 days ago - 2,210 stars on GitHub
github.com/vrknetha/mcp-server-firecrawl v3.2.0+incompatible
Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM cl...15 versions - Latest release: 15 days ago - 2,210 stars on GitHub
Top 8.8% on proxy.golang.org
53 versions - Latest release: about 1 year ago - 1 dependent repositories - 24 stars on GitHub
github.com/tech-engine/goscrapy v0.16.0
GoScrapy: Harnessing Go's power for efficient web scraping, inspired by Python's Scrapy framework.53 versions - Latest release: about 1 year ago - 1 dependent repositories - 24 stars on GitHub
Top 4.6% on proxy.golang.org
32 versions - Latest release: about 1 month ago - 1 dependent package - 2 dependent repositories - 138 stars on GitHub
github.com/gosom/scrapemate v0.9.6 💰
Golang Crawling and scraping framework32 versions - Latest release: about 1 month ago - 1 dependent package - 2 dependent repositories - 138 stars on GitHub
Top 5.8% on proxy.golang.org
15 versions - Latest release: 15 days ago - 4,190 stars on GitHub
github.com/mendableai/firecrawl-mcp-server v3.2.0+incompatible
🔥 Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM ...15 versions - Latest release: 15 days ago - 4,190 stars on GitHub
Top 6.7% on proxy.golang.org
90 versions - Latest release: 19 days ago - 876 stars on GitHub
github.com/webrecorder/browsertrix-crawler v1.7.1 💰
Run a high-fidelity browser-based web archiving crawler in a single Docker container90 versions - Latest release: 19 days ago - 876 stars on GitHub
Top 5.5% on proxy.golang.org
15 versions - Latest release: over 4 years ago - 81 stars on GitHub
github.com/creekorful/bathyscaphe v1.0.0
Fast, highly configurable, cloud native dark web crawler.15 versions - Latest release: over 4 years ago - 81 stars on GitHub
Top 6.5% on proxy.golang.org
24 versions - Latest release: 20 days ago - 48,469 stars on GitHub
github.com/mendableai/firecrawl v2.2.0+incompatible
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with...24 versions - Latest release: 20 days ago - 48,469 stars on GitHub
Top 9.1% on proxy.golang.org
Latest release: 19 days ago - 0 stars on GitHub
github.com/mlmhl/gcrawler
Package gcrawler provides a very simple and flexible web crawler that supports three running modes:Latest release: 19 days ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
2 versions - Latest release: over 6 years ago - 0 stars on GitHub
github.com/rapidclock/web-octopus v1.2.2
A concurent web crawler library written in Go.2 versions - Latest release: over 6 years ago - 0 stars on GitHub
Top 6.4% on proxy.golang.org
Latest release: 21 days ago - 0 stars on GitHub
github.com/psidex/nomad
An experimental web crawler to visualise & map the connections between domainsLatest release: 21 days ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
70 versions - Latest release: about 2 months ago - 296 stars on GitHub
github.com/s0rg/crawley v1.7.13
The unix-way web crawler70 versions - Latest release: about 2 months ago - 296 stars on GitHub
Top 5.7% on proxy.golang.org
1 version - Latest release: about 4 years ago - 79 stars on GitHub
github.com/redcode-labs/UnChain v0.0.0-20210721214645-e0bc12722569
A tool to find redirection chains in multiple URLs1 version - Latest release: about 4 years ago - 79 stars on GitHub
Top 8.6% on proxy.golang.org
Latest release: 25 days ago - 0 stars on GitHub
github.com/R00tendo/GoldDigger
GolDigger web crawlerLatest release: 25 days ago - 0 stars on GitHub
Top 7.4% on proxy.golang.org
Latest release: 26 days ago - 2 stars on GitHub
github.com/robcharlwood/spider-man
Your friendly neighbourhood web crawlerLatest release: 26 days ago - 2 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 8 years ago - 0 stars on GitHub
github.com/panux/crawler-benchmark v0.0.0-20170608024305-5ce7344df2bc
A tool used to compare ARM and Intel server systems for General Purpose Computing (GPC)1 version - Latest release: over 8 years ago - 0 stars on GitHub
Top 5.4% on proxy.golang.org
Latest release: 27 days ago - 6 stars on GitHub
github.com/gustycube/spyder
Distributed DNS + HTTP crawler with queueing, deduplication, tracing, and ingest batching.Latest release: 27 days ago - 6 stars on GitHub
Top 9.0% on proxy.golang.org
Latest release: 28 days ago - 0 stars on GitHub
github.com/apkuznetsov/go
Web Crawler collects and saves links from a given page, storing them in the memory for you to acc...Latest release: 28 days ago - 0 stars on GitHub
Top 5.7% on proxy.golang.org
63 versions - Latest release: 7 months ago - 805 stars on GitHub
github.com/gildas-lormeau/single-file-cli v2.0.75+incompatible
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on Single...63 versions - Latest release: 7 months ago - 805 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: about 7 years ago - 1 stars on GitHub
github.com/psatler/go-web-crawler v0.0.0-20181002024630-af05757a8be5
A simple web crawler written in Golang1 version - Latest release: about 7 years ago - 1 stars on GitHub
Top 6.7% on proxy.golang.org
16 versions - Latest release: 8 months ago - 825 stars on GitHub
github.com/postmodern/spidr v0.7.2 💰
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or...16 versions - Latest release: 8 months ago - 825 stars on GitHub
Top 8.2% on proxy.golang.org
8 versions - Latest release: over 5 years ago - 33 stars on GitHub
github.com/nicksherron/proxi v0.1.7
Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxie...8 versions - Latest release: over 5 years ago - 33 stars on GitHub
Top 7.8% on proxy.golang.org
1 version - Latest release: almost 3 years ago - 2 stars on GitHub
github.com/star-inc/butterfly v0.0.0-20221108064423-2ef8d7227940
The World-Wide-Web crawler for Apache Solr. (Developing)1 version - Latest release: almost 3 years ago - 2 stars on GitHub
Top 5.6% on proxy.golang.org
Latest release: about 1 month ago - 1 stars on GitHub
github.com/DrW3b/goshooter
GoShooter is a command-line interface (CLI) tool, written in Go, designed to take screenshots of...Latest release: about 1 month ago - 1 stars on GitHub
github.com/teohrt/abrasion v0.0.0-20191001034743-c6bfbce89da2
Collect emails through web scraping1 version - Latest release: about 6 years ago - 0 stars on GitHub
Top 7.2% on proxy.golang.org
Latest release: about 2 months ago - 5 stars on GitHub
github.com/fuchsia-programming/scrapers
Scraper ReplicantsLatest release: about 2 months ago - 5 stars on GitHub
Top 5.7% on proxy.golang.org
Latest release: about 2 months ago - 0 stars on GitHub
github.com/iAziz786/wego
A web crawler to keep you updated about things you want to keep track ofLatest release: about 2 months ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 8 years ago - 0 stars on GitHub
github.com/dineshsprabu/concurrent-web-crawler v0.0.0-20170505055956-d21717df7b11
Concurrent Web Crawler is a light weight configurable crawler with concurrecy enabled. Configrati...1 version - Latest release: over 8 years ago - 0 stars on GitHub
github.com/agrawalnaman/web-crawler-using-Golang v0.0.0-20200510005849-404a325006f4
web crawler which fetches URLs and outputs crawl results to console as the crawl proceeds1 version - Latest release: over 5 years ago - 0 stars on GitHub
Top 7.6% on proxy.golang.org
Latest release: 2 months ago - 0 stars on GitHub
github.com/oguzhand95/go-web-crawler-starter
Starter project for web crawler with controller per siteLatest release: 2 months ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 5 years ago - 0 stars on GitHub
github.com/agrawalnaman/web-crawler-using-golang v0.0.0-20200510005849-404a325006f4
web crawler which fetches URLs and outputs crawl results to console as the crawl proceeds1 version - Latest release: over 5 years ago - 0 stars on GitHub
github.com/JarvisPrestidge/sitemap v0.0.0-20170326150601-a0b4d21ea3e1
Simple command line program to crawl a single domain implemented in Go .1 version - Latest release: over 8 years ago - 1 stars on GitHub
Top 9.9% on proxy.golang.org
1 version - Latest release: about 2 years ago - 0 stars on GitHub
github.com/drw3b/goshooter v0.0.0-20230731165456-a67922abd929
GoShooter is a command-line interface (CLI) tool, written in Go, designed to take screenshots of...1 version - Latest release: about 2 years ago - 0 stars on GitHub
Top 8.6% on proxy.golang.org
1 version - Latest release: about 6 years ago - 0 stars on GitHub
github.com/benmatselby/crawl v0.0.0-20190828205130-d35440badfad
🕷 A web crawler to build a sitemap1 version - Latest release: about 6 years ago - 0 stars on GitHub
Top 5.9% on proxy.golang.org
Latest release: 3 months ago - 2 stars on GitHub
github.com/heinrichb/scrapey-cli
Scrapey CLI is a lightweight, modular command-line tool built in Go for web crawling and scraping...Latest release: 3 months ago - 2 stars on GitHub
Top 5.4% on proxy.golang.org
1 version - Latest release: over 3 years ago - 2 dependent repositories - 270 stars on GitHub
github.com/yields/ant v0.0.0-20220301182912-6c7a55c5708c
Package ant implements a web crawler.1 version - Latest release: over 3 years ago - 2 dependent repositories - 270 stars on GitHub
Top 6.0% on proxy.golang.org
Latest release: 4 months ago - 0 stars on GitHub
github.com/Huy-Ngo/cov-news
A web crawler that takes content from https://ncov.moh.gov.vn/dong-thoi-gianLatest release: 4 months ago - 0 stars on GitHub
Top 5.9% on proxy.golang.org
Latest release: 4 months ago - 0 stars on GitHub
github.com/mmycin/npminfo
A CLI tool to check the downloads of an NPM package written in GoLatest release: 4 months ago - 0 stars on GitHub
Top 8.6% on proxy.golang.org
Latest release: 4 months ago - 0 stars on GitHub
github.com/apkuznetsov/computer-networks-2019
Web Crawler collects and saves links from a given page, storing them in the memory for you to acc...Latest release: 4 months ago - 0 stars on GitHub
Top 3.5% on proxy.golang.org
1 version - Latest release: over 5 years ago - 2 dependent packages - 3 dependent repositories - 248 stars on GitHub
github.com/antchfx/antch v0.0.0-20200531151208-b3a15a669131
Antch, a fast, powerful and extensible web crawling & scraping framework for Go1 version - Latest release: over 5 years ago - 2 dependent packages - 3 dependent repositories - 248 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 6 years ago - 2 stars on GitHub
github.com/integralist/go-web-crawler v0.0.0-20190301093913-65544983c478
A web crawler built in the Go programming language1 version - Latest release: over 6 years ago - 2 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 6 years ago - 0 stars on GitHub
github.com/iAziz786/go-web-crawler v0.0.0-20190206074040-a589353c8195
A web crawler in Golang1 version - Latest release: over 6 years ago - 0 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: over 8 years ago - 0 stars on GitHub
github.com/georgynet/crawler v0.0.0-20170224084818-026000c7ddb9
Web Crawler1 version - Latest release: over 8 years ago - 0 stars on GitHub
Top 5.3% on proxy.golang.org
1 version - Latest release: almost 4 years ago - 149 stars on GitHub
github.com/saeeddhqan/evine v0.0.0-20211015222239-46051ded7145
Interactive CLI Web Crawler1 version - Latest release: almost 4 years ago - 149 stars on GitHub
Top 7.6% on proxy.golang.org
Latest release: 10 months ago - 0 stars on GitHub
github.com/theognis1002/go-rotten-tomatoes
rottentomatoes.com web crawler written in goLatest release: 10 months ago - 0 stars on GitHub
Related Keywords
golang
34
crawler
32
go
26
spider
21
web-scraping
14
webcrawler
13
scrapy
11
docker
10
scraping
10
webspider
10
crawling-tasks
9
crawlab
9
platform
9
scrapyd-ui
9
spiders-management
9
scraper
8
crawling
8
web-spider
8
cli
7
multithreading
4
elasticsearch
3
lightweight
3
web-scraper
3
golang-application
3
nodejs
3
data-mining
3
rag
3
markdown
3
llm
3
html-to-markdown
3
ai-scraping
3
ai
3
proxy
3
archiving
3
data-collection
2
http
2
content-extraction
2
claude
2
batch-processing
2
linux
2
zeno
2
web-crawling
2
ftp
2
smtp
2
tcp
2
tlv
2
udp
2
ruby
2
concurrent-web-crawler
2
concurrency
2
architecture
2
web-archiving
2
screenshot-tool
2
website-screenshot
2
data-extraction
2
search-api
2
model-context-protocol
2
mcp-server
2
llm-tools
2
javascript-rendering
2
firecrawl-ai
2
firecrawl
2
url
2
web
2
markout
2
package
2
xpath
2
apify
2
automation
2
headless
2
web-crawlers
2
scraping-python
2
typescript
2
puppeteer
2
playwright
2
npm
2
javascript
2
headless-chrome
2
automated-scraper
2
cicd
1
covid
1
solr
1
crwaler
1
butterfly
1
proxypool
1
proxy-list
1
news
1
http-proxy
1
beautifulsoup
1
spider-links
1
framework
1
deno
1
cobra
1
mysql
1
gohttp
1
osint
1
dockerfile
1
fuzzing
1
docker-compose
1
npminfo
1