Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
proxy.golang.org "robots-txt" keyword
Top 1.5% on proxy.golang.org
3 versions - Latest release: about 3 years ago - 1,275 dependent packages - 1,655 dependent repositories - 246 stars on GitHub
github.com/temoto/robotstxt v1.1.2
Package robotstxt implements the robots.txt Exclusion Protocol as specified in http://www.robotst...3 versions - Latest release: about 3 years ago - 1,275 dependent packages - 1,655 dependent repositories - 246 stars on GitHub
Top 2.0% on proxy.golang.org
8 versions - Latest release: about 3 years ago - 4 dependent packages - 7 dependent repositories - 2,009 stars on GitHub
github.com/PuerkitoBio/gocrawl v1.1.0 💰
Package gocrawl is a polite, slim and concurrent web crawler written in Go.8 versions - Latest release: about 3 years ago - 4 dependent packages - 7 dependent repositories - 2,009 stars on GitHub
Top 5.9% on proxy.golang.org
7 versions - Latest release: about 2 years ago - 1 dependent package - 1 dependent repositories - 89 stars on GitHub
github.com/jimsmart/grobotstxt v1.0.3
Package grobotstxt is a Go port of Google's robots.txt parser and matcher C++ library. See: http...7 versions - Latest release: about 2 years ago - 1 dependent package - 1 dependent repositories - 89 stars on GitHub
Top 3.7% on proxy.golang.org
3 versions - Latest release: about 3 years ago - 1 dependent package - 6 dependent repositories - 246 stars on GitHub
github.com/temoto/robotstxt.go v1.1.2
Package robotstxt implements the robots.txt Exclusion Protocol as specified in http://www.robotst...3 versions - Latest release: about 3 years ago - 1 dependent package - 6 dependent repositories - 246 stars on GitHub
Top 8.2% on proxy.golang.org
8 versions - Latest release: about 3 years ago - 2,009 stars on GitHub
github.com/puerkitobio/gocrawl v1.1.0 💰
Package gocrawl is a polite, slim and concurrent web crawler written in Go.8 versions - Latest release: about 3 years ago - 2,009 stars on GitHub
Top 4.4% on proxy.golang.org
3 versions - Latest release: about 3 years ago - 18 dependent repositories - 246 stars on GitHub
github.com/temoto/robotstxt-go v1.1.2
Package robotstxt implements the robots.txt Exclusion Protocol as specified in http://www.robotst...3 versions - Latest release: about 3 years ago - 18 dependent repositories - 246 stars on GitHub
Top 8.2% on proxy.golang.org
6 versions - Latest release: about 3 years ago - 775 stars on GitHub
github.com/puerkitobio/fetchbot v1.4.0 💰
Package fetchbot provides a simple and flexible web crawler that follows the robots.txt policies ...6 versions - Latest release: about 3 years ago - 775 stars on GitHub
Top 2.3% on proxy.golang.org
6 versions - Latest release: about 3 years ago - 4 dependent packages - 12 dependent repositories - 775 stars on GitHub
github.com/PuerkitoBio/fetchbot v1.4.0 💰
Package fetchbot provides a simple and flexible web crawler that follows the robots.txt policies ...6 versions - Latest release: about 3 years ago - 4 dependent packages - 12 dependent repositories - 775 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: almost 2 years ago - 0 stars on GitHub
github.com/hvlck/robots v0.0.0-20220808231330-f74d5a434efc
robots.txt parser1 version - Latest release: almost 2 years ago - 0 stars on GitHub
Top 9.2% on proxy.golang.org
Latest release: 12 days ago - 0 stars on GitHub
github.com/aafeher/go-sitemap-parser
Go language library for parsing SitemapsLatest release: 12 days ago - 0 stars on GitHub
Top 6.6% on proxy.golang.org
1 version - Latest release: over 6 years ago - 1 dependent package - 1 dependent repositories - 16 stars on GitHub
github.com/samclarke/robotstxt v0.0.0-20171127213916-2817654b7988
Package robotstxt parses robots.txt files Aims to follow the Google robots.txt specification, se...1 version - Latest release: over 6 years ago - 1 dependent package - 1 dependent repositories - 16 stars on GitHub
Top 9.0% on proxy.golang.org
6 versions - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 3 stars on GitHub
github.com/benjaminestes/robots v2.0.4+incompatible
Package robots implements robots.txt parsing and matching based on Google's specification. For a ...6 versions - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 3 stars on GitHub
Top 6.1% on proxy.golang.org
6 versions - Latest release: over 4 years ago - 4 dependent packages - 3 stars on GitHub
github.com/benjaminestes/robots/v2 v2.0.5
Package robots implements robots.txt parsing and matching based on Google's specification. For a ...6 versions - Latest release: over 4 years ago - 4 dependent packages - 3 stars on GitHub