proxy.golang.org "robots-txt" keyword
View the packages on the proxy.golang.org package registry that are tagged with the "robots-txt" keyword.
Top 2.0% on proxy.golang.org
8 versions - Latest release: over 4 years ago - 4 dependent packages - 7 dependent repositories - 2,009 stars on GitHub
github.com/PuerkitoBio/gocrawl v1.1.0 💰
Package gocrawl is a polite, slim and concurrent web crawler written in Go.8 versions - Latest release: over 4 years ago - 4 dependent packages - 7 dependent repositories - 2,009 stars on GitHub
Top 1.5% on proxy.golang.org
3 versions - Latest release: over 4 years ago - 1,275 dependent packages - 1,655 dependent repositories - 276 stars on GitHub
github.com/temoto/robotstxt v1.1.2
Package robotstxt implements the robots.txt Exclusion Protocol as specified in http://www.robotst...3 versions - Latest release: over 4 years ago - 1,275 dependent packages - 1,655 dependent repositories - 276 stars on GitHub
Top 8.2% on proxy.golang.org
6 versions - Latest release: over 4 years ago - 775 stars on GitHub
github.com/puerkitobio/fetchbot v1.4.0 💰
Package fetchbot provides a simple and flexible web crawler that follows the robots.txt policies ...6 versions - Latest release: over 4 years ago - 775 stars on GitHub
Top 2.3% on proxy.golang.org
6 versions - Latest release: over 4 years ago - 4 dependent packages - 12 dependent repositories - 775 stars on GitHub
github.com/PuerkitoBio/fetchbot v1.4.0 💰
Package fetchbot provides a simple and flexible web crawler that follows the robots.txt policies ...6 versions - Latest release: over 4 years ago - 4 dependent packages - 12 dependent repositories - 775 stars on GitHub
Top 8.2% on proxy.golang.org
8 versions - Latest release: over 4 years ago - 2,009 stars on GitHub
github.com/puerkitobio/gocrawl v1.1.0 💰
Package gocrawl is a polite, slim and concurrent web crawler written in Go.8 versions - Latest release: over 4 years ago - 2,009 stars on GitHub
Top 5.6% on proxy.golang.org
11 versions - Latest release: 6 months ago - 3 stars on GitHub
github.com/middlewares/robots v2.1.0+incompatible
PSR-15 middleware to enable/disable the robots of the search engines11 versions - Latest release: 6 months ago - 3 stars on GitHub
Top 9.2% on proxy.golang.org
11 versions - Latest release: 3 months ago - 0 stars on GitHub
github.com/aafeher/go-sitemap-parser v0.2.0
Go language library for parsing Sitemaps11 versions - Latest release: 3 months ago - 0 stars on GitHub
Top 5.8% on proxy.golang.org
7 versions - Latest release: 4 months ago - 6 stars on GitHub
github.com/holysoles/bot-wrangler-traefik-plugin v0.6.0
Package bot_wrangler_traefik_plugin a plugin for managing bot traffic with automatically updating...7 versions - Latest release: 4 months ago - 6 stars on GitHub
Top 5.9% on proxy.golang.org
7 versions - Latest release: over 3 years ago - 1 dependent package - 1 dependent repositories - 89 stars on GitHub
github.com/jimsmart/grobotstxt v1.0.3
Package grobotstxt is a Go port of Google's robots.txt parser and matcher C++ library. See: http...7 versions - Latest release: over 3 years ago - 1 dependent package - 1 dependent repositories - 89 stars on GitHub
Top 6.1% on proxy.golang.org
6 versions - Latest release: almost 6 years ago - 4 dependent packages - 7 stars on GitHub
github.com/benjaminestes/robots/v2 v2.0.5
Package robots implements robots.txt parsing and matching based on Google's specification. For a ...6 versions - Latest release: almost 6 years ago - 4 dependent packages - 7 stars on GitHub
Top 9.0% on proxy.golang.org
6 versions - Latest release: almost 7 years ago - 1 dependent package - 1 dependent repositories - 7 stars on GitHub
github.com/benjaminestes/robots v2.0.4+incompatible
Package robots implements robots.txt parsing and matching based on Google's specification. For a ...6 versions - Latest release: almost 7 years ago - 1 dependent package - 1 dependent repositories - 7 stars on GitHub
Top 6.7% on proxy.golang.org
1 version - Latest release: almost 4 years ago - 64 stars on GitHub
github.com/liameno/librengine v0.1.0-alpha
Privacy Web Search Engine (not meta, own crawler)1 version - Latest release: almost 4 years ago - 64 stars on GitHub
Top 3.7% on proxy.golang.org
3 versions - Latest release: over 4 years ago - 1 dependent package - 6 dependent repositories - 246 stars on GitHub
github.com/temoto/robotstxt.go v1.1.2
Package robotstxt implements the robots.txt Exclusion Protocol as specified in http://www.robotst...3 versions - Latest release: over 4 years ago - 1 dependent package - 6 dependent repositories - 246 stars on GitHub
Top 4.4% on proxy.golang.org
3 versions - Latest release: over 4 years ago - 18 dependent repositories - 246 stars on GitHub
github.com/temoto/robotstxt-go v1.1.2
Package robotstxt implements the robots.txt Exclusion Protocol as specified in http://www.robotst...3 versions - Latest release: over 4 years ago - 18 dependent repositories - 246 stars on GitHub
Top 8.2% on proxy.golang.org
1 version - Latest release: about 3 years ago - 0 stars on GitHub
github.com/hvlck/robots v0.0.0-20220808231330-f74d5a434efc
robots.txt parser1 version - Latest release: about 3 years ago - 0 stars on GitHub
Top 6.6% on proxy.golang.org
1 version - Latest release: almost 8 years ago - 1 dependent package - 1 dependent repositories - 16 stars on GitHub
github.com/samclarke/robotstxt v0.0.0-20171127213916-2817654b7988
Package robotstxt parses robots.txt files Aims to follow the Google robots.txt specification, se...1 version - Latest release: almost 8 years ago - 1 dependent package - 1 dependent repositories - 16 stars on GitHub
Related Keywords
go
7
crawler
5
golang
5
go-library
3
golang-library
3
production-ready
3
status-active
3
web
3
robotstxt
2
robots-exclusion-standard
1
noctis
1
websearchengine
1
robots-parser
1
websearch
1
spider
1
self-hosted
1
search-engine
1
rsa
1
privacy
1
frontend
1
encryption
1
cpp
1
robots-exclusion-protocol
1
traefik-plugin
1
traefik
1
plugin
1
llm
1
bot
1
ai
1
sitemapxml
1
sitemaps
1
sitemap-xml-gz
1
sitemap-xml
1
sitemap-parser
1
sitemap
1
seo
1
psr-15
1
middleware
1
http
1