Data

Tools

Indexes

Applications

Experiments

Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

hex.pm "robots-parser" keyword

gollum 0.3.3
Robots.txt parser with caching. Modelled after Kryten.
6 versions - Latest release: over 6 years ago - 5 dependent repositories - 4.99 thousand downloads total - 14 stars on GitHub - 1 maintainer
robots 1.1.2
A parser for robots.txt.
4 versions - Latest release: about 2 years ago - 1 dependent package - 22.8 thousand downloads total - 3 stars on GitHub - 1 maintainer
Related Keywords
robots-txt 2 crawler 1 elixir 1 crawling 1 erlang 1 erlang-library 1 parser 1 parsing 1 parsing-library 1 rfc-9309 1 robots-exclusion-standard 1