Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
crates.io : robotxt
The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.
Registry
-
Source
- Homepage
- Documentation
- JSON
purl: pkg:cargo/robotxt
Keywords: scraper, crawler, framework, web, bytes, parser, robots-txt, sitemap-xml
License: MIT
Latest release: 3 months ago
First release: about 1 year ago
Downloads: 4,135 total
Stars: 2 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 4 days ago