Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
crates.io : robotxt
The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.
Registry
-
Source
- Homepage
- Documentation
- JSON
purl: pkg:cargo/robotxt
Keywords: web, framework, crawler, scraper, bytes, parser, robots-txt, sitemap-xml
License: MIT
Latest release: 2 months ago
First release: about 1 year ago
Downloads: 3,624 total
Stars: 2 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 12 days ago