Ecosyste.ms: Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

crates.io : robotxt

The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.

Registry - Source - Homepage - Documentation - JSON
purl: pkg:cargo/robotxt
Keywords: scraper, crawler, framework, web, bytes, parser, robots-txt, sitemap-xml
License: MIT
Latest release: 3 months ago
First release: about 1 year ago
Downloads: 4,135 total
Stars: 2 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 3 days ago

Oleh Martsokha
8 packages
22,048 downloads