Top 8.4% downloads on pypi.org
pypi.org : robotsparser
Parse robots.txt files and find indexed urls
Registry
-
Source
- Documentation
- JSON
purl: pkg:pypi/robotsparser
Keywords:
robots.txt
, sitemap
, crawler
, data mining
License: MIT
Latest release: about 2 years ago
First release: over 2 years ago
Downloads: 605 last month
Stars: 0 on GitHub
Forks: 1 on GitHub
Total Commits: 20
Committers: 2
Average commits per author: 10.0
Development Distribution Score (DDS): 0.05
More commit stats: commits.ecosyste.ms
See more repository details: repos.ecosyste.ms
Last synced: 17 days ago
No dependent packages found.