Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
packagist.org : crwlr/robots-txt
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
Registry
-
Source
- Homepage
- JSON
purl: pkg:composer/crwlr/robots-txt
Keywords: parser, scraper, spider, crawler, bot, robots.txt, robots, protocol, standard, scraping, bots, exclusion, crwlr, hacktoberfest, robots-exclusion-protocol, robots-exclusion-standard, robots-txt, robots-txt-parser, web-crawling, web-scraping
License: MIT
Latest release: over 1 year ago
First release: over 2 years ago
Namespace: crwlr
Dependent packages: 1
Dependent repositories: 1
Downloads: 6,045 total
Stars: 8 on GitHub
Forks: 2 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 6 days ago