packagist.org : crwlr/robots-txt
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
Registry
-
Source
- Homepage
- JSON
purl: pkg:composer/crwlr/robots-txt
Keywords:
parser
, scraper
, spider
, crawler
, bot
, robots.txt
, robots
, protocol
, standard
, scraping
, bots
, exclusion
, crwlr
, hacktoberfest
, robots-exclusion-protocol
, robots-exclusion-standard
, robots-txt
, robots-txt-parser
, web-crawling
, web-scraping
License: MIT
Latest release: 6 months ago
First release: over 3 years ago
Namespace: crwlr
Dependent packages: 1
Dependent repositories: 1
Downloads: 16,405 total
Stars: 8 on GitHub
Forks: 2 on GitHub
See more repository details: repos.ecosyste.ms
Funding links: https://github.com/sponsors/otsch
Last synced: 12 days ago