npmjs.org : @crawlify.io/robots
Parses robots.txt files to provide meaningful, useful output as well as reporting syntax errors.
Registry
-
Source
- Homepage
- JSON
purl: pkg:npm/%40crawlify.io/robots
Keywords:
robots.txt
, parse
, seo
, meta
, crawl
, scraper
, robots
License: MIT
Latest release: over 6 years ago
First release: over 6 years ago
Namespace: crawlify.io
Dependent packages: 1
Dependent repositories: 1
Downloads: 5 last month
Stars: 1 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 20 days ago
Loading...
Readme
Loading...