Ecosyste.ms: Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

npmjs.org "robots-exclusion-standard" keyword

robots-txt-component 0.0.4
Lightweight robots.txt parsing component without any external dependencies for Node.js.
3 versions - Latest release: over 1 year ago - 23 downloads last month - 0 stars on GitHub - 1 maintainer
Top 1.9% on npmjs.org
robots-parser 3.0.1
A specification compliant robots.txt parser with wildcard (*) matching support.
11 versions - Latest release: over 1 year ago - 45 dependent packages - 5,176 dependent repositories - 3.38 million downloads last month - 137 stars on GitHub - 1 maintainer
Top 6.4% on npmjs.org
robots-txt-parser 2.0.3
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
17 versions - Latest release: about 2 years ago - 2 dependent packages - 9 dependent repositories - 2.09 thousand downloads last month - 11 stars on GitHub - 1 maintainer
Related Keywords
robots.txt 3 robots-parser 3 javascript 3 robots-txt 3 robots 2 node-js 2 crawler 2 nodejs 2 bot 2 parser 2 spider 2 scraper 2 robots-txt-parser 1 robotstxt 1 txt 1 user-agent 1 robots-txt-node 1 robots-node 1 crawlers 1 js 1