Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
npmjs.org "crawlers" keyword
@yeskiy/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers2 versions - 31 downloads last month - 0 stars on GitHub - 1 maintainer
@xianyukeji/isbot 2.3.0
detects bots/crawlers/spiders via the user agent.1 version - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 7 downloads last month - 841 stars on GitHub - 2 maintainers
Top 1.2% on npmjs.org
1,059 versions - Latest release: 22 days ago - 14 dependent packages - 88 dependent repositories - 122 thousand downloads last month - 10,899 stars on GitHub - 10 maintainers
@crawlee/utils 3.10.0
A set of shared utilities that can be used by crawlers1,059 versions - Latest release: 22 days ago - 14 dependent packages - 88 dependent repositories - 122 thousand downloads last month - 10,899 stars on GitHub - 10 maintainers
@wmod/sitemapper 3.2.11
Parser for XML Sitemaps to be used with Robots.txt and web crawlers2 versions - Latest release: 4 months ago - 11 downloads last month - 0 stars on GitHub - 1 maintainer
Top 1.2% on npmjs.org
126 versions - Latest release: about 1 month ago - 98 dependent packages - 7,363 dependent repositories - 2.09 million downloads last month - 841 stars on GitHub - 2 maintainers
isbot 5.1.6
🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.126 versions - Latest release: about 1 month ago - 98 dependent packages - 7,363 dependent repositories - 2.09 million downloads last month - 841 stars on GitHub - 2 maintainers
robots-txt-component 0.0.4
Lightweight robots.txt parsing component without any external dependencies for Node.js.3 versions - Latest release: over 1 year ago - 23 downloads last month - 0 stars on GitHub - 1 maintainer
simplecrawling 0.0.3
Crawler made simple3 versions - Latest release: over 9 years ago - 1 dependent repositories - 12 downloads last month - 1 maintainer
@drorgl/sitemapper 2.1.17
Parser for XML Sitemaps to be used with Robots.txt and web crawlers3 versions - Latest release: almost 4 years ago - 1 dependent package - 1 dependent repositories - 12 downloads last month - 0 stars on GitHub - 1 maintainer
crawler-js 2.0.10
Opensource Framework Crawler in Node.js48 versions - Latest release: over 5 years ago - 1 dependent package - 3 dependent repositories - 149 downloads last month - 1 maintainer
@passmarked/crawlable 1.0.1
Fake server that produces the specified amount of pages in directories to test web crawlers2 versions - Latest release: over 7 years ago - 2 dependent packages - 1 dependent repositories - 2 downloads last month - 1 stars on GitHub - 1 maintainer
norch-crawlers 0.0.3
A NodeJS crawler library to quick and easy build versatile crawlers.3 versions - Latest release: about 6 years ago - 2 dependent packages - 1 dependent repositories - 6 downloads last month - 1 stars on GitHub - 1 maintainer
@qbbsh/sitemapper
Parser for XML Sitemaps to be used with Robots.txt and web crawlers1 version - 98 stars on GitHub - 1 maintainer
adibenmat-sitemapper 2.1.19
Parser for XML Sitemaps to be used with Robots.txt and web crawlers2 versions - Latest release: about 3 years ago - 2 dependent packages - 10 downloads last month - 0 stars on GitHub - 1 maintainer
bot-detector 1.0.10
It uses the user-agents.org xml file for detecting bots.11 versions - Latest release: almost 9 years ago - 1 dependent package - 1 dependent repositories - 141 downloads last month - 1 maintainer
aliyu-crawlers 1.0.1
crawlers2 versions - Latest release: almost 2 years ago - 2 dependent packages - 4 downloads last month - 1 maintainer
sitemapper_mos 3.2.9
Parser for XML Sitemaps to be used with Robots.txt and web crawlers6 versions - Latest release: about 2 years ago - 1 dependent package - 119 downloads last month - 0 stars on GitHub - 1 maintainer
@mastixmc/sitemapper 3.2.0
Parser for XML Sitemaps to be used with Robots.txt and web crawlers. (Extended version by mastixmc)3 versions - Latest release: about 5 years ago - 1 dependent package - 1 dependent repositories - 6 downloads last month - 1 maintainer
isbot-bundle 1.0.4
isbot Bundle UMD5 versions - Latest release: about 4 years ago - 1 dependent package - 9 downloads last month - 0 stars on GitHub - 1 maintainer
Top 4.4% on npmjs.org
6 versions - Latest release: about 7 years ago - 7 dependent packages - 3,977 dependent repositories - 49.1 thousand downloads last month - 4 stars on GitHub - 1 maintainer
robot-directives 0.4.0
Parse robot directives within HTML meta and/or HTTP headers.6 versions - Latest release: about 7 years ago - 7 dependent packages - 3,977 dependent repositories - 49.1 thousand downloads last month - 4 stars on GitHub - 1 maintainer
@cabbiepete/sitemapper 3.0.2
Parser for XML Sitemaps to be used with Robots.txt and web crawlers3 versions - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 10 downloads last month - 1 stars on GitHub - 1 maintainer
meetup-crawler 0.1.0
This is a crawler that search all the next events of all the events under a set of categories and...1 version - Latest release: over 7 years ago - 2 dependent packages - 1 dependent repositories - 6 downloads last month - 2 stars on GitHub - 1 maintainer
@javivelasco/isbot 3.3.3
🤖 detect bots/crawlers/spiders via the user agent.2 versions - Latest release: over 2 years ago - 1 dependent package - 18 downloads last month - 838 stars on GitHub - 1 maintainer
@context-labs/sitemapper 3.2.10
Parser for XML Sitemaps to be used with Robots.txt and web crawlers2 versions - Latest release: about 1 year ago - 4 dependent packages - 736 downloads last month - 98 stars on GitHub - 3 maintainers
Top 6.6% on npmjs.org
4 versions - Latest release: over 1 year ago - 8 dependent packages - 1 dependent repositories - 2 downloads last month - 11,973 stars on GitHub - 1 maintainer
@vladfrangu-dev/crawlee-utils 3.3.0
A set of shared utilities that can be used by crawlers4 versions - Latest release: over 1 year ago - 8 dependent packages - 1 dependent repositories - 2 downloads last month - 11,973 stars on GitHub - 1 maintainer
@public-square/sitemapper 3.2.5
Parser for XML Sitemaps to be used with Robots.txt and web crawlers2 versions - Latest release: almost 2 years ago - 2 dependent packages - 1 dependent repositories - 9 downloads last month - 98 stars on GitHub - 2 maintainers
Top 2.3% on npmjs.org
44 versions - Latest release: 7 months ago - 37 dependent packages - 76 dependent repositories - 88.6 thousand downloads last month - 98 stars on GitHub - 1 maintainer
sitemapper 3.2.8
Parser for XML Sitemaps to be used with Robots.txt and web crawlers44 versions - Latest release: 7 months ago - 37 dependent packages - 76 dependent repositories - 88.6 thousand downloads last month - 98 stars on GitHub - 1 maintainer
ccht 0.1.2 💰
A simple command0line tool to crawl and test your website3 versions - Latest release: almost 2 years ago - 1 dependent package - 5 downloads last month - 1 stars on GitHub - 1 maintainer
planisphere 1.0.1
A straightforward sitemap generator written in TypeScript.3 versions - Latest release: about 3 years ago - 1 dependent package - 16 downloads last month - 5 stars on GitHub - 1 maintainer
@jarronb/site-mapper 1.0.5
Site mapper utilizing puppeteer to generate a xml sitemap6 versions - Latest release: almost 4 years ago - 1 dependent package - 7 downloads last month - 0 stars on GitHub - 1 maintainer
@microlink/ua 1.2.4
A simple redis primitives to incr() and top() user agents11 versions - Latest release: about 1 month ago - 400 downloads last month - 1 stars on GitHub - 1 maintainer
hidemyemail 0.1.0
A jQuery plugin that helps you to hide your email on your page and prevent crawlers to get it!1 version - Latest release: almost 9 years ago - 1 dependent package - 1 dependent repositories - 7 downloads last month - 2 stars on GitHub - 1 maintainer
@wmod/sitemap-url-scrapper 1.0.0 removed
Parser for XML Sitemaps to be used with Robots.txt and web crawlers1 version - Latest release: 4 months ago - 1 maintainer
@myagizmaktav/sitemapper 3.2.8 removed
Parser for XML Sitemaps to be used with Robots.txt and web crawlers1 version - Latest release: over 1 year ago - 58 downloads last month - 0 stars on GitHub - 1 maintainer
Related Keywords
robots.txt
17
sitemap
15
sitemaps
15
parse
14
xml
14
webcrawler
14
javascript
7
spiders
7
crawler
7
crawling
5
bot
5
useragent
5
googlebot
5
user-agent
4
user-agent-parser
4
nodejs
4
sitemap-xml
4
puppeteer
3
robots
3
web-crawlers
3
user-agent-analysis
3
spider
2
seo
2
🤖
2
http
2
user agent parsing
2
web-scraping
2
web-crawling
2
web-crawler
2
utils
2
utilities
2
automation
2
apify
2
headless
2
typescript
2
scraping
2
scraper
2
playwright
2
npm
2
headless-chrome
2
bot detection
1
header
1
html
1
npl
1
meta
1
robots-parser
1
metadata
1
nofollow
1
noindex
1
attributes
1
headers
1
meetup
1
cli
1
generator
1
navigation
1
node-module
1
sitemap-builder
1
sitemap-generator
1
typescript-module
1
redis
1
jquery-plugin
1
ecosystem:jquery
1
jquery
1
antispam
1
email-protection
1
hide-my-email
1
script
1
robots-exclusion-standard
1
js
1
node-js
1
robots-txt
1
robots-node
1
robots-txt-node
1
web-request
1
request
1
requests
1
scrapy
1
hacker
1
robot
1
dom
1
extraction
1
nsa
1
bigdata
1
passmarked
1
performance
1
pagespeed
1
security
1
compatibility
1
standards
1
crawlable
1
fake
1
emulate
1
directory
1
browse
1
benchmarking
1
testing
1