Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
npmjs.org "crawlers" keyword
aliyu-crawlers 1.0.1
crawlers2 versions - Latest release: almost 2 years ago - 2 dependent packages - 8 downloads last month - 1 maintainer
@context-labs/sitemapper 3.2.10
Parser for XML Sitemaps to be used with Robots.txt and web crawlers2 versions - Latest release: 12 months ago - 4 dependent packages - 736 downloads last month - 98 stars on GitHub - 3 maintainers
adibenmat-sitemapper 2.1.19
Parser for XML Sitemaps to be used with Robots.txt and web crawlers2 versions - Latest release: about 3 years ago - 2 dependent packages - 10 downloads last month - 0 stars on GitHub - 1 maintainer
Top 1.2% on npmjs.org
1,008 versions - Latest release: 30 days ago - 14 dependent packages - 88 dependent repositories - 111 thousand downloads last month - 10,899 stars on GitHub - 10 maintainers
@crawlee/utils 3.9.2
A set of shared utilities that can be used by crawlers1,008 versions - Latest release: 30 days ago - 14 dependent packages - 88 dependent repositories - 111 thousand downloads last month - 10,899 stars on GitHub - 10 maintainers
Top 6.6% on npmjs.org
4 versions - Latest release: over 1 year ago - 8 dependent packages - 1 dependent repositories - 2 downloads last month - 11,973 stars on GitHub - 1 maintainer
@vladfrangu-dev/crawlee-utils 3.3.0
A set of shared utilities that can be used by crawlers4 versions - Latest release: over 1 year ago - 8 dependent packages - 1 dependent repositories - 2 downloads last month - 11,973 stars on GitHub - 1 maintainer
@public-square/sitemapper 3.2.5
Parser for XML Sitemaps to be used with Robots.txt and web crawlers2 versions - Latest release: almost 2 years ago - 2 dependent packages - 1 dependent repositories - 9 downloads last month - 98 stars on GitHub - 2 maintainers
Top 2.3% on npmjs.org
44 versions - Latest release: 6 months ago - 37 dependent packages - 76 dependent repositories - 88.6 thousand downloads last month - 98 stars on GitHub - 1 maintainer
sitemapper 3.2.8
Parser for XML Sitemaps to be used with Robots.txt and web crawlers44 versions - Latest release: 6 months ago - 37 dependent packages - 76 dependent repositories - 88.6 thousand downloads last month - 98 stars on GitHub - 1 maintainer
ccht 0.1.2 💰
A simple command0line tool to crawl and test your website3 versions - Latest release: almost 2 years ago - 1 dependent package - 5 downloads last month - 1 stars on GitHub - 1 maintainer
@javivelasco/isbot 3.3.3
🤖 detect bots/crawlers/spiders via the user agent.2 versions - Latest release: over 2 years ago - 1 dependent package - 26 downloads last month - 838 stars on GitHub - 1 maintainer
Top 1.2% on npmjs.org
123 versions - Latest release: 18 days ago - 98 dependent packages - 7,363 dependent repositories - 2.1 million downloads last month - 838 stars on GitHub - 2 maintainers
isbot 5.1.6
🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.123 versions - Latest release: 18 days ago - 98 dependent packages - 7,363 dependent repositories - 2.1 million downloads last month - 838 stars on GitHub - 2 maintainers
planisphere 1.0.1
A straightforward sitemap generator written in TypeScript.3 versions - Latest release: almost 3 years ago - 1 dependent package - 16 downloads last month - 5 stars on GitHub - 1 maintainer
@jarronb/site-mapper 1.0.5
Site mapper utilizing puppeteer to generate a xml sitemap6 versions - Latest release: over 3 years ago - 1 dependent package - 7 downloads last month - 0 stars on GitHub - 1 maintainer
@microlink/ua 1.2.4
A simple redis primitives to incr() and top() user agents11 versions - Latest release: 10 days ago - 400 downloads last month - 1 stars on GitHub - 1 maintainer
hidemyemail 0.1.0
A jQuery plugin that helps you to hide your email on your page and prevent crawlers to get it!1 version - Latest release: over 8 years ago - 1 dependent package - 1 dependent repositories - 7 downloads last month - 2 stars on GitHub - 1 maintainer
@xianyukeji/isbot 2.3.0
detects bots/crawlers/spiders via the user agent.1 version - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 5 downloads last month - 837 stars on GitHub - 2 maintainers
@wmod/sitemapper 3.2.11
Parser for XML Sitemaps to be used with Robots.txt and web crawlers2 versions - Latest release: 3 months ago - 31 downloads last month - 0 stars on GitHub - 1 maintainer
robots-txt-component 0.0.4
Lightweight robots.txt parsing component without any external dependencies for Node.js.3 versions - Latest release: over 1 year ago - 15 downloads last month - 0 stars on GitHub - 1 maintainer
simplecrawling 0.0.3
Crawler made simple3 versions - Latest release: over 9 years ago - 1 dependent repositories - 17 downloads last month - 1 maintainer
@drorgl/sitemapper 2.1.17
Parser for XML Sitemaps to be used with Robots.txt and web crawlers3 versions - Latest release: almost 4 years ago - 1 dependent package - 1 dependent repositories - 14 downloads last month - 0 stars on GitHub - 1 maintainer
crawler-js 2.0.10
Opensource Framework Crawler in Node.js48 versions - Latest release: over 5 years ago - 1 dependent package - 3 dependent repositories - 126 downloads last month - 1 maintainer
norch-crawlers 0.0.3
A NodeJS crawler library to quick and easy build versatile crawlers.3 versions - Latest release: about 6 years ago - 2 dependent packages - 1 dependent repositories - 6 downloads last month - 1 stars on GitHub - 1 maintainer
@passmarked/crawlable 1.0.1
Fake server that produces the specified amount of pages in directories to test web crawlers2 versions - Latest release: over 7 years ago - 2 dependent packages - 1 dependent repositories - 5 downloads last month - 1 stars on GitHub - 1 maintainer
Top 4.4% on npmjs.org
6 versions - Latest release: about 7 years ago - 7 dependent packages - 3,977 dependent repositories - 49.5 thousand downloads last month - 3 stars on GitHub - 1 maintainer
robot-directives 0.4.0
Parse robot directives within HTML meta and/or HTTP headers.6 versions - Latest release: about 7 years ago - 7 dependent packages - 3,977 dependent repositories - 49.5 thousand downloads last month - 3 stars on GitHub - 1 maintainer
bot-detector 1.0.10
It uses the user-agents.org xml file for detecting bots.11 versions - Latest release: almost 9 years ago - 1 dependent package - 1 dependent repositories - 79 downloads last month - 1 maintainer
sitemapper_mos 3.2.9
Parser for XML Sitemaps to be used with Robots.txt and web crawlers6 versions - Latest release: about 2 years ago - 1 dependent package - 277 downloads last month - 0 stars on GitHub - 1 maintainer
@mastixmc/sitemapper 3.2.0
Parser for XML Sitemaps to be used with Robots.txt and web crawlers. (Extended version by mastixmc)3 versions - Latest release: about 5 years ago - 1 dependent package - 1 dependent repositories - 1 downloads last month - 1 maintainer
isbot-bundle 1.0.4
isbot Bundle UMD5 versions - Latest release: about 4 years ago - 1 dependent package - 10 downloads last month - 0 stars on GitHub - 1 maintainer
@cabbiepete/sitemapper 3.0.2
Parser for XML Sitemaps to be used with Robots.txt and web crawlers3 versions - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 24 downloads last month - 1 stars on GitHub - 1 maintainer
meetup-crawler 0.1.0
This is a crawler that search all the next events of all the events under a set of categories and...1 version - Latest release: over 7 years ago - 2 dependent packages - 1 dependent repositories - 3 downloads last month - 2 stars on GitHub - 1 maintainer
@wmod/sitemap-url-scrapper 1.0.0 removed
Parser for XML Sitemaps to be used with Robots.txt and web crawlers1 version - Latest release: 3 months ago - 1 maintainer
@myagizmaktav/sitemapper 3.2.8 removed
Parser for XML Sitemaps to be used with Robots.txt and web crawlers1 version - Latest release: about 1 year ago - 58 downloads last month - 0 stars on GitHub - 1 maintainer
Related Keywords
robots.txt
15
sitemap
13
sitemaps
13
parse
12
xml
12
webcrawler
12
javascript
7
spiders
7
crawler
7
crawling
5
useragent
5
googlebot
5
bot
5
user-agent-parser
4
user-agent
4
nodejs
4
sitemap-xml
4
robots
3
puppeteer
3
web-crawlers
3
user-agent-analysis
3
spider
2
seo
2
apify
2
🤖
2
user agent parsing
2
utilities
2
utils
2
automation
2
headless-chrome
2
npm
2
playwright
2
scraper
2
scraping
2
typescript
2
headless
2
http
2
web-scraping
2
web-crawling
2
web-crawler
2
extraction
1
nsa
1
dom
1
bigdata
1
npl
1
passmarked
1
performance
1
pagespeed
1
security
1
compatibility
1
standards
1
crawlable
1
fake
1
emulate
1
directory
1
browse
1
benchmarking
1
testing
1
header
1
html
1
meta
1
metadata
1
nofollow
1
noindex
1
attributes
1
headers
1
bot detection
1
meetup
1
cli
1
generator
1
navigation
1
node-module
1
sitemap-builder
1
sitemap-generator
1
typescript-module
1
redis
1
jquery-plugin
1
ecosystem:jquery
1
jquery
1
antispam
1
email-protection
1
hide-my-email
1
script
1
robots-parser
1
robots-exclusion-standard
1
js
1
node-js
1
robots-txt
1
robots-node
1
robots-txt-node
1
web-request
1
request
1
requests
1
scrapy
1
hacker
1
robot
1