crates.io "web-crawler" keyword
View the packages on the crates.io package registry that are tagged with the "web-crawler" keyword.
siteprobe 1.0.0
Siteprobe is a Rust-based CLI tool that fetches all URLs from a given `sitemap.xml` url, checks t...8 versions - Latest release: 3 days ago - 3.37 thousand downloads total - 3 stars on GitHub - 1 maintainer
crusty-core 0.82.0
Library for creating blazing fast and configurable web crawlers87 versions - Latest release: almost 4 years ago - 1 dependent package - 1 dependent repositories - 103 thousand downloads total - 16 stars on GitHub - 1 maintainer
spider-client 0.1.77
Spider Cloud client71 versions - Latest release: 10 days ago - 35 thousand downloads total - 19 stars on GitHub - 1 maintainer
tarzi 0.0.15
Rust-native lite search for AI applications10 versions - Latest release: about 2 months ago - 3.15 thousand downloads total - 2 stars on GitHub - 1 maintainer
waxy 0.2.0
web crawler for community driven search engine7 versions - Latest release: over 3 years ago - 7.81 thousand downloads total - 2 stars on GitHub - 1 maintainer
spider-cloud-cli 0.1.77
The Spider Cloud CLI for web crawling and scraping56 versions - Latest release: 10 days ago - 21.3 thousand downloads total - 1 maintainer
maman 0.13.1
Rust Web Crawler16 versions - Latest release: almost 7 years ago - 26.4 thousand downloads total - 44 stars on GitHub - 1 maintainer
spider_utils 2.37.153
Utilities to use for Spider Web Crawler.850 versions - Latest release: 16 days ago - 694 thousand downloads total - 842 stars on GitHub - 1 maintainer
frangipani 0.3.1 💰
Scraping framework for rust4 versions - Latest release: over 2 years ago - 4.74 thousand downloads total - 3 stars on GitHub - 1 maintainer
firecrawl 1.2.1
Rust SDK for Firecrawl API.5 versions - Latest release: 27 days ago - 4.61 thousand downloads total - 48,469 stars on GitHub - 2 maintainers
firecrawl_rs 0.1.1
Rust SDK for Firecrawl API.2 versions - Latest release: about 1 year ago - 2.12 thousand downloads total - 48,469 stars on GitHub - 2 maintainers
seaward 1.1.0
seaward is a grep-like tool for the web.5 versions - Latest release: 3 months ago - 5.25 thousand downloads total - 3 stars on GitHub - 1 maintainer
wrake 0.3.0
Collect links from the given URL4 versions - Latest release: almost 4 years ago - 4.51 thousand downloads total - 0 stars on GitHub - 1 maintainer
dyer 3.3.2
dyer is designed for reliable, flexible and fast Request-Response based service, including data ...19 versions - Latest release: about 3 years ago - 1 dependent package - 1.37 million downloads total - 127 stars on GitHub - 1 maintainer
capp 0.4.3
Common things i use to build Rust CLI tools for web crawlers.9 versions - Latest release: 10 months ago - 9.58 thousand downloads total - 1 stars on GitHub - 1 maintainer
dyer-macros 0.2.0
Dyer-macros is a collection of macros when compiling and building dyer.2 versions - Latest release: almost 4 years ago - 1 dependent package - 1 dependent repositories - 144 thousand downloads total - 127 stars on GitHub - 1 maintainer
website_crawler 0.9.9 💰
gRPC tokio based web crawler built with spider203 versions - Latest release: about 1 year ago - 168 thousand downloads total - 53 stars on GitHub - 1 maintainer
Related Keywords
crawler
13
spider
8
scraper
6
web-scraping
5
ai-scraping
4
scraping
4
ai
4
rust
4
html-to-markdown
3
rag
3
web-indexer
3
web-framework
2
web
2
webscraping
2
markdown
2
data
2
llm
2
rust-programming-language
2
web-scraper
2
async
2
request-response
1
web-crawling
1
data-processing
1
executor
1
dyer
1
rustcrawler
1
cli
1
dyer-cli
1
site-map-generator
1
a11ywatch
1
grpc
1
continuous-crawler
1
llm-crawler
1
indexer
1
headless-chrome
1
http
1
search
1
supabase
1
llm-webcrawler
1
ai-agents
1
tokio
1
multi-threaded
1
url-checker
1
sitemap
1
performance
1
http-monitoring
1