Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
proxy.golang.org : github.com/dineshsprabu/concurrent-web-crawler
Concurrent Web Crawler is a light weight configurable crawler with concurrecy enabled. Configrations like 'MaxConcurrencyLimit' will be helpful for controlled concurrency with limited number of subroutines, 'CrawlDelay' can be helpful to slow down the crawling rate on subroutine level. 'StoragePath' can be useful for specifying stroage location where pages could be saved. Note: Please be careful while crawling websites and do read the robots.txt before you crawl a website.
Registry
-
Source
- Documentation
- JSON
purl: pkg:golang/github.com/dineshsprabu/concurrent-web-crawler
Keywords: concurrent-web-crawler, crawler, go-crawler, spider, web-crawler
License: MIT
Latest release: about 7 years ago
First release: about 7 years ago
Namespace: github.com/dineshsprabu
Stars: 0 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 19 days ago