proxy.golang.org : github.com/mohae/geomi
geomi, 거미, is a basic spider for crawling specific webs. It was designed to crawl either a site, or a subset of a site, indexing the pages and where they link to. Any links that go outside of the provided base URL are noted but not followed. Even though geomi was designed with the intended use being crawling ones own site, or a client's site, it does come with some basic behavior configuration to make it a friendly bot: To start, go get the geomi package: Import it into your code: Set up the site for the spider: The url that is passed to the NewSite() function is the base case, after which a BFS traversal is done.
Registry
-
Source
- Documentation
- JSON
- codemeta.json
purl: pkg:golang/github.com/mohae/geomi
License: MIT
Latest release: about 10 years ago
First release: about 10 years ago
Namespace: github.com/mohae
Stars: 0 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 24 days ago