proxy.golang.org : github.com/stevenayers/clamber/service/app
Package app provides the clamber crawling package. To initiate a crawl, create a Crawler with an empty sync.WaitGroup and struct map. DbWaitGroup is needed to ensure the clamber process does not exit before the crawler is done writing to the database. AlreadyCrawled keeps track of the URLs which have been crawled already in that crawl process. The rest are self explanatory. Create a page object with the starting URL of your crawl. Call Crawl on the Crawler object, passing in your page, and the depth of the crawl you want. Ensure your go process does not end before the crawled data has been saved to dgraph. If you need more logic to execute first, put the line below after this, as your application will hang on Wait() until we're done writing.
Registry
-
Source
- Documentation
- JSON
purl: pkg:golang/github.com/stevenayers/clamber/service/app
License: Apache-2.0
Latest release: over 5 years ago
First release: over 5 years ago
Namespace: github.com/stevenayers/clamber/service
Last synced: 12 days ago