Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
proxy.golang.org : github.com/spitfire55/goctftime
Package goctftime parses and stores data from ctftime.org into a Firestore database so that it can be easily indexed and queried by an Android application that displays CTF Time data. The various handlers listen on the server for a GET request to their respective path. This triggers the execution of the handler logic. Each handler is responsible for parsing and storing some portion of ctftime.org. Handler logic is broken up into two phases. The first phase triggers multiple goroutines to parse and store data concurrently. By default, the maximum number of goroutines running at once is 10. To change the maximum number of goroutines running at once, modify the maxRoutines variable. This concurrent phase only requests pages that we have scraped before. The second phase operates on a single thread and checks to see if new content exists. If new content exists, it is parsed and stored in Firestore. Finally, we update the value used in phase one to delineate the range of known content. Each scraper operates concurrently as it requests known content. The definition of "known" is based on a Firestore value for the respective piece of content (teams, events, etc.). This value is retrieved and possibly updated on each scraping iteration.
Registry
-
Source
- Documentation
- JSON
purl: pkg:golang/github.com/spitfire55/goctftime
Keywords: appengine, appengine-go, firebase-database, firestore, go, go-web, golang, scraping, scraping-websites
License: MIT
Latest release: over 6 years ago
First release: over 6 years ago
Namespace: github.com/spitfire55
Stars: 0 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 5 days ago