Ecosyste.ms: Packages

proxy.golang.org : github.com/kristianlyng/skogul

Package skogul is a framework for receiving, processing and forwarding data, typically metric data or event-oriented data, at high throughput. It is designed to be as agnostic as possible with regards to how it transmits data and how it receives it, and the processors in between need not worry about how the data got there or how it will be treated in the next chain. This means you can use Skogul to receive data on a influxdb-like line-based TCP interface and send it on to postgres - or influxdb - without having to write explicit support, just set up the chain. The guiding principles of Skogul is: - Make as few assumptions as possible about how data is received - Be stupid fast End users should only need to worry about the cmd/skogul tool, which comes fully equipped with self-contained documentation. Adding new logic to Skogul should also be fairly easy. New developers should focus on understanding two things: 1. The skogul.Container data structure - which is the heart of Skogul. 2. The relationship from receiver to handler to sender. The Container is documented in this very package. Receivers are where data originates within Skogul. The typical Receiver will receive data from the outside world, e.g. by other tools posting data to a HTTP endpoint. Receivers can also be used to "create" data, either test data or, for example, log data. When skogul starts, it will start all receivers that are configured. Handlers determine what is done with the data once received. They are responsible for parsing raw data and optionally transform it. This is the only place where it is allowed to _modify_ data. Today, the only transformer is the "templater", which allows a collection of metrics which share certain attributes (e.g.: all collected at the same time and from the same machine) to provide these shared attributes in a template which the "templater" transformer then applies to all metrics. Other examples of transformations that make sense are: - Adding a metadata field - Removing a metadata field - Removing all but a specific set of fields - Converting nested metrics to multiple metrics or flatten them Once a handler has done its deed, it sends the Container to the sender, and this is where "the fun begins" so to speak. Senders consist of just a data structure that implements the Send() interface. They are not allowed to change the container, but besides that, they can do "whatever". The most obvious example is to send the container to a suitable storage system - e.g., a time series database. So if you want to add support for a new time series database in Skogul, you will write a sender. In addition to that, many senders serve only to add internal logic and pass data on to other senders. Each sender should only do one specific thing. For example, if you want to write data both to InfluxDB and MySQL, you need three senders: The "MySQL" and "InfluxDB" senders, and the "dupe" sender, which just takes a list of other senders and sends whatever it receives on to all of them. Today, Senders and Receivers both have an identical "Auto"-system, found in auto.go of the relevant directories. This is how the individual implementations are made discoverable to the configuration system, and how documentation is provided. Documentation for the settings of a sender/receiver is handled as struct tags. Once more parsers/transformers are added, they will likely also use a similar system.

Registry - Source - Documentation - JSON
purl: pkg:golang/github.com/kristianlyng/skogul
Keywords: api, golang, hacktoberfest, http, influxdb, metrics, metrics-gathering, mysql
License: LGPL-2.1
Latest release: about 2 years ago
First release: over 4 years ago
Namespace: github.com/kristianlyng
Stars: 22 on GitHub
Forks: 12 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 11 days ago

• Readme
• Versions 48
• Dependent Packages 0
• Maintainers 0
• Related Packages 1