Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
proxy.golang.org : github.com/Doist/s3logger
Program s3logger collects json messages over TCP, stores them into gzip-compressed files split by time and uploads these files to AWS s3 bucket. Once started, service accepts TCP connections and expects clients to send streams of json objects over such connections. s3logger only closes connection if it encounters malformed json or single object size exceeds 4 MiB limit. s3logger only reads data from the client. s3logger accumulates received messages over predefined time window (-t flag) to a temporary log file creating new ones as needed; previous files are uploaded to s3 bucket in background and removed after successful upload. Optionally maximum size of input read can be specified in megabytes (-mb flag) to rotate file before reaching predefined time. Program only writes to a single temporary log file at a time, so json messages received from multiple concurrent connections are interleaved into a single json stream. It does its best not to lose messages, but can still drop them if they're coming faster than could be saved on disk or there's any disk write error. Stored messages are separated by new line (0xa). s3logger uploads files to a specified bucket using predefined s3 object naming scheme: It uses dt=YYYY-MM-DD "directories", object name base starting with date and time when log file was created (UTC) followed by hex-encoded 64-bit random value and .json.gz suffix. See https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables on how to configure s3 bucket access credentials. s3logger does not use TLS for its listener at the moment as it is expected to run on localhost or inside trusted network.
Registry
-
Source
- Documentation
- JSON
purl: pkg:golang/github.com/%21doist/s3logger
License: MIT
Latest release: about 2 years ago
First release: over 6 years ago
Namespace: github.com/Doist
Stars: 1 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 23 days ago