{"id":3512940,"name":"github.com/codahale/lunk","ecosystem":"go","description":"Package lunk provides a set of tools for structured logging in the style of\nGoogle's Dapper or Twitter's Zipkin.\n\nWhen we consider a complex event in a distributed system, we're actually\nconsidering a partially-ordered tree of events from various services,\nlibraries, and modules.\n\nConsider a user-initiated web request. Their browser sends an HTTP request to\nan edge server, which extracts the credentials (e.g., OAuth token) and\nauthenticates the request by communicating with an internal authentication\nservice, which returns a signed set of internal credentials (e.g., signed\nuser ID). The edge web server then proxies the request to a cluster of web\nservers, each running a PHP application. The PHP application loads some data\nfrom several databases, places the user in a number of treatment groups for\nrunning A/B experiments, writes some data to a Dynamo-style distributed\ndatabase, and returns an HTML response. The edge server receives this\nresponse and proxies it to the user's browser.\n\nIn this scenario we have a number of infrastructure-specific events:\n\nThis scenario also involves a number of events which have little to do with\nthe infrastructure, but are still critical information for the business the\nsystem supports:\n\nThere are a number of different teams all trying to monitor and improve\naspects of this system. Operational staff need to know if a particular host\nor service is experiencing a latency spike or drop in throughput. Development\nstaff need to know if their application's response times have gone down as a\nresult of a recent deploy. Customer support staff need to know if the system\nis operating nominally as a whole, and for customers in particular. Product\ndesigners and managers need to know the effect of an A/B test on user\nbehavior. But the fact that these teams will be consuming the data in\ndifferent ways for different purposes does mean that they are working on\ndifferent systems.\n\nIn order to instrument the various components of the system, we need a common\ndata model.\n\nWe adopt Dapper's notion of a tree to mean a partially-ordered tree of events\nfrom a distributed system. A tree in Lunk is identified by its root ID, which\nis the unique ID of its root event. All events in a common tree share a root\nID.  In our photo example, we would assign a unique root ID as soon as the\nedge server received the request.\n\nEvents inside a tree are causally ordered: each event has a unique ID, and an\noptional parent ID. By passing the IDs across systems, we establish causal\nordering between events. In our photo example, the two database queries from\nthe app would share the same parent ID--the ID of the event corresponding to\nthe app handling the request which caused those queries.\n\nEach event has a schema of properties, which allow us to record specific\npieces of information about each event. For HTTP requests, we can record the\nmethod, the request URI, the elapsed time to handle the request, etc.\n\nLunk is agnostic in terms of aggregation technologies, but two use cases seem\nclear: real-time process monitoring and offline causational analysis.\n\nFor real-time process monitoring, events can be streamed to a aggregation\nservice like Riemann (http://riemann.io) or Storm\n(http://storm.incubator.apache.org), which can calculate process statistics\n(e.g., the 95th percentile latency for the edge server responses) in\nreal-time. This allows for adaptive monitoring of all services, with the\noption of including example root IDs in the alerts (e.g., 95th percentile\nlatency is over 300ms, mostly as a result of requests like those in tree\nXXXXX).\n\nFor offline causational analysis, events can be written in batches to batch\nprocessing systems like Hadoop or OLAP databases like Vertica. These\naggregates can be queried to answer questions traditionally reserved for A/B\ntesting systems. \"Did users who were show the new navbar view more photos?\"\n\"Did the new image optimization algorithm we enabled for 1% of views run\nfaster? Did it produce smaller images? Did it have any effect on user\nengagement?\" \"Did any services have increased exception rates after any\nrecent deploys?\" \u0026tc \u0026tc\n\nBy capturing the root ID of a particular web request, we can assemble a\npartially-ordered tree of events which were involved in the handling of that\nrequest. All events with a common root ID are in a common tree, which allows\nfor O(M) retrieval for a tree of M events.\n\nTo send a request with a root ID and a parent ID, use the Event-ID HTTP\nheader:\n\nThe header value is simply the root ID and event ID, hex-encoded and\nseparated with a slash. If the event has a parent ID, that may be included as\nan optional third parameter.  A server that receives a request with this\nheader can use this to properly parent its own events.\n\nEach event has a set of named properties, the keys and values of which are\nstrings. This allows aggregation layers to take advantage of simplifying\nassumptions and either store events in normalized form (with event data\nseparate from property data) or in denormalized form (essentially\npre-materializing an outer join of the normalized relations). Durations are\nalways recorded as fractional milliseconds.\n\nLunk currently provides two formats for log entries: text and\nJSON. Text-based logs encode each entry as a single line of text, using\nkey=\"value\" formatting for all properties. Event property keys are scoped to\navoid collisions. JSON logs encode each entry as a single JSON object.","homepage":"https://github.com/codahale/lunk","licenses":"MIT","normalized_licenses":["MIT"],"repository_url":"https://github.com/codahale/lunk","keywords_array":[],"namespace":"github.com/codahale","versions_count":1,"first_release_published_at":"2014-11-20T05:46:18.000Z","latest_release_published_at":"2014-11-20T05:46:18.000Z","latest_release_number":"v0.0.0-20141120054618-3bb9f3be3053","last_synced_at":"2026-05-05T00:03:46.168Z","created_at":"2022-04-10T21:23:06.045Z","updated_at":"2026-05-05T00:03:46.169Z","registry_url":"https://pkg.go.dev/github.com/codahale/lunk","install_command":"go get github.com/codahale/lunk","documentation_url":"https://pkg.go.dev/github.com/codahale/lunk#section-documentation","metadata":{},"repo_metadata":{"id":16507708,"uuid":"19260671","full_name":"codahale/lunk","owner":"codahale","description":"lunk provides a set of tools for structured logging in the style of Google's Dapper or Twitter's Zipkin","archived":true,"fork":false,"pushed_at":"2014-11-20T05:46:18.000Z","size":316,"stargazers_count":127,"open_issues_count":0,"forks_count":6,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-06-19T02:02:08.117Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codahale.png","metadata":{"files":{"readme":null,"changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-04-29T02:09:18.000Z","updated_at":"2024-02-27T07:47:52.000Z","dependencies_parsed_at":"2022-08-25T16:41:10.199Z","dependency_job_id":null,"html_url":"https://github.com/codahale/lunk","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/codahale/lunk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codahale%2Flunk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codahale%2Flunk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codahale%2Flunk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codahale%2Flunk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codahale","download_url":"https://codeload.github.com/codahale/lunk/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codahale%2Flunk/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269642534,"owners_count":24452341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-09T02:00:10.424Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"},"owner_record":{"login":"codahale","name":"Coda Hale","uuid":"207","kind":"user","description":"A real person.","email":"","website":"https://codahale.com","location":"Fort Collins, CO","twitter":null,"company":null,"icon_url":"https://avatars.githubusercontent.com/u/207?u=bca8f37568423d246fcbec047a92ec4b89280b51\u0026v=4","repositories_count":116,"last_synced_at":"2023-04-10T21:25:44.623Z","metadata":{"has_sponsors_listing":false},"html_url":"https://github.com/codahale","funding_links":[],"total_stars":null,"followers":null,"following":null,"created_at":"2022-11-02T16:59:22.612Z","updated_at":"2023-04-10T21:25:44.799Z","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codahale","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codahale/repositories"},"tags":[]},"repo_metadata_updated_at":"2025-08-09T23:21:50.161Z","dependent_packages_count":0,"downloads":null,"downloads_period":null,"dependent_repos_count":1,"rankings":{"downloads":null,"dependent_repos_count":4.773700988852565,"dependent_packages_count":9.411358606266116,"stargazers_count":4.142825235030153,"forks_count":7.691729781315357,"docker_downloads_count":null,"average":6.504903652866048},"purl":"pkg:golang/github.com/codahale/lunk","advisories":[],"docker_usage_url":"https://docker.ecosyste.ms/usage/go/github.com/codahale/lunk","docker_dependents_count":null,"docker_downloads_count":null,"usage_url":"https://repos.ecosyste.ms/usage/go/github.com/codahale/lunk","dependent_repositories_url":"https://repos.ecosyste.ms/api/v1/usage/go/github.com/codahale/lunk/dependencies","status":null,"funding_links":[],"critical":null,"issue_metadata":{"last_synced_at":"2023-05-31T03:16:37.143Z","issues_count":1,"pull_requests_count":1,"avg_time_to_close_issue":15716.0,"avg_time_to_close_pull_request":47196.0,"issues_closed_count":1,"pull_requests_closed_count":1,"pull_request_authors_count":1,"issue_authors_count":1,"avg_comments_per_issue":1.0,"avg_comments_per_pull_request":1.0,"merged_pull_requests_count":1,"bot_issues_count":0,"bot_pull_requests_count":0,"past_year_issues_count":0,"past_year_pull_requests_count":0,"past_year_avg_time_to_close_issue":null,"past_year_avg_time_to_close_pull_request":null,"past_year_issues_closed_count":0,"past_year_pull_requests_closed_count":0,"past_year_pull_request_authors_count":0,"past_year_issue_authors_count":0,"past_year_avg_comments_per_issue":null,"past_year_avg_comments_per_pull_request":null,"past_year_bot_issues_count":0,"past_year_bot_pull_requests_count":0,"past_year_merged_pull_requests_count":0,"issues_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/codahale%2Flunk/issues","maintainers":[],"active_maintainers":[]},"versions_url":"https://packages.ecosyste.ms/api/v1/registries/proxy.golang.org/packages/github.com%2Fcodahale%2Flunk/versions","version_numbers_url":"https://packages.ecosyste.ms/api/v1/registries/proxy.golang.org/packages/github.com%2Fcodahale%2Flunk/version_numbers","latest_version_url":"https://packages.ecosyste.ms/api/v1/registries/proxy.golang.org/packages/github.com%2Fcodahale%2Flunk/latest_version","dependent_packages_url":"https://packages.ecosyste.ms/api/v1/registries/proxy.golang.org/packages/github.com%2Fcodahale%2Flunk/dependent_packages","related_packages_url":"https://packages.ecosyste.ms/api/v1/registries/proxy.golang.org/packages/github.com%2Fcodahale%2Flunk/related_packages","codemeta_url":"https://packages.ecosyste.ms/api/v1/registries/proxy.golang.org/packages/github.com%2Fcodahale%2Flunk/codemeta","maintainers":[]}