proxy.golang.org : github.com/agentstation/tokenizer
High-performance tokenizer implementations in Go with unified CLI. Features Llama 3 tokenizer with exact compatibility, streaming support, and comprehensive tooling.
Registry
-
Source
- Documentation
- JSON
purl: pkg:golang/github.com/agentstation/tokenizer
Keywords:
ai
, bpe
, cli
, golang
, gomarkdoc
, goreleaser
, llama3
, llm
, machine-learning
, natural-language-processing
, nlp
, text-processing
, tokenizer
License: MIT
Latest release: 28 days ago
First release: 2 months ago
Stars: 0 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 26 days ago