proxy.golang.org : github.com/gomlx/tokenizers
Package tokenizers provides an implementation of today's most used tokenizers, with a focus on performance and versatility. It is currently a wrapper around the Rust implementation in https://github.com/huggingface/tokenizers/tree/main/tokenizers. For now, it only provides the encoding and decoding functionality -- not training a new tokenizers. It includes reading from [HuggingFace's pretrained tokenizers](https://huggingface.co/docs/tokenizers/index) using `FromPretrained`.
Registry
-
Source
- Documentation
- JSON
- codemeta.json
purl: pkg:golang/github.com/gomlx/tokenizers
License: MIT
Latest release: about 2 years ago
First release: about 2 years ago
Namespace: github.com/gomlx
Stars: 6 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: about 1 month ago