crates.io : tokenizers-enfer
Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Registry
-
Source
- Homepage
- Documentation
- JSON
purl: pkg:cargo/tokenizers-enfer
Keywords:
bpe
, huggingface
, nlp
, tokenizer
, wordpiece
License: Apache-2.0
Latest release: 8 months ago
First release: 8 months ago
Downloads: 3,182 total
Last synced: 11 days ago
Loading...
Readme
Loading...