crates.io : bpetok
A simple CLI for tokenizing text input using Byte Pair Encoding (BPE).
Registry
- Documentation
- JSON
purl: pkg:cargo/bpetok
Keywords:
bpe
, cli
, text
, tokenization
, tokenizer
License: Other
Latest release: 10 months ago
First release: 10 months ago
Downloads: 2,495 total
Last synced: 18 days ago
Loading...
Readme
Loading...