crates.io : token-counter
`wc` for tokens: count tokens in files with HF Tokenizers
Registry
-
Source
- Documentation
- JSON
purl: pkg:cargo/token-counter
Keywords:
cli
, nlp
, tokenizer
License: MIT
Latest release: about 1 year ago
First release: about 1 year ago
Downloads: 1,115 total
Stars: 7 on GitHub
Forks: 1 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 10 days ago
Loading...
Readme
Loading...