pypi.org : count-tokens
Count number of tokens in the text file using toktoken tokenizer from OpenAI.
Registry
-
Source
- Documentation
- JSON
purl: pkg:pypi/count-tokens
Keywords:
count
, tokens
, toktoken
, openai
, tokenizer
, tiktoken
, token-count
, tokenization
, tokenizer-nlp
License: MIT
Latest release: 3 months ago
First release: almost 2 years ago
Downloads: 6,634 last month
Stars: 6 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 6 days ago
No dependent packages found.