Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
pypi.org : count-tokens
Count number of tokens in the text file using toktoken tokenizer from OpenAI.
Registry
-
Source
- Documentation
- JSON
purl: pkg:pypi/count-tokens
Keywords: count, tokens, toktoken, openai, tokenizer, tiktoken, token-count, tokenization, tokenizer-nlp
License: MIT
Latest release: 8 months ago
First release: 11 months ago
Downloads: 2,037 last month
Stars: 3 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 20 days ago
Loading...
Readme
Loading...