pypi.org : tokendagger
High-Performance Implementation of OpenAI's TikToken - 2x Throughput, 4x Faster Code Tokenization
Registry
-
Source
- Documentation
- JSON
purl: pkg:pypi/tokendagger
Keywords:
tokenization
, tiktoken
, nlp
, text-processing
, openai
, bpe
License: Other
Latest release: 2 months ago
First release: 2 months ago
Downloads: 163 last month
Stars: 432 on GitHub
Forks: 7 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 22 days ago
Loading...
Readme
Loading...