pypi.org : ultra-tokenizer
Advanced tokenizer with support for BPE, WordPiece, and Unigram algorithms
Registry
-
Source
- Documentation
- JSON
purl: pkg:pypi/ultra-tokenizer
Keywords:
tokenizer
, nlp
, bpe
, wordpiece
, unigram
License: Apache-2.0
Latest release: 23 days ago
First release: 23 days ago
Stars: 1 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 23 days ago
Loading...
Readme
Loading...