pypi.org : tokenization-scorer

Package for evaluating text tokenizations.

Registry - Source - Documentation - JSON
purl: pkg:pypi/tokenization-scorer
Keywords: tokenization , evaluation , natural language processing , bpe , segmentation , subword
License: MIT
Latest release: 3 months ago
First release: almost 2 years ago
Downloads: 447 last month
Stars: 39 on GitHub
Forks: 4 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 10 days ago

• Readme
• Versions 12
• Dependent Packages 0
• Maintainers 1