pypi.org : tokenization-scorer
Package for evaluating text tokenizations.
Registry
-
Source
- Documentation
- JSON
purl: pkg:pypi/tokenization-scorer
Keywords:
tokenization
, evaluation
, natural language processing
, bpe
, segmentation
, subword
License: MIT
Latest release: 8 months ago
First release: over 2 years ago
Downloads: 82 last month
Stars: 44 on GitHub
Forks: 5 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: about 1 month ago
Loading...
Readme
Loading...