Ecosyste.ms: Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

pypi.org : tokengeex

TokenGeeX is an efficient tokenizer for code based on UnigramLM and TokenMonster.

Registry - Source - Homepage - Documentation - JSON
purl: pkg:pypi/tokengeex
Keywords: tokenizer, nlp
License: Apache-2.0
Latest release: 17 days ago
First release: 4 months ago
Downloads: 3,960 last month
Stars: 3 on GitHub
Forks: 1 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: about 14 hours ago

Readme