Ecosyste.ms: Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

crates.io : tokengeex

TokenGeeX is an efficient tokenizer for code based on UnigramLM and TokenMonster.

Registry - Source - Homepage - Documentation - JSON
purl: pkg:cargo/tokengeex
Keywords: tokenizer, nlp, llm, codegeex
License: Apache-2.0
Latest release: 17 days ago
First release: 4 months ago
Downloads: 3,512 total
Stars: 3 on GitHub
Forks: 1 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: about 8 hours ago

Readme