Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
Top 4.0% downloads on crates.io
Top 3.0% dependent packages on crates.io
Top 2.4% dependent repos on crates.io
Top 1.5% forks on crates.io
Top 3.1% docker downloads on crates.io
crates.io : tokenizers
Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Registry
-
Source
- Documentation
- JSON
purl: pkg:cargo/tokenizers
Keywords: nlp, tokenizer, wordpiece, bpe, huggingface, bert, gpt, language-model, natural-language-processing, natural-language-understanding, transformers
License: Apache-2.0
Latest release: about 1 month ago
First release: almost 5 years ago
Dependent packages: 60
Dependent repositories: 281
Downloads: 772,466 total
Stars: 8,489 on GitHub
Forks: 729 on GitHub
Docker dependents: 35
Docker downloads: 23,287,808
Total Commits: 1640
Committers: 97
Average commits per author: 16.907
Development Distribution Score (DDS): 0.43
More commit stats: commits.ecosyste.ms
See more repository details: repos.ecosyste.ms
Last synced: 4 days ago