Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
Top 5.3% downloads on rubygems.org
Top 2.1% dependent packages on rubygems.org
Top 4.6% dependent repos on rubygems.org
Top 8.3% forks on rubygems.org
rubygems.org : tokenizer
A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.
Registry
-
Source
- Documentation
- JSON
purl: pkg:gem/tokenizer
Keywords: natural-language-processing, nlp, ruby, rubynlp, tokenizer
License: MIT
Latest release: over 8 years ago
First release: about 13 years ago
Dependent packages: 9
Dependent repositories: 28
Downloads: 224,851 total
Stars: 45 on GitHub
Forks: 11 on GitHub
See more repository details: repos.ecosyste.ms
Funding links: https://github.com/sponsors/arbox
Last synced: 6 days ago