proxy.golang.org : github.com/LanguageMachines/ucto
Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --
Registry
-
Source
- Documentation
- JSON
purl: pkg:golang/github.com/%21language%21machines/ucto
Keywords:
computational-linguistics
, folia
, language
, natural-language-processing
, nlp
, punctuation
, tokeniser
License: GPL-3.0
Latest release: over 1 year ago
First release: over 9 years ago
Namespace: github.com/LanguageMachines
Stars: 69 on GitHub
Forks: 14 on GitHub
Total Commits: 1360
Committers: 8
Average commits per author: 170.0
Development Distribution Score (DDS): 0.491
More commit stats: commits.ecosyste.ms
See more repository details: repos.ecosyste.ms
Last synced: 28 days ago