Top 0.4% dependent packages on proxy.golang.org
Top 0.3% dependent repos on proxy.golang.org
Top 3.9% forks on proxy.golang.org
Top 1.0% docker downloads on proxy.golang.org
proxy.golang.org : github.com/neurosnap/sentences
Package sentences is a golang package that will convert a blob of text into a list of sentences. This package attempts to support a multitude of languages: Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Italian, Norwegian, Polish, Portuguese, Slovene, Spanish, Swedish, and Turkish. An unsupervised multilingual sentence boundary detection library for golang. The goal of this library is to be able to break up any text into a list of sentences in multiple languages. The way the punkt system accomplishes this goal is through training the tokenizer with text in that given language. Once the likelihoods of abbreviations, collocations, and sentence starters are determined, finding sentence boundaries becomes easier. There are many problems that arise when tokenizing text into sentences, the primary issue being abbreviations. The punkt system attempts to determine whether a word is an abbreviation, an end to a sentence, or even both through training the system with text in the given language. The punkt system incorporates both token- and type-based analysis on the text through two different phases of annotation. Original research article: http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=BAE5C34E5C3B9DC60DFC4D93B85D8BB1?doi=10.1.1.85.5017&rep=rep1&type=pdf
Registry
-
Source
- Documentation
- JSON
purl: pkg:golang/github.com/neurosnap/sentences
Keywords:
cli
, sentence-tokenizer
, sentences
, tokenizer
License: MIT
Latest release: over 2 years ago
First release: over 9 years ago
Namespace: github.com/neurosnap
Dependent packages: 143
Dependent repositories: 392
Stars: 355 on GitHub
Forks: 35 on GitHub
Docker dependents: 11
Docker downloads: 3,440
See more repository details: repos.ecosyste.ms
Last synced: about 19 hours ago