Data

Tools

Indexes

Applications

Experiments

Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

proxy.golang.org "tokenizing" keyword

Top 7.7% on proxy.golang.org
github.com/bzick/tokenizer v1.4.10
Package tokenizer provides a high performance generic tokenizer (lexer) that can parse any string...
11 versions - Latest release: about 1 year ago - 2 dependent packages - 1 dependent repositories - 89 stars on GitHub
Top 7.3% on proxy.golang.org
github.com/alasdairforsythe/tokenmonster v0.0.0-20231115032503-8d32435658a8
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
3 versions - Latest release: over 2 years ago - 600 stars on GitHub
Related Keywords
tokenizer 2 golang 1 lexer 1 parse 1 parser 1 text-tokenization 1 tokenisation 1 tokenization 1 tokenize 1 vocabulary 1 vocabulary-builder 1 vocabulary-generator 1