Ecosyste.ms: Packages
An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.
crates.io "tokenizer" keyword
parsit 0.2.0
very simple lib, the parsing combinators, recursive descendent that uses logos as lexer17 versions - Latest release: 10 months ago - 1 dependent package - 1 dependent repositories - 5.28 thousand downloads total - 7 stars on GitHub - 1 maintainer
luther-derive 0.1.0
The proc macro generator for the Luther lexer generator.1 version - Latest release: almost 6 years ago - 1.3 thousand downloads total - 5 stars on GitHub - 1 maintainer
cang-jie 0.18.0
A Chinese tokenizer for tantivy20 versions - Latest release: 7 months ago - 6 dependent packages - 13 dependent repositories - 25 thousand downloads total - 68 stars on GitHub - 1 maintainer
c_lexer 0.1.1
C lexer2 versions - Latest release: about 5 years ago - 1 dependent package - 1 dependent repositories - 2.22 thousand downloads total - 6 stars on GitHub - 1 maintainer
bytepiece 0.2.0
Rust version of bytepiece tokenizer2 versions - Latest release: 8 months ago - 527 downloads total - 9 stars on GitHub - 1 maintainer
lindera-tantivy 0.27.1 💰
Lindera Tokenizer for Tantivy.40 versions - Latest release: 6 months ago - 5 dependent packages - 7 dependent repositories - 21.7 thousand downloads total - 46 stars on GitHub - 4 maintainers
tantivy-stemmers 0.2.0
A collection of Tantivy stemmer tokenizers2 versions - Latest release: 19 days ago - 185 downloads total - 0 stars on GitHub - 2 maintainers
logos2 💰
Create ridiculously fast Lexers6 versions - Latest release: 19 days ago - 1.51 thousand downloads total - 2,632 stars on GitHub - 1 maintainer
logos-cli2 💰
Create ridiculously fast Lexers6 versions - Latest release: 19 days ago - 1.5 thousand downloads total - 2,632 stars on GitHub - 1 maintainer
logos-cli 0.14.0 💰
Create ridiculously fast Lexers2 versions - Latest release: 4 months ago - 608 downloads total - 2,632 stars on GitHub - 2 maintainers
tantivy-czech-stemmer 0.2.1
Czech stemmer as Tantivy tokenizer2 versions - Latest release: 21 days ago - 281 downloads total - 0 stars on GitHub - 2 maintainers
pkl_fast 0.1.1
A library aiming to easily and efficiently work with Apple's PKL format.2 versions - Latest release: 3 months ago - 620 downloads total - 3 stars on GitHub - 1 maintainer
luther 0.1.0
The runtime components of the Luther lexer generator.1 version - Latest release: almost 6 years ago - 1 dependent package - 1 dependent repositories - 1.84 thousand downloads total - 5 stars on GitHub - 1 maintainer
rust_transformers 0.2.0
High performance tokenizers for Rust2 versions - Latest release: over 4 years ago - 1 dependent package - 1.01 thousand downloads total - 270 stars on GitHub - 1 maintainer
fileql 0.3.0 💰
A tool to run SQL-like query on local files using GitQL SDK3 versions - Latest release: 23 days ago - 919 downloads total - 55 stars on GitHub - 1 maintainer
xxcalc 0.2.1
Embeddable or standalone robust floating-point polynomial calculator4 versions - Latest release: over 7 years ago - 1 dependent repositories - 8.17 thousand downloads total - 13 stars on GitHub - 1 maintainer
html5tokenizer 0.5.2
An HTML5 tokenizer with code span support.7 versions - Latest release: 8 months ago - 1 dependent repositories - 2.2 thousand downloads total - 1 maintainer
tele_tokenizer 0.2.0
A CSS tokenizer2 versions - Latest release: about 2 years ago - 3 dependent packages - 1 dependent repositories - 1.69 thousand downloads total - 199 stars on GitHub - 1 maintainer
bytepiece_rs 0.2.2
The Bytepiece Tokenizer Implemented in Rust7 versions - Latest release: 6 months ago - 1 dependent package - 2.07 thousand downloads total - 14 stars on GitHub - 1 maintainer
svgparser 0.8.1
Featureful, pull-based, zero-allocation SVG parser.21 versions - Latest release: about 6 years ago - 4 dependent packages - 98 dependent repositories - 101 thousand downloads total - 22 stars on GitHub - 1 maintainer
tusk_lexer 0.4.7
The lexical analysis component of Tusk.21 versions - Latest release: almost 3 years ago - 1 dependent package - 6.9 thousand downloads total - 1 maintainer
lexical_scanner 0.1.18
A simple lexer which creates over 115+ various tokens based on the rust programming language. Thi...19 versions - Latest release: about 2 years ago - 7.25 thousand downloads total - 2 stars on GitHub - 1 maintainer
rye-grain 0.0.1
A Python to Rust translator1 version - Latest release: over 1 year ago - 401 downloads total - 1 stars on GitHub - 1 maintainer
vibrato 0.5.1
Vibrato: viterbi-based accelerated tokenizer11 versions - Latest release: about 1 year ago - 1 dependent package - 1 dependent repositories - 11.6 thousand downloads total - 292 stars on GitHub - 2 maintainers
azul-simplecss 0.1.1
A very simple CSS 2.1 tokenizer.2 versions - Latest release: almost 5 years ago - 1 dependent package - 4 dependent repositories - 17.3 thousand downloads total - 29 stars on GitHub - 1 maintainer
Related Keywords
lexer
37
parser
37
rust
33
analyzer
23
morphological
22
library
20
nlp
20
multilingual
19
parsing
17
japanese
12
dictionary
10
lexical
10
scanner
10
analysis
9
lexer-generator
9
no_std
9
tokenization
8
token
7
text
6
tantivy
6
generator
6
python
6
machine-learning
5
builder
5
rust-lang
4
morphological-analysis
4
sql
4
segmentation
4
bpe
4
deep-learning
4
natural-language-processing
4
ipadic
4
ai
4
openai
4
gpt
3
lex
3
dutch
3
svg
3
regex
3
html
3
text-processing
3
alpino
3
sentence
3
cli
3
rust-crate
3
language
3
thai
3
stemmer
3
chinese
3
css
2
graph
2
neologd
2
dfa
2
transformer
2
cc-cedict
2
unidic
2
ko-dic
2
korean
2
xml
2
thai-language
2
c
2
indentation
2
nodejs
2
hacktoberfest
2
word-segmentation
2
html5
2
whatwg
2
wfst
2
processing
2
transducers
2
speech-recognition
2
shortest-path
2
openfst
2
kaldi-asr
2
kaldi
2
fsts
2
finite-state-transducers
2
finite-state-acceptors
2
composition
2
automata
2
asr
2
transducer
2
acceptor
2
fst
2
chatgpt
2
rust-wrapper
2
language-model
2
javascript
2
string
2
parser-generator
2
splitter
2
sqlite
2
blingfire
2
split
2
chess
1
pgn
1
tiktoken
1
gpt-4
1
json
1
bindings
1