lexer 56 parser 51 rust 46 nlp 35 analyzer 26 morphological 24 library 23 multilingual 21 parsing 19 scanner 17 bpe 14 japanese 14 dictionary 12 lexical 11 token 11 tokenization 11 text 10 ai 10 analysis 9 lexer-generator 9 no_std 8 python 7 builder 6 machine-learning 6 sql 6 cli 6 tantivy 6 generator 6 openai 5 segmentation 5 rust-lang 5 llm 5 gpt 4 natural-language-processing 4 deep-learning 4 text-processing 4 split 4 wordpiece 4 language 4 morphological-analysis 4 sentence 4 regex 4 chinese 4 ipadic 4 parser-generator 4 tokeniser 3 dutch 3 alpino 3 thai 3 word-segmentation 3 nodejs 3 html 3 stemmer 3 lex 3 svg 3 encoding 3 segmenter 3 sqlite 3 rust-crate 3 transformer 3 korean 3 unidic 3 c 3 lalr 3 compiler 3 sql-parser 3 sentencepiece 2 transducers 2 blingfire 2 rust-wrapper 2 wfst 2 word 2 extension 2 javascript 2 fst 2 transducer 2 graph 2 crate 2 speech-recognition 2 openfst 2 kaldi-asr 2 kaldi 2 fsts 2 lexing 2 finite-state-transducers 2 finite-state-acceptors 2 composition 2 automata 2 asr 2 acceptor 2 shortest-path 2 huggingface 2 language-model 2 virtual-machine 2 dfa 2 hacktoberfest 2 xml 2 css 2 ko-dic 2 thai-language 2