lexer 54 parser 48 rust 43 nlp 34 analyzer 26 morphological 24 library 23 multilingual 21 parsing 18 scanner 17 japanese 14 bpe 13 dictionary 12 lexical 11 token 10 tokenization 10 text 10 lexer-generator 9 analysis 9 ai 8 no_std 8 python 7 machine-learning 6 generator 6 tantivy 6 builder 6 sql 6 cli 6 segmentation 5 llm 5 rust-lang 5 openai 4 wordpiece 4 ipadic 4 sentence 4 language 4 deep-learning 4 split 4 text-processing 4 regex 4 natural-language-processing 4 morphological-analysis 4 tokeniser 3 lex 3 alpino 3 parser-generator 3 stemmer 3 dutch 3 html 3 korean 3 compiler 3 chinese 3 unidic 3 segmenter 3 c 3 svg 3 gpt 3 transformer 3 nodejs 3 rust-crate 3 encoding 3 thai 3 word-segmentation 3 splitter 2 css 2 acceptor 2 fst 2 graph 2 thai-language 2 shortest-path 2 transducer 2 asr 2 automata 2 javascript 2 composition 2 finite-state-acceptors 2 tiktoken 2 indentation 2 processing 2 crate 2 huggingface 2 apple 2 hacktoberfest 2 pkl 2 virtual-machine 2 algorithm 2 lalr 2 sql-parser 2 lexing 2 sqlite 2 string 2 dfa 2 chatgpt 2 sentencepiece 2 word 2 blingfire 2 rust-wrapper 2 html5 2 whatwg 2 neologd 2