pypi.org "tokenizer" keyword
msqlparse 0.4.4
A non-validating SQL parser.2 versions - Latest release: over 4 years ago - 1 dependent repositories - 33 downloads last month - 4,021 stars on GitHub - 1 maintainer
Top 0.7% on pypi.org
40 versions - Latest release: 19 days ago - 429 dependent packages - 201,134 dependent repositories - 129 million downloads last month - 4,021 stars on GitHub - 1 maintainer
sqlparse 0.6.1
A non-validating SQL parser.40 versions - Latest release: 19 days ago - 429 dependent packages - 201,134 dependent repositories - 129 million downloads last month - 4,021 stars on GitHub - 1 maintainer
toknroll 0.4.0
Token-perfect LLM tokenization for Python: the model's own tokens, faster, with no dependencies.2 versions - Latest release: about 4 hours ago - 93 downloads last month - 190 stars on GitHub - 1 maintainer
Top 4.4% on pypi.org
19 versions - Latest release: 9 months ago - 5 dependent packages - 696 dependent repositories - 1.4 million downloads last month - 42 stars on GitHub - 4 maintainers
segments 2.4.0
Segmentation with orthography profiles19 versions - Latest release: 9 months ago - 5 dependent packages - 696 dependent repositories - 1.4 million downloads last month - 42 stars on GitHub - 4 maintainers
tokenicer 0.0.21
A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventab...20 versions - Latest release: 3 days ago - 31.1 thousand downloads last month - 12 stars on GitHub - 1 maintainer
text-chipper 1.0.3
Token-budgeted, Markdown-aware text chunking for RAG and LLM pipelines, with pluggable tokenizers...9 versions - Latest release: about 7 hours ago - 1.43 thousand downloads last month - 0 stars on GitHub - 1 maintainer
riptoken 0.2.4
Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken7 versions - Latest release: 7 months ago - 174 downloads last month - 6 stars on GitHub - 1 maintainer
ffwf-tau-tokens 0.16.0
Local tokenizers for τ: tiktoken encodings, Hugging Face tokenizers and chat templates, for exact...1 version - Latest release: about 10 hours ago - 3 stars on GitHub - 1 maintainer
mayatok 2.1.5
High performance BPE tokenizer written in Rust with Python bindings5 versions - Latest release: about 1 month ago - 392 downloads last month - 1 stars on GitHub - 1 maintainer
Top 0.6% on pypi.org
130 versions - Latest release: about 1 month ago - 380 dependent packages - 14,571 dependent repositories - 218 million downloads last month - 11,254 stars on GitHub - 4 maintainers
tokenizers 0.23.3
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production130 versions - Latest release: about 1 month ago - 380 dependent packages - 14,571 dependent repositories - 218 million downloads last month - 11,254 stars on GitHub - 4 maintainers
Top 2.9% on pypi.org
23 versions - Latest release: almost 5 years ago - 8 dependent packages - 353 dependent repositories - 398 thousand downloads last month - 171 stars on GitHub - 1 maintainer
segtok 1.5.11
sentence segmentation and word tokenization tools23 versions - Latest release: almost 5 years ago - 8 dependent packages - 353 dependent repositories - 398 thousand downloads last month - 171 stars on GitHub - 1 maintainer
divyanx-tokenizers 0.20.0.dev0
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production1 version - Latest release: over 2 years ago - 51 downloads last month - 11,254 stars on GitHub - 1 maintainer
tokenizers-gt 0.15.2.post0
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production3 versions - Latest release: almost 3 years ago - 690 downloads last month - 11,254 stars on GitHub - 1 maintainer
pytokens 0.4.1
A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.16 versions - Latest release: 10 months ago - 83.4 million downloads last month - 4 stars on GitHub - 1 maintainer
plane 0.2.1 💰
A lib for text preprocessing20 versions - Latest release: almost 6 years ago - 3 dependent repositories - 308 downloads last month - 11 stars on GitHub - 1 maintainer
rishe 1.0.6
Rishe (ریشه): High-performance Persian NLP engine in Rust — tokenization, SBD, lemmatization, eza...31 versions - Latest release: about 19 hours ago - 9.12 thousand downloads last month - 8 stars on GitHub - 1 maintainer
tokie 0.3.0 💰
Blazingly fast tokenizer — 50x faster, 10x smaller, 100% accurate17 versions - Latest release: 9 days ago - 510 thousand downloads last month - 57 stars on GitHub - 1 maintainer
lindera-python 6.0.0 💰
Renamed to 'lindera'. This stub only depends on the new name.27 versions - Latest release: about 2 months ago - 1.03 thousand downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-ipadic 2.2.0 💰
Python binding for Lindera with IPADIC dictionary11 versions - Latest release: 9 months ago - 205 downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-cc-cedict 2.2.0 💰
Python binding for Lindera with CC-CEDICT Chinese dictionary11 versions - Latest release: 9 months ago - 222 downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-ko-dic 2.2.0 💰
Python binding for Lindera with ko-dic Korean dictionary11 versions - Latest release: 9 months ago - 77 downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-unidic 2.2.0 💰
Python binding for Lindera with UniDic dictionary11 versions - Latest release: 9 months ago - 26 downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-jieba 2.2.0 💰
Python binding for Lindera with Jieba Chinese dictionary1 version - Latest release: 8 months ago - 30 downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-cjk 2.2.0 💰
Python binding for Lindera with CJK dictionaries (IPADIC, ko-dic, CC-CEDICT)11 versions - Latest release: 9 months ago - 69 downloads last month - 707 stars on GitHub - 1 maintainer
Top 1.6% on pypi.org
53 versions - Latest release: 3 months ago - 120 dependent packages - 5,564 dependent repositories - 1.59 million downloads last month - 498 stars on GitHub - 3 maintainers
sacremoses 0.2.0
SacreMoses53 versions - Latest release: 3 months ago - 120 dependent packages - 5,564 dependent repositories - 1.59 million downloads last month - 498 stars on GitHub - 3 maintainers
suzume 0.10.0
Lightweight Japanese tokenizer with POS tagging, lemmatization, and keyword extraction for Python12 versions - Latest release: 1 day ago - 216 downloads last month - 9 stars on GitHub - 1 maintainer
glitchlings 1.0.1
Deterministic, high performance, principled natural language augmentation.41 versions - Latest release: 1 day ago - 537 downloads last month - 41 stars on GitHub - 1 maintainer
Top 6.9% on pypi.org
192 versions - Latest release: almost 2 years ago - 4 dependent repositories - 901 downloads last month - 305 stars on GitHub - 1 maintainer
text2text 1.9.5
Text2Text Language Modeling Toolkit192 versions - Latest release: almost 2 years ago - 4 dependent repositories - 901 downloads last month - 305 stars on GitHub - 1 maintainer
quebra-frases 0.3.7 💰
quebra_frases chunks strings into byte sized pieces15 versions - Latest release: over 5 years ago - 4 dependent packages - 2 dependent repositories - 49.6 thousand downloads last month - 1 stars on GitHub - 3 maintainers
anyseg 0.4.0
Fast segmentation of text into tokens and sentences, written in Rust with a Python API5 versions - Latest release: 1 day ago - 6.42 thousand downloads last month - 0 stars on GitHub - 1 maintainer
Top 2.4% on pypi.org
13 versions - Latest release: over 3 years ago - 6 dependent packages - 73 dependent repositories - 86.2 thousand downloads last month - 1,365 stars on GitHub - 2 maintainers
natasha 1.6.0
Named-entity recognition for russian language13 versions - Latest release: over 3 years ago - 6 dependent packages - 73 dependent repositories - 86.2 thousand downloads last month - 1,365 stars on GitHub - 2 maintainers
latexdetok 0.7.3
A tolerant LaTeX tokeniser: a .tex file read into a tree of groups, commands and text, which rewr...20 versions - Latest release: 1 day ago - 485 downloads last month - 2 stars on GitHub - 1 maintainer
context-shrink-mcp 1.1.0
OpenAI-compatible context guard MCP server: estimate, trim and compact a chat history under its c...2 versions - Latest release: 1 day ago - 134 downloads last month - 0 stars on GitHub - 1 maintainer
stepaudio2-minicpmo 0.1.1
StepAudio2: Audio tokenizer and TTS model2 versions - Latest release: 10 months ago - 596 downloads last month - 1,530 stars on GitHub - 1 maintainer
Top 10.0% on pypi.org
2 versions - Latest release: 10 months ago - 41 downloads last month - 1,530 stars on GitHub - 1 maintainer
token2wav-cosyvoice-stepaudio2 0.1.3
StepAudio2: Audio tokenizer and TTS model2 versions - Latest release: 10 months ago - 41 downloads last month - 1,530 stars on GitHub - 1 maintainer
sacremoses-rs 0.1.0
Rust-accelerated drop-in classes for sacremoses (Moses tokenizer, detokenizer, punctuation normal...1 version - Latest release: 2 days ago - 1 maintainer
langtoken-toolkit 0.2.0
A modular Python library for text tokenization.2 versions - Latest release: 2 days ago - 1 maintainer
banglatok 0.2.4
Fast, lossless subword tokenizers for Bangla, English and Banglish (romanized Bengali), including...5 versions - Latest release: 2 days ago - 851 downloads last month - 0 stars on GitHub - 1 maintainer
tuetoken 0.1.8
The fastest tokenizer for modern LLMs, up to 20x faster. Drop-in for transformers.AutoTokenizer, ...8 versions - Latest release: 3 days ago - 1.35 thousand downloads last month - 7 stars on GitHub - 1 maintainer
space-wrap 0.0.3
Automated Spacy wrapper to turn plain text into Spacy doc objects3 versions - Latest release: almost 4 years ago - 50 downloads last month - 1 maintainer
mlx-model-doctor 0.10.2 💰
Validate MLX model repositories before loading them.21 versions - Latest release: 3 days ago - 449 downloads last month - 6 stars on GitHub - 1 maintainer
hindko-tokenizer 1.0.0
Hindko tokenizer (SentencePiece Unigram, 32,768 ids) with a small Python SDK: tokenize, encode, d...1 version - Latest release: 3 days ago - 1 maintainer
extractionstring 0.8.2
Basic tools to tokenize (i.e. to construct atomic-entities/sub-strings of) a string, for Natural ...1 version - Latest release: almost 4 years ago - 18 downloads last month - 0 stars on framagit.org - 1 maintainer
persian-tokenizer 1.0.0
Persian text tokenizer: normalization, ZWNJ fixing, punctuation/digit conversion, stop words, ste...1 version - Latest release: 4 days ago - 1 maintainer
lectura-graphemiseur 4.3.9
Graphémiseur neural du français (IPA → orthographe) — P2G + POS + Morpho (BiLSTM multi-tâche)12 versions - Latest release: 4 days ago - 48 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-diphone 2.1.0
Synthèse vocale du français par diphones WORLD (Lectura), en local — French diphone TTS with the ...5 versions - Latest release: 4 days ago - 48 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-formules 3.7.11
Lecture algorithmique des formules (nombres, sigles, dates, téléphones, heures, monnaies, etc.) p...20 versions - Latest release: 4 days ago - 63 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-g2p 4.1.5
Pipeline G2P complet du français — Tokeniseur + Formules + Phonémiseur + Groupes de lecture19 versions - Latest release: 4 days ago - 58 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-p2g 4.6.5
Pipeline P2G complet du francais — Graphemiseur + Formules + Noms propres (IPA -> orthographe)20 versions - Latest release: 4 days ago - 93 downloads last month - 1 stars on GitHub - 2 maintainers
lectura-aligneur 4.0.5
Aligneur graphème-phonème et syllabeur phonologique du français — alignement, syllabation, attaqu...8 versions - Latest release: 4 days ago - 37 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-phonemiseur 4.1.11
Phonémiseur neural du français — G2P + POS + Morpho + Liaison + Groupes de lecture (BiLSTM multi-...13 versions - Latest release: 4 days ago - 67 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-lexique 1.5.4
French lexical database access — lemmas, inflected forms with IPA phonetics and syllabification (...9 versions - Latest release: 4 days ago - 61 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-tokeniseur 2.3.7
Normalisateur et tokeniseur complet pour le français — détection de formules (nombres, sigles, da...12 versions - Latest release: 4 days ago - 101 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-vc-zeroshot 1.3.0
Conversion vocale zero-shot du français par OpenVoice (ONNX), en local, sons marqués par AudioSea...7 versions - Latest release: 4 days ago - 18 downloads last month - 1 stars on GitHub - 1 maintainer
pinyintokenizer 0.0.3
Pinyin Tokenizer, chinese pinyin tokenizer3 versions - Latest release: almost 2 years ago - 678 downloads last month - 31 stars on GitHub - 1 maintainer
vibrato 0.2.3
🎤 vibrato: VIterbi-Based acceleRAted TOkenizer5 versions - Latest release: 6 months ago - 434 downloads last month - 46 stars on GitHub - 2 maintainers
bocconcini 4.3.1
Local LLM client for small usable windows. Zero runtime dependencies.2 versions - Latest release: 4 days ago - 1 maintainer
Top 3.3% on pypi.org
29 versions - Latest release: 6 days ago - 5 dependent packages - 28 dependent repositories - 286 thousand downloads last month - 424 stars on GitHub - 1 maintainer
nagisa 0.3.2
A Japanese tokenizer based on recurrent neural networks29 versions - Latest release: 6 days ago - 5 dependent packages - 28 dependent repositories - 286 thousand downloads last month - 424 stars on GitHub - 1 maintainer
Top 7.3% on pypi.org
47 versions - Latest release: 5 months ago - 524 downloads last month - 4 stars on GitHub - 1 maintainer
xerv-crayon 5.3.9
Omni-Backend Tokenizer - CPU (AVX2/512), CUDA (NVIDIA), ROCm (AMD) with automatic hardware detection47 versions - Latest release: 5 months ago - 524 downloads last month - 4 stars on GitHub - 1 maintainer
crossandra 2.3.0 💰
A fast and simple enum/regex-based tokenizer with decent configurability13 versions - Latest release: 7 months ago - 1 dependent package - 1 dependent repositories - 331 downloads last month - 9 stars on GitHub - 1 maintainer
semantic-text-splitter 0.33.3
Split text into semantic chunks, up to a desired chunk size. Supports calculating length by chara...58 versions - Latest release: 4 days ago - 1 dependent package - 365 thousand downloads last month - 646 stars on GitHub - 1 maintainer
scanpars 0.0.0 removed
scanpars umbrella project1 version - Latest release: about 4 years ago - 11 downloads last month - 0 stars on GitHub - 1 maintainer
iapp-ai 2.0.0
Python SDK for the iApp AI Marketplace APIs (https://iapp.co.th) — Thai OCR, eKYC, NLP, speech an...6 versions - Latest release: 5 days ago - 1 dependent repositories - 63 downloads last month - 3 stars on GitHub - 1 maintainer
lectura-tts-dipho 1.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code3 versions - Latest release: 5 months ago - 48 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-tts-mono 1.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code3 versions - Latest release: 5 months ago - 44 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-monospeaker 4.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code8 versions - Latest release: 5 months ago - 48 downloads last month - 1 stars on GitHub - 1 maintainer
toktok-rs 0.5.0
Fast, exact BPE tokenizer for OpenAI encodings — Rust core, tiktoken-identical ids8 versions - Latest release: 5 days ago - 162 thousand downloads last month - 13 stars on GitHub - 1 maintainer
lectura-decodeur 3.1.0
Module retiré (génération v0 de Lectura) — withdrawn, no code4 versions - Latest release: 4 months ago - 39 downloads last month - 1 stars on GitHub - 1 maintainer
iapp-nlp 0.1.0
Free, offline Thai word tokenizer from iApp Technology (https://iapp.co.th)1 version - Latest release: 5 days ago - 3 stars on GitHub - 1 maintainer
lectura-vc-locuteurs 1.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code3 versions - Latest release: 5 months ago - 19 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-tts-multi 1.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code3 versions - Latest release: 5 months ago - 52 downloads last month - 1 stars on GitHub - 1 maintainer
lectura 4.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code10 versions - Latest release: 5 months ago - 43 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-multispeaker 5.0.0
Module retiré (génération v0 de Lectura) — withdrawn, no code16 versions - Latest release: 5 months ago - 112 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-stt 3.2.3
Module retiré (génération v0 de Lectura) — withdrawn, no code15 versions - Latest release: 5 months ago - 64 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-correcteur 1.1.2
Module retiré (génération v0 de Lectura) — withdrawn, no code6 versions - Latest release: 5 months ago - 43 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-stt-formules 0.1.1
Module retiré (génération v0 de Lectura) — withdrawn, no code3 versions - Latest release: 5 months ago - 20 downloads last month - 1 stars on GitHub - 1 maintainer
word-piece-tokenizer 1.0.1 💰
A Lightweight Word Piece Tokenizer2 versions - Latest release: about 4 years ago - 656 downloads last month - 6 stars on GitHub - 1 maintainer
Top 5.5% on pypi.org
20 versions - Latest release: 3 months ago - 6 dependent packages - 25 dependent repositories - 292 thousand downloads last month - 218 stars on GitHub - 1 maintainer
simplemma 2.0.0 💰
Fast and zero-dependency lemmatization, tokenization and sentence splitting for 54 languages.20 versions - Latest release: 3 months ago - 6 dependent packages - 25 dependent repositories - 292 thousand downloads last month - 218 stars on GitHub - 1 maintainer
korrektor-py 1.0.0
Python wrapper for the https://korrektor.uz/api1 version - Latest release: over 4 years ago - 8 downloads last month - 3 stars on GitHub - 1 maintainer
ai-model-explorer 1.0.10
Recover a Hugging Face model's architecture from its safetensors headers, without downloading the...10 versions - Latest release: 6 days ago - 649 downloads last month - 0 stars on GitHub - 1 maintainer
mecab-ko-python 0.5.0
한국어 형태소 분석기 - MeCab-Ko Python 바인딩 (Korean Morphological Analyzer)2 versions - Latest release: 8 months ago - 5.16 thousand downloads last month - 29 stars on GitHub - 1 maintainer
gigatoken 0.11.0 💰
Tokenize your documents at GB/s12 versions - Latest release: 18 days ago - 295 thousand downloads last month - 4,155 stars on GitHub - 1 maintainer
sindhi-tokenizer 1.1.0
A SentencePiece Unigram tokenizer and lightweight Python SDK purpose-built for Sindhi.1 version - Latest release: 6 days ago - 0 stars on GitHub - 1 maintainer
pyroid 1.0.1
Faster Rust versions of ftfy, fastText, Trafilatura, datatrove, json_repair and other popular Pyt...13 versions - Latest release: 7 days ago - 1.59 thousand downloads last month - 4 stars on GitHub - 1 maintainer
langlock 0.1.3
Keep stray Chinese characters out of Korean LLM output at decode time: a byte-aware script mask f...1 version - Latest release: 7 days ago - 1 maintainer
slash-tokens 0.0.2 💰
Token budget estimation in 4.8 KB WASM. Sub-millisecond. Zero dependencies.2 versions - Latest release: 8 months ago - 62 downloads last month - 8 stars on GitHub - 1 maintainer
mecrab 0.1.2 💰
High-performance morphological analyzer compatible with MeCab1 version - Latest release: 11 months ago - 23 downloads last month - 9 stars on GitHub - 1 maintainer
textmate-grammar-python 1.1.1 💰
A Python port of the vscode-textmate grammar tokenization engine.16 versions - Latest release: 13 days ago - 1 dependent package - 3.04 thousand downloads last month - 13 stars on GitHub - 1 maintainer
deeplatent-nlp 0.3.15
High-performance Arabic-first tokenizer with morphology awareness17 versions - Latest release: 10 months ago - 1.01 thousand downloads last month - 1 maintainer
Top 9.0% on pypi.org
153 versions - Latest release: 21 days ago - 3 dependent packages - 2 dependent repositories - 3.3 thousand downloads last month - 30 stars on GitHub - 1 maintainer
cereja 2.6.0
Cereja is a bundle of useful functions that I don't want to rewrite.153 versions - Latest release: 21 days ago - 3 dependent packages - 2 dependent repositories - 3.3 thousand downloads last month - 30 stars on GitHub - 1 maintainer
rectokens 0.2.2
A tokenizer library for sequential recommendation systems2 versions - Latest release: 6 months ago - 54 downloads last month - 10 stars on GitHub - 1 maintainer
Top 3.6% on pypi.org
68 versions - Latest release: 10 months ago - 3 dependent packages - 103 dependent repositories - 48.6 thousand downloads last month - 344 stars on GitHub - 4 maintainers
pyonmttok 1.38.1
Fast and customizable text tokenization library with BPE and SentencePiece support68 versions - Latest release: 10 months ago - 3 dependent packages - 103 dependent repositories - 48.6 thousand downloads last month - 344 stars on GitHub - 4 maintainers
tokforge 0.1.1
Fast BPE tokenizer with a from-scratch byte-level core, verified against reference implementations2 versions - Latest release: 8 days ago - 257 downloads last month - 0 stars on GitHub - 1 maintainer
specpair 0.1.0
Check whether a draft model can speculate for a target model (GGUF headers or HF tokenizers) and ...1 version - Latest release: 8 days ago - 1 maintainer
Top 2.3% on pypi.org
18 versions - Latest release: 8 months ago - 8 dependent packages - 126 dependent repositories - 9.2 thousand downloads last month - 1,435 stars on GitHub - 1 maintainer
hazm 0.12.1
Persian NLP Toolkit18 versions - Latest release: 8 months ago - 8 dependent packages - 126 dependent repositories - 9.2 thousand downloads last month - 1,435 stars on GitHub - 1 maintainer
tokenwatt-verify 0.2.0
Check that an LLM API endpoint serves the model it names, honours the parameters it accepts and b...3 versions - Latest release: 8 days ago - 0 downloads last month - 1 maintainer
jf-tokenize-package 1.0.3
A simple tokenizer function for NLP4 versions - Latest release: over 4 years ago - 26 downloads last month - 0 stars on GitHub - 1 maintainer
vaporetto 0.3.3
Python wrapper of Vaporetto tokenizer7 versions - Latest release: 6 months ago - 1 dependent repositories - 992 downloads last month - 22 stars on GitHub - 1 maintainer
unicodetokenizer 0.2.2
UnicodeTokenizer: tokenize all Unicode text25 versions - Latest release: about 3 years ago - 102 downloads last month - 0 stars on GitHub - 1 maintainer
tokensim 1.0.5
SIMD optimized fast tokenizer, 97x faster than tiktoken, 4.5x faster than gigatoken6 versions - Latest release: 9 days ago - 0 downloads last month - 1 maintainer
Related Keywords
nlp
221
llm
106
python
103
bpe
72
tokenization
64
natural-language-processing
38
tiktoken
36
rust
34
machine-learning
33
NLP
33
text-processing
31
openai
31
ai
30
tts
27
huggingface
27
transformers
26
gpt
25
phonetics
24
morphology
24
speech-recognition
23
g2p
22
grapheme-to-phoneme
21
french
21
ipa
21
onnnx
21
speech-synthesis
21
spell-checker
21
syllabification
21
text-to-speach
21
voice-conversion
21
token
21
multilingual
20
tokens
20
parser
18
sentencepiece
16
language-model
16
cli
16
japanese
16
segmentation
15
lexer
15
bert
15
transformer
14
embeddings
14
rag
14
claude
13
pytorch
13
deep-learning
12
text
12
byte-pair-encoding
12
analyzer
12
korean
11
language
11
nlp-library
11
python3
10
anthropic
10
llama
10
wordpiece
10
word-segmentation
10
gemini
9
regex
9
analysis
9
pos-tagging
9
pyo3
9
mecab
9
morphological
9
context-window
9
Tokenizer
8
sentence
8
library
8
bpe-tokenizer
8
ner
7
tokeniser
7
thai
7
unicode
7
large-language-models
7
lemmatization
7
spacy
7
BPE
7
linguistics
7
natural language processing
7
prompt-engineering
7
parsing
6
morphological-analysis
6
chunking
6
tokenizers
6
tokenisation
6
cpp
6
benchmark
6
unigram
6
chatgpt
6
retrieval
6
LLM
6
tensorflow
6
ast
6
lemmatizer
6
natural
6
preprocessing
6
normalization
5
developer-tools
5
processing
5