An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

pypi.org "tokenizer" keyword

msqlparse 0.4.4
A non-validating SQL parser.
2 versions - Latest release: over 4 years ago - 1 dependent repositories - 33 downloads last month - 4,021 stars on GitHub - 1 maintainer
Top 0.7% on pypi.org
sqlparse 0.6.1
A non-validating SQL parser.
40 versions - Latest release: 19 days ago - 429 dependent packages - 201,134 dependent repositories - 129 million downloads last month - 4,021 stars on GitHub - 1 maintainer
toknroll 0.4.0
Token-perfect LLM tokenization for Python: the model's own tokens, faster, with no dependencies.
2 versions - Latest release: about 4 hours ago - 93 downloads last month - 190 stars on GitHub - 1 maintainer
Top 4.4% on pypi.org
segments 2.4.0
Segmentation with orthography profiles
19 versions - Latest release: 9 months ago - 5 dependent packages - 696 dependent repositories - 1.4 million downloads last month - 42 stars on GitHub - 4 maintainers
tokenicer 0.0.21
A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventab...
20 versions - Latest release: 3 days ago - 31.1 thousand downloads last month - 12 stars on GitHub - 1 maintainer
text-chipper 1.0.3
Token-budgeted, Markdown-aware text chunking for RAG and LLM pipelines, with pluggable tokenizers...
9 versions - Latest release: about 7 hours ago - 1.43 thousand downloads last month - 0 stars on GitHub - 1 maintainer
riptoken 0.2.4
Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken
7 versions - Latest release: 7 months ago - 174 downloads last month - 6 stars on GitHub - 1 maintainer
ffwf-tau-tokens 0.16.0
Local tokenizers for τ: tiktoken encodings, Hugging Face tokenizers and chat templates, for exact...
1 version - Latest release: about 10 hours ago - 3 stars on GitHub - 1 maintainer
mayatok 2.1.5
High performance BPE tokenizer written in Rust with Python bindings
5 versions - Latest release: about 1 month ago - 392 downloads last month - 1 stars on GitHub - 1 maintainer
Top 0.6% on pypi.org
tokenizers 0.23.3
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
130 versions - Latest release: about 1 month ago - 380 dependent packages - 14,571 dependent repositories - 218 million downloads last month - 11,254 stars on GitHub - 4 maintainers
Top 2.9% on pypi.org
segtok 1.5.11
sentence segmentation and word tokenization tools
23 versions - Latest release: almost 5 years ago - 8 dependent packages - 353 dependent repositories - 398 thousand downloads last month - 171 stars on GitHub - 1 maintainer
divyanx-tokenizers 0.20.0.dev0
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
1 version - Latest release: over 2 years ago - 51 downloads last month - 11,254 stars on GitHub - 1 maintainer
tokenizers-gt 0.15.2.post0
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
3 versions - Latest release: almost 3 years ago - 690 downloads last month - 11,254 stars on GitHub - 1 maintainer
pytokens 0.4.1
A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
16 versions - Latest release: 10 months ago - 83.4 million downloads last month - 4 stars on GitHub - 1 maintainer
plane 0.2.1 💰
A lib for text preprocessing
20 versions - Latest release: almost 6 years ago - 3 dependent repositories - 308 downloads last month - 11 stars on GitHub - 1 maintainer
rishe 1.0.6
Rishe (ریشه): High-performance Persian NLP engine in Rust — tokenization, SBD, lemmatization, eza...
31 versions - Latest release: about 19 hours ago - 9.12 thousand downloads last month - 8 stars on GitHub - 1 maintainer
tokie 0.3.0 💰
Blazingly fast tokenizer — 50x faster, 10x smaller, 100% accurate
17 versions - Latest release: 9 days ago - 510 thousand downloads last month - 57 stars on GitHub - 1 maintainer
lindera-python 6.0.0 💰
Renamed to 'lindera'. This stub only depends on the new name.
27 versions - Latest release: about 2 months ago - 1.03 thousand downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-ipadic 2.2.0 💰
Python binding for Lindera with IPADIC dictionary
11 versions - Latest release: 9 months ago - 205 downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-cc-cedict 2.2.0 💰
Python binding for Lindera with CC-CEDICT Chinese dictionary
11 versions - Latest release: 9 months ago - 222 downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-ko-dic 2.2.0 💰
Python binding for Lindera with ko-dic Korean dictionary
11 versions - Latest release: 9 months ago - 77 downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-unidic 2.2.0 💰
Python binding for Lindera with UniDic dictionary
11 versions - Latest release: 9 months ago - 26 downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-jieba 2.2.0 💰
Python binding for Lindera with Jieba Chinese dictionary
1 version - Latest release: 8 months ago - 30 downloads last month - 707 stars on GitHub - 1 maintainer
lindera-python-cjk 2.2.0 💰
Python binding for Lindera with CJK dictionaries (IPADIC, ko-dic, CC-CEDICT)
11 versions - Latest release: 9 months ago - 69 downloads last month - 707 stars on GitHub - 1 maintainer
Top 1.6% on pypi.org
sacremoses 0.2.0
SacreMoses
53 versions - Latest release: 3 months ago - 120 dependent packages - 5,564 dependent repositories - 1.59 million downloads last month - 498 stars on GitHub - 3 maintainers
suzume 0.10.0
Lightweight Japanese tokenizer with POS tagging, lemmatization, and keyword extraction for Python
12 versions - Latest release: 1 day ago - 216 downloads last month - 9 stars on GitHub - 1 maintainer
glitchlings 1.0.1
Deterministic, high performance, principled natural language augmentation.
41 versions - Latest release: 1 day ago - 537 downloads last month - 41 stars on GitHub - 1 maintainer
Top 6.9% on pypi.org
text2text 1.9.5
Text2Text Language Modeling Toolkit
192 versions - Latest release: almost 2 years ago - 4 dependent repositories - 901 downloads last month - 305 stars on GitHub - 1 maintainer
quebra-frases 0.3.7 💰
quebra_frases chunks strings into byte sized pieces
15 versions - Latest release: over 5 years ago - 4 dependent packages - 2 dependent repositories - 49.6 thousand downloads last month - 1 stars on GitHub - 3 maintainers
anyseg 0.4.0
Fast segmentation of text into tokens and sentences, written in Rust with a Python API
5 versions - Latest release: 1 day ago - 6.42 thousand downloads last month - 0 stars on GitHub - 1 maintainer
Top 2.4% on pypi.org
natasha 1.6.0
Named-entity recognition for russian language
13 versions - Latest release: over 3 years ago - 6 dependent packages - 73 dependent repositories - 86.2 thousand downloads last month - 1,365 stars on GitHub - 2 maintainers
latexdetok 0.7.3
A tolerant LaTeX tokeniser: a .tex file read into a tree of groups, commands and text, which rewr...
20 versions - Latest release: 1 day ago - 485 downloads last month - 2 stars on GitHub - 1 maintainer
context-shrink-mcp 1.1.0
OpenAI-compatible context guard MCP server: estimate, trim and compact a chat history under its c...
2 versions - Latest release: 1 day ago - 134 downloads last month - 0 stars on GitHub - 1 maintainer
stepaudio2-minicpmo 0.1.1
StepAudio2: Audio tokenizer and TTS model
2 versions - Latest release: 10 months ago - 596 downloads last month - 1,530 stars on GitHub - 1 maintainer
Top 10.0% on pypi.org
token2wav-cosyvoice-stepaudio2 0.1.3
StepAudio2: Audio tokenizer and TTS model
2 versions - Latest release: 10 months ago - 41 downloads last month - 1,530 stars on GitHub - 1 maintainer
sacremoses-rs 0.1.0
Rust-accelerated drop-in classes for sacremoses (Moses tokenizer, detokenizer, punctuation normal...
1 version - Latest release: 2 days ago - 1 maintainer
langtoken-toolkit 0.2.0
A modular Python library for text tokenization.
2 versions - Latest release: 2 days ago - 1 maintainer
banglatok 0.2.4
Fast, lossless subword tokenizers for Bangla, English and Banglish (romanized Bengali), including...
5 versions - Latest release: 2 days ago - 851 downloads last month - 0 stars on GitHub - 1 maintainer
tuetoken 0.1.8
The fastest tokenizer for modern LLMs, up to 20x faster. Drop-in for transformers.AutoTokenizer, ...
8 versions - Latest release: 3 days ago - 1.35 thousand downloads last month - 7 stars on GitHub - 1 maintainer
space-wrap 0.0.3
Automated Spacy wrapper to turn plain text into Spacy doc objects
3 versions - Latest release: almost 4 years ago - 50 downloads last month - 1 maintainer
mlx-model-doctor 0.10.2 💰
Validate MLX model repositories before loading them.
21 versions - Latest release: 3 days ago - 449 downloads last month - 6 stars on GitHub - 1 maintainer
hindko-tokenizer 1.0.0
Hindko tokenizer (SentencePiece Unigram, 32,768 ids) with a small Python SDK: tokenize, encode, d...
1 version - Latest release: 3 days ago - 1 maintainer
extractionstring 0.8.2
Basic tools to tokenize (i.e. to construct atomic-entities/sub-strings of) a string, for Natural ...
1 version - Latest release: almost 4 years ago - 18 downloads last month - 0 stars on framagit.org - 1 maintainer
persian-tokenizer 1.0.0
Persian text tokenizer: normalization, ZWNJ fixing, punctuation/digit conversion, stop words, ste...
1 version - Latest release: 4 days ago - 1 maintainer
lectura-graphemiseur 4.3.9
Graphémiseur neural du français (IPA → orthographe) — P2G + POS + Morpho (BiLSTM multi-tâche)
12 versions - Latest release: 4 days ago - 48 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-diphone 2.1.0
Synthèse vocale du français par diphones WORLD (Lectura), en local — French diphone TTS with the ...
5 versions - Latest release: 4 days ago - 48 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-formules 3.7.11
Lecture algorithmique des formules (nombres, sigles, dates, téléphones, heures, monnaies, etc.) p...
20 versions - Latest release: 4 days ago - 63 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-g2p 4.1.5
Pipeline G2P complet du français — Tokeniseur + Formules + Phonémiseur + Groupes de lecture
19 versions - Latest release: 4 days ago - 58 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-p2g 4.6.5
Pipeline P2G complet du francais — Graphemiseur + Formules + Noms propres (IPA -> orthographe)
20 versions - Latest release: 4 days ago - 93 downloads last month - 1 stars on GitHub - 2 maintainers
lectura-aligneur 4.0.5
Aligneur graphème-phonème et syllabeur phonologique du français — alignement, syllabation, attaqu...
8 versions - Latest release: 4 days ago - 37 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-phonemiseur 4.1.11
Phonémiseur neural du français — G2P + POS + Morpho + Liaison + Groupes de lecture (BiLSTM multi-...
13 versions - Latest release: 4 days ago - 67 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-lexique 1.5.4
French lexical database access — lemmas, inflected forms with IPA phonetics and syllabification (...
9 versions - Latest release: 4 days ago - 61 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-tokeniseur 2.3.7
Normalisateur et tokeniseur complet pour le français — détection de formules (nombres, sigles, da...
12 versions - Latest release: 4 days ago - 101 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-vc-zeroshot 1.3.0
Conversion vocale zero-shot du français par OpenVoice (ONNX), en local, sons marqués par AudioSea...
7 versions - Latest release: 4 days ago - 18 downloads last month - 1 stars on GitHub - 1 maintainer
pinyintokenizer 0.0.3
Pinyin Tokenizer, chinese pinyin tokenizer
3 versions - Latest release: almost 2 years ago - 678 downloads last month - 31 stars on GitHub - 1 maintainer
vibrato 0.2.3
🎤 vibrato: VIterbi-Based acceleRAted TOkenizer
5 versions - Latest release: 6 months ago - 434 downloads last month - 46 stars on GitHub - 2 maintainers
bocconcini 4.3.1
Local LLM client for small usable windows. Zero runtime dependencies.
2 versions - Latest release: 4 days ago - 1 maintainer
Top 3.3% on pypi.org
nagisa 0.3.2
A Japanese tokenizer based on recurrent neural networks
29 versions - Latest release: 6 days ago - 5 dependent packages - 28 dependent repositories - 286 thousand downloads last month - 424 stars on GitHub - 1 maintainer
Top 7.3% on pypi.org
xerv-crayon 5.3.9
Omni-Backend Tokenizer - CPU (AVX2/512), CUDA (NVIDIA), ROCm (AMD) with automatic hardware detection
47 versions - Latest release: 5 months ago - 524 downloads last month - 4 stars on GitHub - 1 maintainer
crossandra 2.3.0 💰
A fast and simple enum/regex-based tokenizer with decent configurability
13 versions - Latest release: 7 months ago - 1 dependent package - 1 dependent repositories - 331 downloads last month - 9 stars on GitHub - 1 maintainer
semantic-text-splitter 0.33.3
Split text into semantic chunks, up to a desired chunk size. Supports calculating length by chara...
58 versions - Latest release: 4 days ago - 1 dependent package - 365 thousand downloads last month - 646 stars on GitHub - 1 maintainer
scanpars 0.0.0 removed
scanpars umbrella project
1 version - Latest release: about 4 years ago - 11 downloads last month - 0 stars on GitHub - 1 maintainer
iapp-ai 2.0.0
Python SDK for the iApp AI Marketplace APIs (https://iapp.co.th) — Thai OCR, eKYC, NLP, speech an...
6 versions - Latest release: 5 days ago - 1 dependent repositories - 63 downloads last month - 3 stars on GitHub - 1 maintainer
lectura-tts-dipho 1.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code
3 versions - Latest release: 5 months ago - 48 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-tts-mono 1.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code
3 versions - Latest release: 5 months ago - 44 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-monospeaker 4.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code
8 versions - Latest release: 5 months ago - 48 downloads last month - 1 stars on GitHub - 1 maintainer
toktok-rs 0.5.0
Fast, exact BPE tokenizer for OpenAI encodings — Rust core, tiktoken-identical ids
8 versions - Latest release: 5 days ago - 162 thousand downloads last month - 13 stars on GitHub - 1 maintainer
lectura-decodeur 3.1.0
Module retiré (génération v0 de Lectura) — withdrawn, no code
4 versions - Latest release: 4 months ago - 39 downloads last month - 1 stars on GitHub - 1 maintainer
iapp-nlp 0.1.0
Free, offline Thai word tokenizer from iApp Technology (https://iapp.co.th)
1 version - Latest release: 5 days ago - 3 stars on GitHub - 1 maintainer
lectura-vc-locuteurs 1.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code
3 versions - Latest release: 5 months ago - 19 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-tts-multi 1.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code
3 versions - Latest release: 5 months ago - 52 downloads last month - 1 stars on GitHub - 1 maintainer
lectura 4.0.1
Module retiré (génération v0 de Lectura) — withdrawn, no code
10 versions - Latest release: 5 months ago - 43 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-multispeaker 5.0.0
Module retiré (génération v0 de Lectura) — withdrawn, no code
16 versions - Latest release: 5 months ago - 112 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-stt 3.2.3
Module retiré (génération v0 de Lectura) — withdrawn, no code
15 versions - Latest release: 5 months ago - 64 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-correcteur 1.1.2
Module retiré (génération v0 de Lectura) — withdrawn, no code
6 versions - Latest release: 5 months ago - 43 downloads last month - 1 stars on GitHub - 1 maintainer
lectura-stt-formules 0.1.1
Module retiré (génération v0 de Lectura) — withdrawn, no code
3 versions - Latest release: 5 months ago - 20 downloads last month - 1 stars on GitHub - 1 maintainer
word-piece-tokenizer 1.0.1 💰
A Lightweight Word Piece Tokenizer
2 versions - Latest release: about 4 years ago - 656 downloads last month - 6 stars on GitHub - 1 maintainer
Top 5.5% on pypi.org
simplemma 2.0.0 💰
Fast and zero-dependency lemmatization, tokenization and sentence splitting for 54 languages.
20 versions - Latest release: 3 months ago - 6 dependent packages - 25 dependent repositories - 292 thousand downloads last month - 218 stars on GitHub - 1 maintainer
korrektor-py 1.0.0
Python wrapper for the https://korrektor.uz/api
1 version - Latest release: over 4 years ago - 8 downloads last month - 3 stars on GitHub - 1 maintainer
ai-model-explorer 1.0.10
Recover a Hugging Face model's architecture from its safetensors headers, without downloading the...
10 versions - Latest release: 6 days ago - 649 downloads last month - 0 stars on GitHub - 1 maintainer
mecab-ko-python 0.5.0
한국어 형태소 분석기 - MeCab-Ko Python 바인딩 (Korean Morphological Analyzer)
2 versions - Latest release: 8 months ago - 5.16 thousand downloads last month - 29 stars on GitHub - 1 maintainer
gigatoken 0.11.0 💰
Tokenize your documents at GB/s
12 versions - Latest release: 18 days ago - 295 thousand downloads last month - 4,155 stars on GitHub - 1 maintainer
sindhi-tokenizer 1.1.0
A SentencePiece Unigram tokenizer and lightweight Python SDK purpose-built for Sindhi.
1 version - Latest release: 6 days ago - 0 stars on GitHub - 1 maintainer
pyroid 1.0.1
Faster Rust versions of ftfy, fastText, Trafilatura, datatrove, json_repair and other popular Pyt...
13 versions - Latest release: 7 days ago - 1.59 thousand downloads last month - 4 stars on GitHub - 1 maintainer
langlock 0.1.3
Keep stray Chinese characters out of Korean LLM output at decode time: a byte-aware script mask f...
1 version - Latest release: 7 days ago - 1 maintainer
slash-tokens 0.0.2 💰
Token budget estimation in 4.8 KB WASM. Sub-millisecond. Zero dependencies.
2 versions - Latest release: 8 months ago - 62 downloads last month - 8 stars on GitHub - 1 maintainer
mecrab 0.1.2 💰
High-performance morphological analyzer compatible with MeCab
1 version - Latest release: 11 months ago - 23 downloads last month - 9 stars on GitHub - 1 maintainer
textmate-grammar-python 1.1.1 💰
A Python port of the vscode-textmate grammar tokenization engine.
16 versions - Latest release: 13 days ago - 1 dependent package - 3.04 thousand downloads last month - 13 stars on GitHub - 1 maintainer
deeplatent-nlp 0.3.15
High-performance Arabic-first tokenizer with morphology awareness
17 versions - Latest release: 10 months ago - 1.01 thousand downloads last month - 1 maintainer
Top 9.0% on pypi.org
cereja 2.6.0
Cereja is a bundle of useful functions that I don't want to rewrite.
153 versions - Latest release: 21 days ago - 3 dependent packages - 2 dependent repositories - 3.3 thousand downloads last month - 30 stars on GitHub - 1 maintainer
rectokens 0.2.2
A tokenizer library for sequential recommendation systems
2 versions - Latest release: 6 months ago - 54 downloads last month - 10 stars on GitHub - 1 maintainer
Top 3.6% on pypi.org
pyonmttok 1.38.1
Fast and customizable text tokenization library with BPE and SentencePiece support
68 versions - Latest release: 10 months ago - 3 dependent packages - 103 dependent repositories - 48.6 thousand downloads last month - 344 stars on GitHub - 4 maintainers
tokforge 0.1.1
Fast BPE tokenizer with a from-scratch byte-level core, verified against reference implementations
2 versions - Latest release: 8 days ago - 257 downloads last month - 0 stars on GitHub - 1 maintainer
specpair 0.1.0
Check whether a draft model can speculate for a target model (GGUF headers or HF tokenizers) and ...
1 version - Latest release: 8 days ago - 1 maintainer
Top 2.3% on pypi.org
hazm 0.12.1
Persian NLP Toolkit
18 versions - Latest release: 8 months ago - 8 dependent packages - 126 dependent repositories - 9.2 thousand downloads last month - 1,435 stars on GitHub - 1 maintainer
tokenwatt-verify 0.2.0
Check that an LLM API endpoint serves the model it names, honours the parameters it accepts and b...
3 versions - Latest release: 8 days ago - 0 downloads last month - 1 maintainer
jf-tokenize-package 1.0.3
A simple tokenizer function for NLP
4 versions - Latest release: over 4 years ago - 26 downloads last month - 0 stars on GitHub - 1 maintainer
vaporetto 0.3.3
Python wrapper of Vaporetto tokenizer
7 versions - Latest release: 6 months ago - 1 dependent repositories - 992 downloads last month - 22 stars on GitHub - 1 maintainer
unicodetokenizer 0.2.2
UnicodeTokenizer: tokenize all Unicode text
25 versions - Latest release: about 3 years ago - 102 downloads last month - 0 stars on GitHub - 1 maintainer
tokensim 1.0.5
SIMD optimized fast tokenizer, 97x faster than tiktoken, 4.5x faster than gigatoken
6 versions - Latest release: 9 days ago - 0 downloads last month - 1 maintainer