crates.io "tokenization" keyword
vtext 0.2.0
NLP with Rust4 versions - Latest release: over 6 years ago - 3 dependent repositories - 15.8 thousand downloads total - 153 stars on GitHub - 1 maintainer
vaporetto 0.6.5
Vaporetto: a pointwise prediction based tokenizer18 versions - Latest release: over 1 year ago - 3 dependent packages - 1 dependent repositories - 281 thousand downloads total - 299 stars on GitHub - 1 maintainer
vaporetto_tantivy 0.24.0
Vaporetto Tokenizer for Tantivy15 versions - Latest release: over 1 year ago - 37 thousand downloads total - 299 stars on GitHub - 1 maintainer
vaporetto_rules 0.6.5
Rule-base filters for Vaporetto12 versions - Latest release: over 1 year ago - 1 dependent package - 1 dependent repositories - 76.9 thousand downloads total - 299 stars on GitHub - 1 maintainer
gaze-mcp-core 0.15.1
Transport-free MCP-shaped chokepoint runtime for Gaze. Enforces redact→manifest→return ordering a...20 versions - Latest release: 4 days ago - 855 downloads total - 12 stars on GitHub - 1 maintainer
moxy-build 0.4.1
Build utilities used by moxy9 versions - Latest release: 4 days ago - 283 downloads total - 14 stars on GitHub - 1 maintainer
moxy-token 0.4.1
Rust token types and lexing9 versions - Latest release: 4 days ago - 266 downloads total - 14 stars on GitHub - 1 maintainer
moxy-ast 0.4.1
Rust AST used by moxy9 versions - Latest release: 4 days ago - 229 downloads total - 14 stars on GitHub - 1 maintainer
moxy-template 0.4.0
Rust quasi-quoting and token templating with explicit interpolation syntax and built-in template ...8 versions - Latest release: 4 days ago - 150 downloads total - 14 stars on GitHub - 1 maintainer
moxy 0.4.0
A modular Rust syntax toolkit for tokenization, parsing, formatting, diagnostics, and quasi-quoting.12 versions - Latest release: 4 days ago - 176 downloads total - 14 stars on GitHub - 1 maintainer
moxy-diagnostic 0.4.0
Rust macro diagnostics for moxy8 versions - Latest release: 4 days ago - 146 downloads total - 14 stars on GitHub - 1 maintainer
moxy-derive 0.4.0
Moxy derive macros13 versions - Latest release: 4 days ago - 303 downloads total - 14 stars on GitHub - 1 maintainer
moxy-fmt 0.4.0
Rust pretty printing (formatting) for moxy8 versions - Latest release: 4 days ago - 150 downloads total - 14 stars on GitHub - 1 maintainer
yang 0.0.0 removed
placeholder1 version - Latest release: 7 months ago - 23 downloads total - 14 stars on GitHub - 1 maintainer
libtqsm 0.6.1
Sentence segmenter that supports ~300 languages1 version - Latest release: over 2 years ago - 1 dependent package - 19.1 thousand downloads total - 3 stars on GitHub - 1 maintainer
spars-nlp 0.1.0
Experimental native Rust inference for official spaCy models1 version - Latest release: 7 days ago - 0 downloads total - 0 stars on GitHub - 1 maintainer
text-scanner 0.0.3
A UTF-8 char-oriented, zero-copy, text and code scanning library3 versions - Latest release: about 3 years ago - 1 dependent package - 8.91 thousand downloads total - 0 stars on GitHub - 1 maintainer
toktier 0.2.9
Rust API preview for TokTier's exact, stateful tokenization runtime9 versions - Latest release: 29 days ago - 152 downloads total - 6 stars on GitHub - 1 maintainer
perl-token 0.17.0 💰
Token definitions for Perl parser12 versions - Latest release: 3 months ago - 6.57 thousand downloads total - 27 stars on GitHub - 1 maintainer
slash-tokens 0.0.2
Token budget estimation in 4.8 KB WASM. Sub-millisecond. Zero dependencies.2 versions - Latest release: 6 months ago - 39 downloads total - 7 stars on GitHub - 1 maintainer
classi-cine 0.7.0
A tool that builds smart video playlists by learning your preferences through Bayesian classifica...14 versions - Latest release: 3 months ago - 12.4 thousand downloads total - 6 stars on GitHub - 1 maintainer
blex 0.2.2
A lightweight lexing framework4 versions - Latest release: over 3 years ago - 1 dependent package - 5.9 thousand downloads total - 0 stars on GitHub - 1 maintainer
colorblast-cli 0.0.1
Syntax highlighting CLI for various programming languages, markup languages and various other for...1 version - Latest release: about 3 years ago - 1.51 thousand downloads total - 0 stars on GitHub - 1 maintainer
fern-tokenization 0.0.0
Empty crate, used only to reserve the name.1 version - Latest release: about 4 years ago - 1.67 thousand downloads total - 20 stars on GitHub - 1 maintainer
tinyjuice 0.2.1
Pluggable token compression for OpenHuman.3 versions - Latest release: 3 months ago - 218 downloads total - 15 stars on GitHub - 1 maintainer
colorblast 0.0.3
Syntax highlighting library for various programming languages, markup languages and various other...3 versions - Latest release: about 3 years ago - 1 dependent package - 4.6 thousand downloads total - 0 stars on GitHub - 1 maintainer
build-trie 0.1.1
Procedural macro for generating match and state code representing a trie structure2 versions - Latest release: over 5 years ago - 3.08 thousand downloads total - 3 stars on GitHub - 1 maintainer
any-lexer 0.0.3
Lexers for various programming languages and formats3 versions - Latest release: about 3 years ago - 2 dependent packages - 8.9 thousand downloads total - 0 stars on GitHub - 1 maintainer
vibrato 0.5.2
Vibrato: viterbi-based accelerated tokenizer12 versions - Latest release: over 1 year ago - 1 dependent package - 1 dependent repositories - 68.1 thousand downloads total - 422 stars on GitHub - 2 maintainers
rustrawi 0.1.2 💰
Rust port of the original PHP Sastrawi3 versions - Latest release: over 3 years ago - 3.68 thousand downloads total - 0 stars on GitHub - 1 maintainer
derive-finite-automaton 0.3.0
Procedural macro for generating finite automaton6 versions - Latest release: about 1 year ago - 1 dependent package - 1 dependent repositories - 19.3 thousand downloads total - 3 stars on GitHub - 1 maintainer
derive-finite-automaton-derive 0.3.0
Procedural macro for generating finite automaton6 versions - Latest release: about 1 year ago - 1 dependent package - 1 dependent repositories - 19.9 thousand downloads total - 3 stars on GitHub - 1 maintainer
sentence 0.0.2
Sentence tokenizes English language sentences for use in TTS applications.2 versions - Latest release: over 6 years ago - 3.12 thousand downloads total - 2 stars on GitHub - 1 maintainer
niblits 0.3.14
Token-aware, multi-format text chunking library with language-aware semantic splitting13 versions - Latest release: 3 months ago - 556 downloads total - 1 maintainer
unscanny 0.1.0 💰
Painless string scanning.1 version - Latest release: over 4 years ago - 8 dependent packages - 28 dependent repositories - 18.7 million downloads total - 58 stars on GitHub - 1 maintainer
strizer 0.1.0
minimal and fast library for text tokenization1 version - Latest release: over 5 years ago - 1.94 thousand downloads total - 1 stars on GitHub - 1 maintainer
chunk 0.10.2
The fastest semantic text chunking library — up to 1TB/s chunking throughput10 versions - Latest release: 4 months ago - 11.1 thousand downloads total - 359 stars on GitHub - 1 maintainer
token-align 0.1.0
Maps overlapping tokens and character positions between tokenized texts1 version - Latest release: 30 days ago - 0 downloads total - 1 maintainer
textprep 0.1.6
Text preprocessing primitives: normalization, tokenization, and fast keyword matching.7 versions - Latest release: 5 months ago - 18.3 thousand downloads total - 0 stars on GitHub - 1 maintainer
language-text-analysis 0.1.0
Small, deterministic multilingual text-analysis pipeline for search and document processing1 version - Latest release: about 2 months ago - 0 downloads total - 1 maintainer
kizzasi-tokenizer 0.2.3 💰
Signal quantization and tokenization for Kizzasi AGSP - VQ-VAE, μ-law, continuous embeddings4 versions - Latest release: about 2 months ago - 268 downloads total - 9 stars on GitHub - 1 maintainer
marqant 1.1.8
Quantum-compressed markdown format for AI consumption with 90% token reduction6 versions - Latest release: 6 months ago - 2 thousand downloads total - 1 stars on GitHub - 1 maintainer
pretok 0.1.0
A string pre-tokenizer for C-like syntaxes.1 version - Latest release: about 6 years ago - 1 dependent repositories - 1.71 thousand downloads total - 0 stars on GitHub - 1 maintainer
memchunk 0.4.0
The fastest semantic text chunking library — up to 1TB/s chunking throughput11 versions - Latest release: 9 months ago - 566 downloads total - 2 stars on GitHub - 1 maintainer
scivex-nlp 0.1.1
Scivex — Tokenization, embeddings, and text processing2 versions - Latest release: 6 months ago - 106 downloads total - 8 stars on GitHub - 1 maintainer
bpetok 0.1.2
A simple CLI for tokenizing text input using Byte Pair Encoding (BPE).3 versions - Latest release: about 2 years ago - 3.08 thousand downloads total - 1 maintainer
wordpieces 0.6.1
Split tokens into word pieces10 versions - Latest release: almost 4 years ago - 3 dependent packages - 3 dependent repositories - 22.7 thousand downloads total - 5 stars on GitHub - 1 maintainer
agrocrypto-core 0.1.0
The core engine of AgroCrypto: a blockchain-native asset tokenization and settlement layer.1 version - Latest release: over 1 year ago - 1.05 thousand downloads total - 1 maintainer
brainwires-datasets 0.8.0 💰
Training data pipelines for the Brainwires Agent Framework — JSONL I/O, tokenization, deduplicati...8 versions - Latest release: 6 months ago - 299 downloads total - 19 stars on GitHub - 1 maintainer
go-brrr 0.1.0
Token-efficient code analysis for LLMs - Rust implementation1 version - Latest release: 9 months ago - 27 downloads total - 1 maintainer
bpe-match 0.1.1
A pattern matching library for BPE tokenization, intended to replace regex-based approaches.2 versions - Latest release: 11 months ago - 387 downloads total - 1 maintainer
tokenizer-lib 1.6.0
Tokenization utilities for building parsers in Rust15 versions - Latest release: over 2 years ago - 2 dependent packages - 1 dependent repositories - 30.2 thousand downloads total - 2 stars on GitHub - 1 maintainer
bytepunch-rs 0.1.0
Profile-aware semantic compression for structured documents (CML and beyond)1 version - Latest release: 9 months ago - 829 downloads total - 1 maintainer
esg-tokenization-protocol 0.1.2
Official Rust implementation of the ESG Tokenization Protocol (ERC-8040 / EIP-8040). MIT-grade co...3 versions - Latest release: 10 months ago - 77 downloads total - 1 maintainer
crossandra 1.0.1 💰
A fast and simple lexical tokenization library.4 versions - Latest release: 7 months ago - 1.97 thousand downloads total - 8 stars on GitHub - 1 maintainer
tuck5 0.2.0
A pragmatic lexer/parser generator4 versions - Latest release: almost 3 years ago - 4.93 thousand downloads total - 0 stars on GitHub - 1 maintainer
tok3niz3r-cli 0.2.0
Command-line tool to train, run, and inspect byte-level BPE tokenizers. Installs the `tok3` binary.2 versions - Latest release: 4 months ago - 30 downloads total - 1 maintainer
tok3niz3r-trainer 0.2.0
Trainer for byte-level BPE tokenizers: learns a vocabulary and merge table from a text corpus. Th...2 versions - Latest release: 4 months ago - 46 downloads total - 1 maintainer
toon_ql 0.0.2
A query language for Toon data2 versions - Latest release: 11 months ago - 51 downloads total - 1 maintainer
tok3niz3r-core 0.2.0
Core types and on-disk schema for the tok3niz3r byte-level BPE tokenizer: token ids, vocabulary, ...2 versions - Latest release: 4 months ago - 74 downloads total - 1 maintainer
tok3niz3r-runtime 0.2.0
Encoder and decoder for byte-level BPE tokenizers, with rkyv (zero-copy) and HuggingFace tokenize...2 versions - Latest release: 4 months ago - 60 downloads total - 1 maintainer
bitcoin-get-json-token 0.1.1
A comprehensive Rust library for parsing and tokenizing JSON data, optimized for Bitcoin applicat...2 versions - Latest release: 10 months ago - 8.85 thousand downloads total - 1 maintainer
Related Keywords
rust
24
parsing
21
nlp
18
tokenizer
16
syntax
11
diagnostics
10
framework
9
lexer
9
formatting
9
quasi-quote
9
rustlang
9
proc
8
macro
8
text
8
bpe
7
token
5
text-processing
5
morphological-analysis
4
segmentation
4
japanese
4
code2image
4
analyzer
4
html-syntax-highlighter
4
lexers
4
render-code
4
syntax-highlighting
4
text-scanner
4
llm
4
ai
4
compression
4
utils
4
unicode
3
morphological
3
chunking
3
streaming
2
highlighter
2
tree-sitter
2
highlighting
2
rust-crate
2
mcp
2
parser
2
cli
2
lex
2
embeddings
2
training
2
simd
2
blockchain
2
tokens
2
parse
1
tokenizing
1
context
1
scanning
1
esg
1
openhuman
1
ai-agents
1
artificial-intelligence
1
developer-tools
1
open-source
1
erc8040
1
eip8040
1
prompt-engineering
1
spacy
1
serde-json
1
cuda
1
serving
1
agent
1
pii
1
redaction
1
scientific-computing
1
ast
1
code-analysis
1
reqwest
1
naive-bayes-classifier
1
http
1
bayesian-inference
1
vlc
1
playlist
1
huggingface
1
sentence
1
english
1
tts
1
fern
1
parser-combinators
1
proc-macro
1
perl
1
vscode
1
static-analysis
1
semantic-analysis
1
refactoring
1
code-completion
1
dap
1
debug-adapter-protocol
1
incremental-parsing
1
language-server-protocol
1
lsp
1
json
1
bitcoin
1
cryptocurrency
1
classification
1
lexical-analysis
1