npmjs.org "tokenizers" keyword
View the packages on the npmjs.org package registry that are tagged with the "tokenizers" keyword.
Top 5.7% on npmjs.org
29 versions - Latest release: about 4 years ago - 16 dependent packages - 47 dependent repositories - 6.63 thousand downloads last month - 8 stars on GitHub - 11 maintainers
string-punctuation-tokenizer 2.2.0
Small library that provides functions to tokenize a string into an array of words with or without...29 versions - Latest release: about 4 years ago - 16 dependent packages - 47 dependent repositories - 6.63 thousand downloads last month - 8 stars on GitHub - 11 maintainers
tokengeex 0.6.2
This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a to...2 versions - Latest release: over 1 year ago - 19 downloads last month - 4 stars on GitHub - 1 maintainer
@anush008/tokenizers-darwin-universal 0.2.0 💰
Multi-arch builds of HuggingFace tokenizers2 versions - Latest release: about 2 years ago - 2 dependent packages - 2 dependent repositories - 6.87 thousand downloads last month - 9 stars on GitHub - 1 maintainer
@anush008/tokenizers 0.2.0 💰
Multi-arch builds of HuggingFace tokenizers2 versions - Latest release: about 2 years ago - 3 dependent packages - 2 dependent repositories - 108 thousand downloads last month - 9 stars on GitHub - 1 maintainer
@anush008/tokenizers-linux-x64-gnu 0.2.0 💰
Multi-arch builds of HuggingFace tokenizers2 versions - Latest release: about 2 years ago - 2 dependent packages - 2 dependent repositories - 101 thousand downloads last month - 9 stars on GitHub - 1 maintainer
@anush008/tokenizers-win32-x64-msvc 0.2.0 💰
Multi-arch builds of HuggingFace tokenizers2 versions - Latest release: about 2 years ago - 2 dependent packages - 2 dependent repositories - 6.52 thousand downloads last month - 9 stars on GitHub - 1 maintainer
openvino-tokenizers-node 2025.3.0
OpenVINOâ„¢ Tokenizers adds text processing operations to openvino-node package5 versions - Latest release: 29 days ago - 123 downloads last month - 40 stars on GitHub - 1 maintainer
@dvdagames/pgn-tokenizer 0.1.6 💰
TypeScript version of PGN Tokenizer, a Byte Pair Encoding (BPE) tokenizer for Chess Portable Game...1 version - Latest release: 6 months ago - 9 downloads last month - 0 stars on GitHub - 1 maintainer
@orama/tokenizers 3.1.13 💰
Additional tokenizers for Orama46 versions - Latest release: 27 days ago - 8.12 thousand downloads last month - 8,156 stars on GitHub - 7 maintainers
kingchop 1.2.0 💰
A text tokenizing library that handles strings, by tokenizing them into arrays, depending on inte...11 versions - Latest release: about 1 year ago - 8 downloads last month - 1 stars on GitHub - 1 maintainer
@naveen521kk/expo-hf-tokenizers 0.1.0
Port of HuggingFace's tokenizers using Expo Modules for React Native Apps1 version - Latest release: 11 months ago - 7 downloads last month - 0 stars on GitHub - 1 maintainer
fib-tokenizers 1.2.1
cpp tokenizer module for fibjs.6 versions - Latest release: about 1 year ago - 4 downloads last month - 1 stars on GitHub - 1 maintainer
Related Keywords
huggingface
5
tokenizer
3
javascript
3
fib-tokenizers
1
CommonJS text tokenizer
1
Flexible text tokenizer
1
Node.js tokenizer
1
Text tokenizer
1
Sentence tokenizer
1
Tokenizer
1
NLP tokenizer
1
vector-search-engine
1
vector-search
1
vector-database-embedding
1
vector-database
1
vector
1
typo-tolerance
1
typescript
1
search-engine
1
search-algorithm
1
node
1
HuggingFace
1
ExpoHfTokenizers
1
expo-hf-tokenizers
1
expo
1
react-native
1
word-tokenizer
1
text-tokenization
1
text-processing
1
sentence-tokenizer
1
paragraph-tokenizer
1
nodejs
1
natural-language-processing
1
Light text tokenizer
1
Easy tokenizer
1
Stable tokenizer
1
Sub sentence tokenizer
1
Paragraph tokenizer
1
text tensors
1
string tensors
1
text processing
1
tokenization
1
openvino-tokenizers-node
1
openvino-tokenizers
1
openvino tokenizers
1
OpenVINO tokenizers
1
openvino
1
OpenVINO
1
NLP
1
segmentation
1
scripture-open-components
1
nlp-library
1
nlp
1
punctuation
1
string
1
data-structures
1
algiorithm
1
chinese
1
mandarin
1
stemming
1
vector search
1
full-text
1
typo-tolerant search
1
fuzzy search
1
search
1
full-text search
1
byte-pair-encoding
1
llm
1
bpe
1
byte pair encoding
1
pgn
1
chess
1