npmjs.org "tokenizers" keyword
@orama/tokenizers 3.1.18 💰
Additional tokenizers for Orama51 versions - Latest release: 10 months ago - 241 thousand downloads last month - 10,581 stars on GitHub - 2 maintainers
openvino-tokenizers-node 2026.4.1
OpenVINO™ Tokenizers adds text processing operations to openvino-node package14 versions - Latest release: 22 days ago - 425 downloads last month - 59 stars on GitHub - 1 maintainer
@zbsearch/tokenizers 4.0.1
Additional tokenizers for ZBSearch8 versions - Latest release: 24 days ago - 72 downloads last month - 304 stars on GitHub - 1 maintainer
@dvdagames/pgn-tokenizer 0.1.6 💰
TypeScript version of PGN Tokenizer, a Byte Pair Encoding (BPE) tokenizer for Chess Portable Game...1 version - Latest release: over 1 year ago - 28 downloads last month - 1 stars on GitHub - 1 maintainer
@naveen521kk/expo-hf-tokenizers 0.1.0
Port of HuggingFace's tokenizers using Expo Modules for React Native Apps1 version - Latest release: almost 2 years ago - 6 downloads last month - 0 stars on GitHub - 1 maintainer
fib-tokenizers 1.2.1
cpp tokenizer module for fibjs.6 versions - Latest release: over 2 years ago - 17 downloads last month - 1 stars on GitHub - 1 maintainer
tokengeex 0.6.2
This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a to...2 versions - Latest release: over 2 years ago - 9 downloads last month - 4 stars on GitHub - 1 maintainer
@anush008/tokenizers-linux-x64-gnu 0.6.0 💰
Multi-arch builds of HuggingFace tokenizers6 versions - Latest release: 7 months ago - 2 dependent packages - 2 dependent repositories - 388 thousand downloads last month - 14 stars on GitHub - 1 maintainer
@anush008/tokenizers-win32-x64-msvc 0.6.0 💰
Multi-arch builds of HuggingFace tokenizers6 versions - Latest release: 7 months ago - 2 dependent packages - 2 dependent repositories - 49.8 thousand downloads last month - 14 stars on GitHub - 1 maintainer
@ohos-ports/anush008-tokenizers-openharmony-arm-gnu 0.6.0-beta.0 💰
Multi-arch builds of HuggingFace tokenizers1 version - Latest release: about 1 month ago - 14 stars on GitHub - 10 maintainers
@anush008/tokenizers-linux-arm64-musl 0.6.0 💰
Multi-arch builds of HuggingFace tokenizers3 versions - Latest release: 7 months ago - 1.65 thousand downloads last month - 14 stars on GitHub - 1 maintainer
kingchop 2.0.4 💰
A text tokenizing library that handles strings, by tokenizing them into arrays, depending on inte...16 versions - Latest release: 4 months ago - 1.44 thousand downloads last month - 1 stars on GitHub - 1 maintainer
@anush008/tokenizers-linux-arm64-gnu 0.6.0 💰
Multi-arch builds of HuggingFace tokenizers2 versions - Latest release: 7 months ago - 6.57 thousand downloads last month - 14 stars on GitHub - 1 maintainer
@anush008/tokenizers 0.6.0 deprecated 💰
Multi-arch builds of HuggingFace tokenizers4 versions - Latest release: 7 months ago - 3 dependent packages - 2 dependent repositories - 435 thousand downloads last month - 14 stars on GitHub - 1 maintainer
@anush008/tokenizers-linux-x64-musl 0.6.0 💰
Multi-arch builds of HuggingFace tokenizers2 versions - Latest release: 7 months ago - 47.6 thousand downloads last month - 14 stars on GitHub - 1 maintainer
@anush008/tokenizers-darwin-universal 0.6.0 💰
Multi-arch builds of HuggingFace tokenizers5 versions - Latest release: 7 months ago - 2 dependent packages - 2 dependent repositories - 37.8 thousand downloads last month - 14 stars on GitHub - 1 maintainer
Top 5.7% on npmjs.org
29 versions - Latest release: about 5 years ago - 16 dependent packages - 47 dependent repositories - 9.28 thousand downloads last month - 8 stars on GitHub - 9 maintainers
string-punctuation-tokenizer 2.2.0
Small library that provides functions to tokenize a string into an array of words with or without...29 versions - Latest release: about 5 years ago - 16 dependent packages - 47 dependent repositories - 9.28 thousand downloads last month - 8 stars on GitHub - 9 maintainers
Related Keywords
huggingface
9
tokenizer
3
javascript
3
search
2
full-text search
2
vector-search
2
vector-database
2
fuzzy search
2
typo-tolerant search
2
search-engine
2
full-text
2
vector search
2
stemming
2
mandarin
2
chinese
2
byte pair encoding
1
pgn
1
chess
1
segmentation
1
scripture-open-components
1
nlp-library
1
nlp
1
punctuation
1
string
1
text tensors
1
string tensors
1
text processing
1
tokenization
1
openvino-tokenizers-node
1
openvino-tokenizers
1
openvino tokenizers
1
OpenVINO tokenizers
1
openvino
1
OpenVINO
1
HuggingFace
1
ExpoHfTokenizers
1
expo-hf-tokenizers
1
expo
1
react-native
1
korean
1
fib-tokenizers
1
byte-pair-encoding
1
llm
1
bpe
1
NLP tokenizer
1
Tokenizer
1
Sentence tokenizer
1
Text tokenizer
1
Node.js tokenizer
1
Flexible text tokenizer
1
CommonJS text tokenizer
1
Paragraph tokenizer
1
Sub sentence tokenizer
1
Stable tokenizer
1
Easy tokenizer
1
Light text tokenizer
1
English text tokenizer
1
natural-language-processing
1
nodejs
1
paragraph-tokenizer
1
sentence-tokenizer
1
text-processing
1
text-tokenization
1
word-tokenizer
1
algiorithm
1
data-structures
1
node
1
search-algorithm
1
typescript
1
typo-tolerance
1
vector
1
vector-database-embedding
1
vector-search-engine
1
NLP
1