npmjs.org "tokenizer" keyword
Top 1.3% on npmjs.org
58 versions - Latest release: 1 day ago - 256 dependent packages - 5,960 dependent repositories - 43.2 million downloads last month - 1,205 stars on GitHub - 1 maintainer
stream-json 3.8.0 π°
A micro-library of stream components for building custom JSON and JSONC processing pipelines with...58 versions - Latest release: 1 day ago - 256 dependent packages - 5,960 dependent repositories - 43.2 million downloads last month - 1,205 stars on GitHub - 1 maintainer
@thasmorato/stream-json 2.1.3 π°
stream-json is the micro-library of Node.js stream components for creating custom JSON processing...6 versions - Latest release: over 1 year ago - 11 downloads last month - 1,205 stars on GitHub - 1 maintainer
@depup/stream-json 3.7.0-depup.0 π°
A micro-library of stream components for building custom JSON and JSONC processing pipelines with...7 versions - Latest release: about 1 month ago - 57 downloads last month - 1,205 stars on GitHub - 1 maintainer
@yahyeapps/stream-json 3.3.2 π°
A micro-library of stream components for building custom JSON and JSONC processing pipelines with...2 versions - Latest release: 3 months ago - 351 downloads last month - 1,205 stars on GitHub - 1 maintainer
stream-json-ak 1.7.511 π°
fork of stream-json3 versions - Latest release: over 3 years ago - 8 downloads last month - 1,205 stars on GitHub - 1 maintainer
Top 0.7% on npmjs.org
51 versions - Latest release: about 9 years ago - 829 dependent packages - 34,659 dependent repositories - 334 thousand downloads last month - 4,380 stars on GitHub - 1 maintainer
jison 0.4.18
A parser generator with Bison's API51 versions - Latest release: about 9 years ago - 829 dependent packages - 34,659 dependent repositories - 334 thousand downloads last month - 4,380 stars on GitHub - 1 maintainer
tokenizers-wasm 1.0.0-rc.3 deprecated
WebAssembly bindings to the Hugging Face tokenizers library, for fast tokenization in the browser.2 versions - Latest release: 11 days ago - 4 dependent packages - 1 dependent repositories - 201 downloads last month - 11,204 stars on GitHub - 2 maintainers
Top 9.2% on npmjs.org
29 versions - Latest release: over 2 years ago - 11 dependent packages - 26 dependent repositories - 91 downloads last month - 5 stars on GitHub - 1 maintainer
happynodetokenizer 7.1.0 π°
A simple, Twitter-aware tokenizer.29 versions - Latest release: over 2 years ago - 11 dependent packages - 26 dependent repositories - 91 downloads last month - 5 stars on GitHub - 1 maintainer
@lpi/react-bootstrap-typeahead 4.0.0-alpha.10
React typeahead with Bootstrap styling1 version - Latest release: almost 7 years ago - 1 dependent package - 1 dependent repositories - 37 downloads last month - 1,006 stars on GitHub - 2 maintainers
@mrizki/natural 0.6.4-build1
General natural language (tokenizing, stemming (English, Russian, Spanish), part-of-speech taggin...1 version - Latest release: almost 7 years ago - 1 dependent package - 3 dependent repositories - 42 downloads last month - 10,884 stars on GitHub - 1 maintainer
Top 0.6% on npmjs.org
154 versions - Latest release: 4 days ago - 499 dependent packages - 24,309 dependent repositories - 5.22 million downloads last month - 10,884 stars on GitHub - 3 maintainers
natural 8.2.6
General natural language (tokenizing, stemming (English, Russian, Spanish), part-of-speech taggin...154 versions - Latest release: 4 days ago - 499 dependent packages - 24,309 dependent repositories - 5.22 million downloads last month - 10,884 stars on GitHub - 3 maintainers
tacnoman-scss-tokenizer 0.3.1
A tokenzier for Sass' SCSS syntax2 versions - Latest release: over 8 years ago - 2 dependent packages - 1 dependent repositories - 7 downloads last month - 25 stars on GitHub - 1 maintainer
Top 1.0% on npmjs.org
1 version - Latest release: almost 11 years ago - 6 dependent packages - 7,486 dependent repositories - 2.93 million downloads last month - 16 maintainers
glsl-token-whitespace-trim 1.0.0
Trim the whitespace within an array of GLSL tokens1 version - Latest release: almost 11 years ago - 6 dependent packages - 7,486 dependent repositories - 2.93 million downloads last month - 16 maintainers
cattown 1.1.1 unpublished
A lightweight, pure JavaScript markdown parser with built-in HTML sanitization using DOMPurify. C...5 versions - Latest release: about 1 year ago - 73 downloads last month - 0 stars on GitHub - 1 maintainer
react-bootstrap-typeahead-ie11-compatible 1.3.0
React-based typeahead using the Bootstrap theme1 version - Latest release: over 9 years ago - 1 dependent package - 1 dependent repositories - 23 downloads last month - 1,006 stars on GitHub - 1 maintainer
@tkloht/react-bootstrap-typeahead 1.3.0
React-based typeahead using the Bootstrap theme1 version - Latest release: over 9 years ago - 2 dependent packages - 1 dependent repositories - 38 downloads last month - 1,006 stars on GitHub - 1 maintainer
react-bootstrap-typeahead-ali 1.0.2
React-based typeahead using the Bootstrap theme by react-bootstrap-typeahead v1.0.0-beta53 versions - Latest release: over 9 years ago - 1 dependent package - 1 dependent repositories - 46 downloads last month - 1,006 stars on GitHub - 1 maintainer
sppro-react-bootstrap-typeahead 5.1.8
React typeahead with Bootstrap styling Forked By SharePointPro4 versions - Latest release: over 5 years ago - 87 downloads last month - 1,006 stars on GitHub - 1 maintainer
react-bootstrap-typeahead-with-restart 1.0.0
React typeahead with Bootstrap styling1 version - Latest release: almost 3 years ago - 63 downloads last month - 1,006 stars on GitHub - 1 maintainer
Top 0.9% on npmjs.org
182 versions - Latest release: over 1 year ago - 226 dependent packages - 2,098 dependent repositories - 719 thousand downloads last month - 1,006 stars on GitHub - 1 maintainer
react-bootstrap-typeahead 6.4.1
React typeahead with Bootstrap styling182 versions - Latest release: over 1 year ago - 226 dependent packages - 2,098 dependent repositories - 719 thousand downloads last month - 1,006 stars on GitHub - 1 maintainer
Top 2.3% on npmjs.org
13 versions - Latest release: about 4 years ago - 66 dependent packages - 998,711 dependent repositories - 4.17 million downloads last month - 25 stars on GitHub - 1 maintainer
scss-tokenizer 0.4.3
A tokenzier for Sass' SCSS syntax13 versions - Latest release: about 4 years ago - 66 dependent packages - 998,711 dependent repositories - 4.17 million downloads last month - 25 stars on GitHub - 1 maintainer
Top 3.5% on npmjs.org
31 versions - Latest release: over 6 years ago - 35 dependent packages - 271 dependent repositories - 50.4 thousand downloads last month - 38 stars on GitHub - 1 maintainer
graphql-query-compress 1.2.4 π°
Compress a GraphQL Query String31 versions - Latest release: over 6 years ago - 35 dependent packages - 271 dependent repositories - 50.4 thousand downloads last month - 38 stars on GitHub - 1 maintainer
Top 2.1% on npmjs.org
9 versions - Latest release: almost 4 years ago - 53 dependent packages - 983 dependent repositories - 120 thousand downloads last month - 171 stars on GitHub - 1 maintainer
gpt3-tokenizer 1.1.5 π°
[](htt...9 versions - Latest release: almost 4 years ago - 53 dependent packages - 983 dependent repositories - 120 thousand downloads last month - 171 stars on GitHub - 1 maintainer
Top 2.4% on npmjs.org
126 versions - Latest release: 4 days ago - 56 dependent packages - 70,737 dependent repositories - 141 million downloads last month - 51 stars on GitHub - 1 maintainer
strtok3 10.6.3 π°
A promise based streaming tokenizer126 versions - Latest release: 4 days ago - 56 dependent packages - 70,737 dependent repositories - 141 million downloads last month - 51 stars on GitHub - 1 maintainer
Top 2.4% on npmjs.org
22 versions - Latest release: about 8 years ago - 84 dependent packages - 9,210 dependent repositories - 2.63 million downloads last month - 52 stars on GitHub - 16 maintainers
glsl-tokenizer 2.1.5
r/w stream of glsl tokens22 versions - Latest release: about 8 years ago - 84 dependent packages - 9,210 dependent repositories - 2.63 million downloads last month - 52 stars on GitHub - 16 maintainers
Top 4.2% on npmjs.org
2 versions - Latest release: over 10 years ago - 5 dependent packages - 7,586 dependent repositories - 4.56 million downloads last month - 5 stars on GitHub - 1 maintainer
glsl-token-inject-block 1.1.0
safely inject a block of tokens into a shader2 versions - Latest release: over 10 years ago - 5 dependent packages - 7,586 dependent repositories - 4.56 million downloads last month - 5 stars on GitHub - 1 maintainer
glsl-token-extract-version 1.0.1
finds a #version field in a set of GLSL tokens2 versions - Latest release: over 11 years ago - 1 dependent package - 1 dependent repositories - 10 downloads last month - 3 stars on GitHub - 1 maintainer
Top 5.3% on npmjs.org
1 version - Latest release: over 12 years ago - 4 dependent packages - 63 dependent repositories - 7.55 thousand downloads last month - 5 stars on GitHub - 1 maintainer
glob-parse 0.0.1
Returns a parsed representation of a glob string; does not require Minimatch.1 version - Latest release: over 12 years ago - 4 dependent packages - 63 dependent repositories - 7.55 thousand downloads last month - 5 stars on GitHub - 1 maintainer
@alexssmusica/strtok3 7.0.1 π°
A promise based streaming tokenizer1 version - Latest release: over 3 years ago - 2 dependent packages - 21 downloads last month - 51 stars on GitHub - 1 maintainer
Top 1.3% on npmjs.org
25 versions - Latest release: 11 months ago - 2,251 dependent packages - 1,940,799 dependent repositories - 858 million downloads last month - 555 stars on GitHub - 1 maintainer
js-tokens 10.0.0
Tiny JavaScript tokenizer.25 versions - Latest release: 11 months ago - 2,251 dependent packages - 1,940,799 dependent repositories - 858 million downloads last month - 555 stars on GitHub - 1 maintainer
Top 2.4% on npmjs.org
40 versions - Latest release: 12 days ago - 36 dependent packages - 22 dependent repositories - 7.78 million downloads last month - 856 stars on GitHub - 1 maintainer
gpt-tokenizer 4.1.0 π°
A pure JavaScript BPE tokenizer with model metadata and pricing for GPT-5, GPT-4o, o-series, and ...40 versions - Latest release: 12 days ago - 36 dependent packages - 22 dependent repositories - 7.78 million downloads last month - 856 stars on GitHub - 1 maintainer
@caoer/strtok3 7.0.3 π°
A promise based streaming tokenizer2 versions - Latest release: almost 4 years ago - 2 dependent packages - 18 downloads last month - 51 stars on GitHub - 1 maintainer
unbash 6.0.2
The best bash parser in JavaScript. Fast, complete, typed.25 versions - Latest release: about 8 hours ago - 61.1 million downloads last month - 139 stars on GitHub - 1 maintainer
Top 4.7% on npmjs.org
5 versions - Latest release: about 1 year ago - 4 dependent packages - 4,330 dependent repositories - 72.4 million downloads last month - 13 stars on GitHub - 1 maintainer
leac 0.7.0 π°
Lexer / tokenizer5 versions - Latest release: about 1 year ago - 4 dependent packages - 4,330 dependent repositories - 72.4 million downloads last month - 13 stars on GitHub - 1 maintainer
@tokenizer/inflate 0.6.0 π°
Tokenized zip support15 versions - Latest release: 6 days ago - 116 million downloads last month - 0 stars on GitHub - 1 maintainer
@ohos-npm-ports/yuku-parser 0.10.2-1
High-performance JavaScript/TypeScript parser7 versions - Latest release: about 1 month ago - 1.11 thousand downloads last month - 7 stars on GitHub - 1 maintainer
Top 3.7% on npmjs.org
19 versions - Latest release: almost 5 years ago - 48 dependent packages - 4,567 dependent repositories - 4.39 million downloads last month - 5 stars on GitHub - 2 maintainers
parenthesis 3.1.8 π°
Parse parentheses from a string19 versions - Latest release: almost 5 years ago - 48 dependent packages - 4,567 dependent repositories - 4.39 million downloads last month - 5 stars on GitHub - 2 maintainers
Top 6.4% on npmjs.org
97 versions - Latest release: about 1 month ago - 2 dependent packages - 7 dependent repositories - 2.93 thousand downloads last month - 214 stars on GitHub - 1 maintainer
is-html-attribute-closing 4.2.6
Is a character on a given index a closing of an HTML attribute?97 versions - Latest release: about 1 month ago - 2 dependent packages - 7 dependent repositories - 2.93 thousand downloads last month - 214 stars on GitHub - 1 maintainer
@papit/lexer 0.0.2
Character-level state machine lexer: describe states and rules, get a token list back. Ships read...2 versions - Latest release: about 13 hours ago - 201 downloads last month - 1 stars on GitHub - 1 maintainer
foldmaker 1.0.5
A small parser generator toolkit that can be used to build parsers, transpilers, formatters.5 versions - Latest release: over 6 years ago - 1 dependent package - 1 dependent repositories - 31 downloads last month - 1 stars on GitHub - 1 maintainer
css-tree-nup 3.0.1 unpublished π°
A tool set for CSS: fast detailed parser (CSS β AST), walker (AST traversal), generator (AST β CS...1 version - Latest release: about 2 years ago - 68 downloads last month - 2,123 stars on GitHub - 1 maintainer
flukejs 0.0.11
tokenizer, lexer12 versions - Latest release: over 8 years ago - 2 dependent packages - 3 dependent repositories - 27 downloads last month - 1 stars on GitHub - 1 maintainer
Top 1.8% on npmjs.org
21 versions - Latest release: 3 months ago - 3 dependent packages - 6,744 dependent repositories - 38.6 million downloads last month - 2,814 stars on GitHub - 1 maintainer
@chevrotain/gast 13.2.0
Grammar AST structure for Chevrotain Parsers21 versions - Latest release: 3 months ago - 3 dependent packages - 6,744 dependent repositories - 38.6 million downloads last month - 2,814 stars on GitHub - 1 maintainer
@depup/css-tree 3.2.1-depup.10 π°
A tool set for CSS: fast detailed parser (CSS β AST), walker (AST traversal), generator (AST β CS...10 versions - Latest release: about 1 month ago - 113 downloads last month - 2,128 stars on GitHub - 1 maintainer
Top 1.6% on npmjs.org
25 versions - Latest release: 3 months ago - 4 dependent packages - 7,794 dependent repositories - 38.8 million downloads last month - 2,814 stars on GitHub - 1 maintainer
@chevrotain/utils 13.2.0
common utilities25 versions - Latest release: 3 months ago - 4 dependent packages - 7,794 dependent repositories - 38.8 million downloads last month - 2,814 stars on GitHub - 1 maintainer
Top 2.3% on npmjs.org
23 versions - Latest release: 3 months ago - 2 dependent packages - 6,533 dependent repositories - 37.8 million downloads last month - 2,814 stars on GitHub - 1 maintainer
@chevrotain/cst-dts-gen 13.2.0
Generates type definitions for Chevrotain CST nodes23 versions - Latest release: 3 months ago - 2 dependent packages - 6,533 dependent repositories - 37.8 million downloads last month - 2,814 stars on GitHub - 1 maintainer
Top 0.9% on npmjs.org
60 versions - Latest release: 8 months ago - 820 dependent packages - 2,569,508 dependent repositories - 500 million downloads last month - 2,128 stars on GitHub - 2 maintainers
css-tree 3.2.1 π°
A tool set for CSS: fast detailed parser (CSS β AST), walker (AST traversal), generator (AST β CS...60 versions - Latest release: 8 months ago - 820 dependent packages - 2,569,508 dependent repositories - 500 million downloads last month - 2,128 stars on GitHub - 2 maintainers
aiot-css-tree 2.3.1 π°
A tool set for CSS: fast detailed parser (CSS β AST), walker (AST traversal), generator (AST β CS...2 versions - Latest release: about 2 years ago - 2.6 thousand downloads last month - 2,128 stars on GitHub - 1 maintainer
@chevrotain/regexp-to-ast 13.2.0
Parses a Regular Expression and outputs an AST12 versions - Latest release: 3 months ago - 1 dependent repositories - 28.8 million downloads last month - 2,814 stars on GitHub - 1 maintainer
Top 1.2% on npmjs.org
26 versions - Latest release: 3 months ago - 13 dependent packages - 7,797 dependent repositories - 84.1 million downloads last month - 2,814 stars on GitHub - 1 maintainer
@chevrotain/types 13.2.0
Type Signatures and API Website for Chevrotain26 versions - Latest release: 3 months ago - 13 dependent packages - 7,797 dependent repositories - 84.1 million downloads last month - 2,814 stars on GitHub - 1 maintainer
Top 0.8% on npmjs.org
175 versions - Latest release: 3 months ago - 282 dependent packages - 17,129 dependent repositories - 43.6 million downloads last month - 2,813 stars on GitHub - 1 maintainer
chevrotain 13.2.0
Chevrotain is a high performance fault tolerant javascript parsing DSL for building recursive dec...175 versions - Latest release: 3 months ago - 282 dependent packages - 17,129 dependent repositories - 43.6 million downloads last month - 2,813 stars on GitHub - 1 maintainer
css-tree-v2 1.1.3-beta01 π°
ε°mdn-dataη±2.0.14εηΊ§ε°2.0.232 versions - Latest release: 10 months ago - 10 downloads last month - 2,127 stars on GitHub - 1 maintainer
Top 3.9% on npmjs.org
11 versions - Latest release: about 2 months ago - 10 dependent packages - 15 dependent repositories - 4.53 thousand downloads last month - 45 stars on GitHub - 1 maintainer
flex-js 2.0.0
FLEX for JavaScript and TypeScript: the fast lexical analyser generator, emitting scanners with n...11 versions - Latest release: about 2 months ago - 10 dependent packages - 15 dependent repositories - 4.53 thousand downloads last month - 45 stars on GitHub - 1 maintainer
Top 1.0% on npmjs.org
43 versions - Latest release: about 1 month ago - 144 dependent packages - 151,327 dependent repositories - 241 million downloads last month - 2,236 stars on GitHub - 1 maintainer
micromark 4.0.3 π°
small commonmark compliant markdown parser with positional info and concrete tokens43 versions - Latest release: about 1 month ago - 144 dependent packages - 151,327 dependent repositories - 241 million downloads last month - 2,236 stars on GitHub - 1 maintainer
@depup/micromark 4.0.3-depup.0 π°
small commonmark compliant markdown parser with positional info and concrete tokens (with updated...2 versions - Latest release: about 1 month ago - 43 downloads last month - 2,236 stars on GitHub - 1 maintainer
parse5-wxmp 3.0.3 π°
HTML parsing/serialization toolset for Node.js. WHATWG HTML Living Standard (aka HTML5)-compliant.1 version - Latest release: about 9 years ago - 2 dependent packages - 1 dependent repositories - 16 downloads last month - 3,938 stars on GitHub - 1 maintainer
Top 7.9% on npmjs.org
17 versions - Latest release: over 7 years ago - 3 dependent packages - 8 dependent repositories - 137 downloads last month - 5 stars on GitHub - 1 maintainer
extract-lemmatized-nonstop-words 1.0.19
Extracts a pure list of lemmatized words of a text filtered by stop words17 versions - Latest release: over 7 years ago - 3 dependent packages - 8 dependent repositories - 137 downloads last month - 5 stars on GitHub - 1 maintainer
Top 7.7% on npmjs.org
9 versions - Latest release: over 5 years ago - 2 dependent packages - 42 dependent repositories - 7.13 thousand downloads last month - 2 stars on GitHub - 1 maintainer
complexion-js 0.2.2
Tokenize JavaScript9 versions - Latest release: over 5 years ago - 2 dependent packages - 42 dependent repositories - 7.13 thousand downloads last month - 2 stars on GitHub - 1 maintainer
Top 7.6% on npmjs.org
27 versions - Latest release: almost 3 years ago - 5 dependent packages - 2 dependent repositories - 485 downloads last month - 8 stars on GitHub - 1 maintainer
expression-engine 1.11.1
An expression tokenizer, parser and evaluator.27 versions - Latest release: almost 3 years ago - 5 dependent packages - 2 dependent repositories - 485 downloads last month - 8 stars on GitHub - 1 maintainer
Top 1.5% on npmjs.org
30 versions - Latest release: almost 6 years ago - 48 dependent packages - 20,812 dependent repositories - 269 thousand downloads last month - 732 stars on GitHub - 1 maintainer
talisman 1.1.4 π°
Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.30 versions - Latest release: almost 6 years ago - 48 dependent packages - 20,812 dependent repositories - 269 thousand downloads last month - 732 stars on GitHub - 1 maintainer
Top 1.4% on npmjs.org
25 versions - Latest release: about 1 month ago - 30 dependent packages - 10,014 dependent repositories - 337 million downloads last month - 1,059 stars on GitHub - 3 maintainers
@csstools/css-tokenizer 4.0.2 π°
Tokenize CSS25 versions - Latest release: about 1 month ago - 30 dependent packages - 10,014 dependent repositories - 337 million downloads last month - 1,059 stars on GitHub - 3 maintainers
llama3-tokenizer-js 1.2.0
JS tokenizer for LLaMA 39 versions - Latest release: about 2 years ago - 20.1 thousand downloads last month - 119 stars on GitHub - 1 maintainer
scbe-aethermoore-cli 4.4.0 π°
Standalone CLI for SCBE-AETHERMOORE: natural-language command resolver with autocorrect ledger, G...14 versions - Latest release: 5 months ago - 20 downloads last month - 7 stars on GitHub - 1 maintainer
escuelle 0.0.11
SQL tokeniser & parser11 versions - Latest release: almost 8 years ago - 2 dependent packages - 2 dependent repositories - 18 downloads last month - 3 stars on GitHub - 1 maintainer
@lenml/tokenizer-text_embedding_ada002 3.7.2
text_embedding_ada002 tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 33 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-qwen1_5 3.7.2
qwen1_5 tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 75 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-llama3_1 3.7.2
llama3_1 tokenizer for NodeJS/Browser6 versions - Latest release: about 1 year ago - 406 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gemma2 3.7.2
gemma2 tokenizer for NodeJS/Browser5 versions - Latest release: about 1 year ago - 2.25 thousand downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-qwen2_5 3.7.2
qwen2_5 tokenizer for NodeJS/Browser5 versions - Latest release: about 1 year ago - 243 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-deepseek_v4 3.7.2
deepseek_v4 tokenizer for NodeJS/Browser1 version - Latest release: 6 months ago - 35 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-llama3 3.7.2
llama3 tokenizer for NodeJS/Browser12 versions - Latest release: about 1 year ago - 1.1 thousand downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gpt4 3.7.2
gpt4 tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 211 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-baichuan2 3.7.2
baichuan2 tokenizer for NodeJS/Browser14 versions - Latest release: about 1 year ago - 24 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizers 3.7.2
a lightweight no-dependency fork of transformers.js (only tokenizers)15 versions - Latest release: about 1 year ago - 108 thousand downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-text_davinci003 3.7.2
text_davinci003 tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 11 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-mistral_nemo 3.7.2
mistral_nemo tokenizer for NodeJS/Browser5 versions - Latest release: about 1 year ago - 5.15 thousand downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gpt4o 3.7.2
gpt4o tokenizer for NodeJS/Browser11 versions - Latest release: about 1 year ago - 321 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-qwen3 3.7.2
qwen3 tokenizer for NodeJS/Browser2 versions - Latest release: about 1 year ago - 2.64 thousand downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-internlm2 3.7.2
internlm2 tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 22 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gpt35turbo 3.7.2
gpt35turbo tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 224 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gemma 3.7.2
gemma tokenizer for NodeJS/Browser14 versions - Latest release: about 1 year ago - 288 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gemma3 3.7.2
gemma3 tokenizer for NodeJS/Browser3 versions - Latest release: about 1 year ago - 52.6 thousand downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gpt35turbo16k 3.7.2
gpt35turbo16k tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 21 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gpt2 3.7.2
gpt2 tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 153 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-text_davinci002 3.7.2
text_davinci002 tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 84 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-yi 3.7.2
yi tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 48 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gemini 3.7.2
gemini tokenizer for NodeJS/Browser5 versions - Latest release: about 1 year ago - 17 thousand downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-claude 3.7.2
claude tokenizer for NodeJS/Browser14 versions - Latest release: about 1 year ago - 15.4 thousand downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gptoss 3.7.2
gptoss tokenizer for NodeJS/Browser1 version - Latest release: about 1 year ago - 348 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-gpt3 3.7.2
gpt3 tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 94 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-claude1 3.7.2
claude1 tokenizer for NodeJS/Browser14 versions - Latest release: about 1 year ago - 30 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-minicpm_v4_5 3.7.2
minicpm_v4_5 tokenizer for NodeJS/Browser1 version - Latest release: about 1 year ago - 10 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-llama2 3.7.2
llama2 tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 3.25 thousand downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-chatglm3 3.7.2
chatglm3 tokenizer for NodeJS/Browser14 versions - Latest release: about 1 year ago - 233 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-llama3_2 3.7.2
llama3_2 tokenizer for NodeJS/Browser4 versions - Latest release: about 1 year ago - 326 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-deepseek_v3 3.7.2
deepseek_v3 tokenizer for NodeJS/Browser4 versions - Latest release: about 1 year ago - 5.69 thousand downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-aya_expanse 3.7.2
aya_expanse tokenizer for NodeJS/Browser5 versions - Latest release: about 1 year ago - 27 downloads last month - 34 stars on GitHub - 1 maintainer
@lenml/tokenizer-command_r_plus 3.7.2
command_r_plus tokenizer for NodeJS/Browser13 versions - Latest release: about 1 year ago - 47 downloads last month - 34 stars on GitHub - 1 maintainer
epayco-tokenizer 1.0.6
Agnostic browser tokenizer for epayco7 versions - Latest release: almost 8 years ago - 2 dependent packages - 2 dependent repositories - 22 downloads last month - 0 stars on GitHub - 1 maintainer
@cjser/tokenizer__inflate 0.6.0-cjser.2 π°
Tokenized zip support3 versions - Latest release: about 22 hours ago - 66 downloads last month - 1 maintainer
Related Keywords
parser
270
lexer
202
nlp
155
typescript
144
llm
126
token
110
javascript
103
ast
96
tokens
92
openai
86
tokenize
72
ai
68
analyzer
65
compiler
65
cli
63
japanese
60
generator
59
analysis
58
chinese
55
anthropic
54
stream
54
wasm
54
claude
53
stemmer
53
gpt
52
mistral
51
nodejs
51
chatgpt
51
llama
48
multilingual
48
react
47
library
47
stopwords
46
tiktoken
46
sentiment analysis
44
tokenization
44
i18n
43
css
43
llama2
41
webassembly
41
morphological
41
nlp.js
41
nlu
41
nlpjs-neo
41
segmentation
40
html
40
parse
39
streaming
36
machine learning
36
mcp
36
typeahead
35
parsing
35
llama3
35
gpt4
35
json
35
korean
34
transfomers
34
chatglm
34
baichuan
34
syntax
33
developer-tools
33
context-window
32
autocomplete
32
text
32
scanner
32
word
31
bpe
31
pos
31
natural language processing
31
string
31
language
30
NLP
29
async
28
flex
28
huggingface
28
gemini
27
search
27
lex
26
split
26
markdown
25
prompt-engineering
25
browser
25
rust
25
jison
24
css-parser
24
regex
23
regexp
23
grammar
22
jieba
21
pos tagger
21
morphological analyzer
21
transformers
21
interpreter
20
data
20
segment
20
fast
20
cost
19
node
19
optimizer
19
AI
19