Ecosyste.ms: Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

npmjs.org "tokenizer" keyword

@flowaccount/node-icu-tokenizer 2.0.0
String Tokenizer for Node.js using ICU's BreakIterators
13 versions - Latest release: 7 months ago - 1 dependent package - 463 downloads last month - 3 stars on GitHub - 5 maintainers
Top 6.7% on npmjs.org
@mikro-orm/mongo-highlighter 1.0.0 πŸ’°
Node.js MongoDB query highlighter for CLI based.
1 version - Latest release: almost 4 years ago - 9 dependent packages - 35 dependent repositories - 3.22 thousand downloads last month - 1 stars on GitHub - 1 maintainer
Top 4.9% on npmjs.org
@mikro-orm/sql-highlighter 1.0.1 πŸ’°
Node.js SQL highlighter for CLI based on doctrine/sql-formatter.
2 versions - Latest release: almost 4 years ago - 23 dependent packages - 126 dependent repositories - 50.2 thousand downloads last month - 9 stars on GitHub - 1 maintainer
Top 8.9% on npmjs.org
@cohensive/select-core 0.10.5
Select input
17 versions - Latest release: over 1 year ago - 3 dependent packages - 1 dependent repositories - 8 downloads last month - 1 maintainer
Top 4.3% on npmjs.org
tokenizer2 2.0.1
tokenize any text stream given some basic regex rules to match tokens
7 versions - Latest release: about 6 years ago - 17 dependent packages - 373 dependent repositories - 13.7 thousand downloads last month - 14 stars on GitHub - 1 maintainer
universal-shortcodes 0.9.2
A universal shortcode parser that can work with Wordpress shortcodes, BBCodes and also more custo...
4 versions - Latest release: almost 6 years ago - 1 dependent repositories - 35 downloads last month - 3 stars on GitHub - 1 maintainer
Top 6.5% on npmjs.org
react-native-parse-html 2.2.6
WHATWG HTML5 specification-compliant, fast and ready for production HTML parsing/serialization to...
4 versions - Latest release: almost 7 years ago - 4 dependent packages - 141 dependent repositories - 2.56 thousand downloads last month - 4 stars on GitHub - 4 maintainers
lexr 1.0.4
Lexical analyzer built in Javascript
26 versions - Latest release: almost 6 years ago - 1 dependent package - 2 dependent repositories - 100 downloads last month - 15 stars on GitHub - 1 maintainer
Top 0.6% on npmjs.org
natural 7.0.5
General natural language (tokenizing, stemming (English, Russian, Spanish), part-of-speech taggin...
139 versions - Latest release: 27 days ago - 499 dependent packages - 24,309 dependent repositories - 576 thousand downloads last month - 10,489 stars on GitHub - 3 maintainers
@mrizki/natural 0.6.4-build1
General natural language (tokenizing, stemming (English, Russian, Spanish), part-of-speech taggin...
1 version - Latest release: over 4 years ago - 1 dependent package - 3 dependent repositories - 9 downloads last month - 10,481 stars on GitHub - 1 maintainer
bassoon 1.0.7
a simplified and optimized fork of oboejs
9 versions - Latest release: about 3 years ago - 1 dependent package - 1 dependent repositories - 26 downloads last month - 2 stars on GitHub - 1 maintainer
lex-es6 1.1.0 πŸ’°
es6 version of aaditmshah/lexer
4 versions - Latest release: over 8 years ago - 2 dependent packages - 1 dependent repositories - 9 downloads last month - 0 stars on GitHub - 1 maintainer
react-markup-formatter 1.1.0
Render HTML-like markup language from strings using React components
2 versions - Latest release: 8 months ago - 9 downloads last month - 0 stars on GitHub - 1 maintainer
Top 1.3% on npmjs.org
js-tokens 9.0.0
Tiny JavaScript tokenizer.
23 versions - Latest release: 3 months ago - 2,251 dependent packages - 1,940,799 dependent repositories - 187 million downloads last month - 482 stars on GitHub - 1 maintainer
Top 1.4% on npmjs.org
@csstools/css-tokenizer 2.3.1 πŸ’°
Tokenize CSS
13 versions - Latest release: 9 days ago - 30 dependent packages - 10,014 dependent repositories - 11.8 million downloads last month - 829 stars on GitHub - 3 maintainers
@onlang-org/onst-schemastore 1.2.2
Standalone package designed as helper for on-lang. Fetches catalog from SchemStore (www.schemasto...
5 versions - Latest release: 2 months ago - 34 downloads last month - 1 stars on GitHub - 1 maintainer
ngramable 0.2.1 πŸ’°
String ngram splitter.
3 versions - Latest release: over 4 years ago - 1 dependent package - 2 dependent repositories - 22 downloads last month - 2 stars on GitHub - 1 maintainer
Top 3.7% on npmjs.org
parenthesis 3.1.8 πŸ’°
Parse parentheses from a string
19 versions - Latest release: over 2 years ago - 48 dependent packages - 4,567 dependent repositories - 1.15 million downloads last month - 5 stars on GitHub - 2 maintainers
@lenml/tokenizer-gpt35turbo 1.0.4
gpt35turbo tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 145 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-qwen1_5 1.0.4
qwen1_5 tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 145 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-llama3 1.0.4
llama3 tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 152 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-yi 1.0.4
yi tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 147 downloads last month - 1 stars on GitHub - 1 maintainer
Top 7.9% on npmjs.org
extract-lemmatized-nonstop-words 1.0.19
Extracts a pure list of lemmatized words of a text filtered by stop words
17 versions - Latest release: almost 5 years ago - 3 dependent packages - 8 dependent repositories - 603 downloads last month - 5 stars on GitHub - 1 maintainer
@lenml/tokenizer-gpt35turbo16k 1.0.4
gpt35turbo16k tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 139 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-claude 1.0.4
claude tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 145 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-text_davinci003 1.0.4
text_davinci003 tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 142 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-text_embedding_ada002 1.0.4
text_embedding_ada002 tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 143 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-chatglm3 1.0.4
chatglm3 tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 144 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-llama2 1.0.4
llama2 tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 157 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-gpt4 1.0.4
gpt4 tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 142 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-claude1 1.0.4
claude1 tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 143 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-command_r_plus 1.0.4
command_r_plus tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 148 downloads last month - 1 stars on GitHub - 1 maintainer
@lenml/tokenizer-gpt3 1.0.4
gpt3 tokenizer for NodeJS/Browser
3 versions - Latest release: 5 days ago - 141 downloads last month - 1 stars on GitHub - 1 maintainer
@adguard/css-tokenizer 0.0.1
CSS / Extended CSS tokenizer
1 version - Latest release: 7 months ago - 7 downloads last month - 47 stars on GitHub - 3 maintainers
@lenml/tokenizers 1.0.4
a lightweight no-dependency fork of transformers.js (only tokenizers)
4 versions - Latest release: 5 days ago - 699 downloads last month - 1 stars on GitHub - 1 maintainer
react-text-tokenizer 1.0.3
React component to convert a text string into React.ReactNodes
4 versions - Latest release: 2 months ago - 1 dependent package - 41 downloads last month - 0 stars on GitHub - 1 maintainer
parse5-wxmp 3.0.3 πŸ’°
HTML parsing/serialization toolset for Node.js. WHATWG HTML Living Standard (aka HTML5)-compliant.
1 version - Latest release: over 6 years ago - 2 dependent packages - 1 dependent repositories - 14 downloads last month - 3,559 stars on GitHub - 1 maintainer
Top 5.2% on npmjs.org
html-tokenizer 4.0.1
Small, fast, event-driven, fault-tolerant html tokenizer. Works in node or browsers.
15 versions - Latest release: over 3 years ago - 4 dependent packages - 4 dependent repositories - 3.65 thousand downloads last month - 35 stars on GitHub - 1 maintainer
Top 2.1% on npmjs.org
gpt3-tokenizer 1.1.5 πŸ’°
[![Build](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml/badge.svg)](htt...
9 versions - Latest release: over 1 year ago - 53 dependent packages - 983 dependent repositories - 82.8 thousand downloads last month - 171 stars on GitHub - 1 maintainer
Top 3.8% on npmjs.org
StringScanner 0.0.3
StringScanner performs lexical scanning operations on a string.
2 versions - Latest release: over 11 years ago - 21 dependent packages - 156 dependent repositories - 19.2 thousand downloads last month - 21 stars on GitHub - 1 maintainer
Top 7.1% on npmjs.org
gpt-token-utils 1.2.0
Isomorphic utilities for GPT-3 tokenization and prompt building.
5 versions - Latest release: about 1 year ago - 2 dependent packages - 7 dependent repositories - 3.48 thousand downloads last month - 11 stars on GitHub - 1 maintainer
@chevrotain/regexp-to-ast 11.0.3
Parses a Regular Expression and outputs an AST
4 versions - Latest release: 9 months ago - 1 dependent repositories - 152 thousand downloads last month - 2,404 stars on GitHub - 1 maintainer
Top 5.5% on npmjs.org
complexion 0.2.2
A fast tokenizer/lexer core that is well tested.
7 versions - Latest release: almost 3 years ago - 5 dependent packages - 44 dependent repositories - 9.64 thousand downloads last month - 4 stars on GitHub - 1 maintainer
Top 4.1% on npmjs.org
snapdragon-lexer 4.0.0
Converts a string into an array of tokens, with useful methods for looking ahead and behind, capt...
5 versions - Latest release: over 5 years ago - 8 dependent packages - 41 dependent repositories - 2.5 thousand downloads last month - 20 stars on GitHub - 1 maintainer
Top 1.5% on npmjs.org
talisman 1.1.4 πŸ’°
Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
30 versions - Latest release: over 3 years ago - 48 dependent packages - 20,812 dependent repositories - 60.2 thousand downloads last month - 698 stars on GitHub - 1 maintainer
Top 2.4% on npmjs.org
glsl-tokenizer 2.1.5
r/w stream of glsl tokens
22 versions - Latest release: over 5 years ago - 84 dependent packages - 9,210 dependent repositories - 1.36 million downloads last month - 50 stars on GitHub - 18 maintainers
Top 7.7% on npmjs.org
complexion-js 0.2.2
Tokenize JavaScript
9 versions - Latest release: almost 3 years ago - 2 dependent packages - 42 dependent repositories - 9.97 thousand downloads last month - 1 stars on GitHub - 1 maintainer
Top 8.5% on npmjs.org
llama-tokenizer-js 1.2.1
JS tokenizer for LLaMA-based LLMs
8 versions - Latest release: about 2 months ago - 1 dependent package - 1 dependent repositories - 12.5 thousand downloads last month - 294 stars on GitHub - 1 maintainer
snapdragon-token 4.0.0
Create a snapdragon token. Used by the snapdragon lexer, but can also be used by plugins.
8 versions - Latest release: about 6 years ago - 1 dependent package - 10 dependent repositories - 46 downloads last month - 2 stars on GitHub - 1 maintainer
Top 6.7% on npmjs.org
toky 0.1.0
simple code like tokenizer
8 versions - Latest release: about 3 years ago - 4 dependent packages - 13 dependent repositories - 2.92 thousand downloads last month - 6 stars on GitHub - 1 maintainer
Top 4.2% on npmjs.org
glsl-token-inject-block 1.1.0
safely inject a block of tokens into a shader
2 versions - Latest release: over 8 years ago - 5 dependent packages - 7,586 dependent repositories - 1.1 million downloads last month - 5 stars on GitHub - 1 maintainer
Top 1.6% on npmjs.org
@chevrotain/utils 11.0.3
common utilities
17 versions - Latest release: 9 months ago - 4 dependent packages - 7,794 dependent repositories - 1.57 million downloads last month - 2,404 stars on GitHub - 1 maintainer
Top 2.3% on npmjs.org
@chevrotain/cst-dts-gen 11.0.3
Generates type definitions for Chevrotain CST nodes
15 versions - Latest release: 9 months ago - 2 dependent packages - 6,533 dependent repositories - 1.26 million downloads last month - 2,404 stars on GitHub - 1 maintainer
Top 1.8% on npmjs.org
@chevrotain/gast 11.0.3
Grammar AST structure for Chevrotain Parsers
13 versions - Latest release: 9 months ago - 3 dependent packages - 6,744 dependent repositories - 1.26 million downloads last month - 2,404 stars on GitHub - 1 maintainer
Top 1.2% on npmjs.org
@chevrotain/types 11.0.3
Type Signatures and API Website for Chevrotain
18 versions - Latest release: 9 months ago - 13 dependent packages - 7,797 dependent repositories - 1.57 million downloads last month - 2,404 stars on GitHub - 1 maintainer
Top 0.8% on npmjs.org
chevrotain 11.0.3
Chevrotain is a high performance fault tolerant javascript parsing DSL for building recursive dec...
167 versions - Latest release: 9 months ago - 282 dependent packages - 17,129 dependent repositories - 3.79 million downloads last month - 2,404 stars on GitHub - 1 maintainer
taibun 1.0.1
Taiwanese Hokkien Transliterator and Tokeniser
3 versions - Latest release: 11 days ago - 140 downloads last month - 1 stars on GitHub - 1 maintainer
andreihar-example 0.0.4 unpublished
Taiwanese Hokkien Transliterator and Tokeniser
4 versions - Latest release: 12 days ago - 1 maintainer
Top 6.4% on npmjs.org
@hikerpig/moo 0.5.2-beta.2
Optimised tokenizer/lexer generator! πŸ„ Much performance. Moo!
2 versions - Latest release: over 2 years ago - 2 dependent packages - 3 dependent repositories - 111 downloads last month - 800 stars on GitHub - 1 maintainer
Top 2.9% on npmjs.org
parse5-with-errors 4.0.4
Fork: HTML parsing/serialization toolset for Node.js. WHATWG HTML Living Standard (aka HTML5)-com...
3 versions - Latest release: over 3 years ago - 2 dependent packages - 974 dependent repositories - 57 thousand downloads last month - 15 maintainers
Top 4.1% on npmjs.org
hyntax-yx 1.0.9
Straightforward HTML parser for Node.js and browser
11 versions - Latest release: almost 2 years ago - 4 dependent packages - 43 dependent repositories - 2.76 thousand downloads last month - 137 stars on GitHub - 1 maintainer
Top 7.0% on npmjs.org
remarked 0.1.4 πŸ’°
Markdown parser and lexer. A fork of marked.js maintained for Assemble.
5 versions - Latest release: almost 10 years ago - 4 dependent packages - 26 dependent repositories - 364 downloads last month - 6 stars on GitHub - 2 maintainers
llama3-tokenizer-js 1.1.3
JS tokenizer for LLaMA 3
6 versions - Latest release: 6 days ago - 905 downloads last month - 28 stars on GitHub - 1 maintainer
Top 3.1% on npmjs.org
hyntax 1.1.9
Straightforward HTML parser for Node.js and browser
32 versions - Latest release: over 3 years ago - 16 dependent packages - 150 dependent repositories - 20.9 thousand downloads last month - 134 stars on GitHub - 1 maintainer
text-tokenizer 1.0.4
A simple perfomant tokenizer
5 versions - Latest release: over 5 years ago - 2 dependent packages - 2 dependent repositories - 136 downloads last month - 0 stars on GitHub - 1 maintainer
Top 1.3% on npmjs.org
kuromoji 0.1.2
JavaScript implementation of Japanese morphological analyzer
7 versions - Latest release: about 6 years ago - 57 dependent packages - 1,756 dependent repositories - 252 thousand downloads last month - 775 stars on GitHub - 1 maintainer
promptfusion 1.0.2
A CLI tool to concatenate all text files in your CWD with headers for GPT prompt engineering.
2 versions - Latest release: 3 months ago - 68 downloads last month - 1 stars on GitHub - 1 maintainer
react-bootstrap-4-typeahead 0.9.0
React-based typeahead using the Bootstrap theme
4 versions - Latest release: almost 7 years ago - 1 dependent package - 1 dependent repositories - 64 downloads last month - 5 stars on GitHub - 1 maintainer
jawi 0.0.1
Jawi
1 version - Latest release: 3 days ago - 1 maintainer
@basd/search 0.0.5
A powerful and flexible text search library for JavaScript that enables you to build a simple tex...
5 versions - Latest release: 5 months ago - 1 dependent package - 31 downloads last month - 0 stars on GitHub - 1 maintainer
Top 5.6% on npmjs.org
pratt-parser 10.0.2
TDOP parser
90 versions - Latest release: 24 days ago - 4 dependent packages - 16 dependent repositories - 3.75 thousand downloads last month - 14 stars on GitHub - 1 maintainer
vtokenize 2.1.1
A simple string tokenizer that makes tokens by looping through the given string.
3 versions - Latest release: about 6 years ago - 2 dependent packages - 7 dependent repositories - 9 downloads last month - 0 stars on GitHub - 1 maintainer
@childrentime/js-interpreter 1.0.9
This project is about to implement a simple javascript parser and visualize the ast
10 versions - Latest release: almost 2 years ago - 2 dependent packages - 1 dependent repositories - 17 downloads last month - 16 stars on GitHub - 1 maintainer
nlpo3 0.2.1
Node.js binding for nlpO3 Thai language processing library
1 version - Latest release: almost 3 years ago - 1 dependent package - 6 downloads last month - 30 stars on GitHub - 2 maintainers
tokenizerflow 1.0.0
Tokenize string using declared patterns and additional flow-control rules
1 version - Latest release: almost 2 years ago - 2 dependent packages - 1 dependent repositories - 3 downloads last month - 0 stars on GitHub - 1 maintainer
kotlin-ace-common 1.3.36
A Kotlin/JS wrapper for the Ace code editor - shared classes between web and worker environments.
4 versions - Latest release: over 5 years ago - 3 dependent packages - 1 dependent repositories - 10 downloads last month - 5 stars on GitHub - 1 maintainer
@lpi/react-bootstrap-typeahead 4.0.0-alpha.10
React typeahead with Bootstrap styling
1 version - Latest release: over 4 years ago - 1 dependent package - 1 dependent repositories - 4 downloads last month - 988 stars on GitHub - 2 maintainers
react-bootstrap-typeahead-ali 1.0.2
React-based typeahead using the Bootstrap theme by react-bootstrap-typeahead v1.0.0-beta5
3 versions - Latest release: about 7 years ago - 1 dependent package - 1 dependent repositories - 6 downloads last month - 988 stars on GitHub - 1 maintainer
@tkloht/react-bootstrap-typeahead 1.3.0
React-based typeahead using the Bootstrap theme
1 version - Latest release: about 7 years ago - 2 dependent packages - 1 dependent repositories - 6 downloads last month - 988 stars on GitHub - 1 maintainer
sppro-react-bootstrap-typeahead 5.1.8
React typeahead with Bootstrap styling Forked By SharePointPro
4 versions - Latest release: almost 3 years ago - 64 downloads last month - 988 stars on GitHub - 1 maintainer
react-bootstrap-typeahead-with-restart 1.0.0
React typeahead with Bootstrap styling
1 version - Latest release: 6 months ago - 13 downloads last month - 988 stars on GitHub - 1 maintainer
Top 0.9% on npmjs.org
react-bootstrap-typeahead 6.3.2
React typeahead with Bootstrap styling
173 versions - Latest release: 7 months ago - 226 dependent packages - 2,098 dependent repositories - 973 thousand downloads last month - 988 stars on GitHub - 1 maintainer
react-bootstrap-typeahead-ie11-compatible 1.3.0
React-based typeahead using the Bootstrap theme
1 version - Latest release: about 7 years ago - 1 dependent package - 1 dependent repositories - 3 downloads last month - 987 stars on GitHub - 1 maintainer
kuromoji-techtech 0.1.4
JavaScript implementation of Japanese morphological analyzer
2 versions - Latest release: 10 months ago - 1 dependent package - 9 downloads last month - 0 stars on GitHub - 1 maintainer
Top 0.7% on npmjs.org
jison 0.4.18
A parser generator with Bison's API
51 versions - Latest release: over 6 years ago - 829 dependent packages - 34,659 dependent repositories - 270 thousand downloads last month - 4,326 stars on GitHub - 1 maintainer
Top 2.3% on npmjs.org
scss-tokenizer 0.4.3
A tokenzier for Sass' SCSS syntax
13 versions - Latest release: almost 2 years ago - 66 dependent packages - 998,711 dependent repositories - 7.26 million downloads last month - 25 stars on GitHub - 1 maintainer
string-tokeniser 2.0.3
A simple string tokeniser
22 versions - Latest release: 4 days ago - 114 downloads last month - 3 stars on GitHub - 1 maintainer
Top 1.3% on npmjs.org
php-parser 3.1.5 πŸ’°
Parse PHP code from JS and returns its AST
74 versions - Latest release: 11 months ago - 101 dependent packages - 3,016 dependent repositories - 602 thousand downloads last month - 513 stars on GitHub - 2 maintainers
figmage 0.13.1
Flexibly generate design tokens from your Figma files
20 versions - Latest release: 4 days ago - 1 dependent package - 345 downloads last month - 8 stars on GitHub - 1 maintainer
x-scanner 0.4.9
Convenient lexical scanner inspired by Ruby's StringScanner
1 version - Latest release: almost 2 years ago - 14 downloads last month - 0 stars on GitHub - 1 maintainer
bareun 1.1.0
Javascript client for bareun
5 versions - Latest release: about 1 year ago - 2 dependent packages - 25 downloads last month - 5 stars on GitHub - 1 maintainer
postcss-color-tokenizer 0.0.2
PostCSS plugin to convert css colours into variables for SCSS
3 versions - Latest release: over 3 years ago - 1 dependent package - 7 downloads last month - 0 stars on GitHub - 1 maintainer
Top 5.3% on npmjs.org
glob-parse 0.0.1
Returns a parsed representation of a glob string; does not require Minimatch.
1 version - Latest release: almost 10 years ago - 4 dependent packages - 63 dependent repositories - 7.7 thousand downloads last month - 5 stars on GitHub - 1 maintainer
kotlin-ace-worker 1.3.36
A Kotlin/JS wrapper for the Ace code editor - web worker environment classes.
5 versions - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 9 downloads last month - 5 stars on GitHub - 1 maintainer
kotlin-ace-web 1.3.36
A Kotlin/JS wrapper for the Ace code editor - web environment classes.
6 versions - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 8 downloads last month - 5 stars on GitHub - 1 maintainer
cjk-tokenizer 0.1.0
A CJK text tokenizer
2 versions - Latest release: over 7 years ago - 1 dependent package - 1 dependent repositories - 8 downloads last month - 4 stars on GitHub - 1 maintainer
antlr4-lexer-helper 1.0.4
A helper library with various class to assist in building a custom Antlr4 lexer
5 versions - Latest release: over 5 years ago - 3 dependent packages - 1 dependent repositories - 8 downloads last month - 0 stars on GitHub - 1 maintainer
kuroshiro-browser 0.1.0
Kuroshiro+Analyzer+Kuromoji Combined and Optimized for Browser Usage Only
1 version - Latest release: 24 days ago - 75 downloads last month - 0 stars on GitHub - 1 maintainer
tf_node_tokenizer 1.0.3
tf.keras tokenizer implemented in nodejs
4 versions - Latest release: about 3 years ago - 2 dependent repositories - 14 downloads last month - 8 stars on GitHub - 1 maintainer
Top 4.6% on npmjs.org
megamark 3.3.0
Markdown with easy tokenization, a fast highlighter, and a lean HTML sanitizer
32 versions - Latest release: almost 8 years ago - 9 dependent packages - 46 dependent repositories - 141 downloads last month - 102 stars on GitHub - 1 maintainer
Related Keywords
parser 176 lexer 150 token 84 javascript 70 tokenize 59 ast 51 compiler 49 generator 47 typescript 41 stream 40 nlp 39 react 38 typeahead 35 css 31 scanner 30 mistral 30 string 29 autocomplete 29 chatgpt 29 llama 28 parse 28 parsing 27 nodejs 27 llama2 26 natural language processing 26 chinese 25 tokenization 25 flex 25 html 25 text 25 async 24 split 24 lex 24 NLP 23 word 23 jison 22 gpt4 21 segmentation 21 analyzer 21 llama3 21 baichuan 20 transfomers 20 tokens 20 chatglm 20 css-parser 19 chinese segmentation 18 text mining 18 data 18 dict 18 react-bootstrap-typeahead 18 machine learning 18 δΈ­ζ–‡ 18 segment 18 react-bootstrap-tokenizer 18 optimizer 18 regexp 18 jieba 18 react-bootstrap 18 bootstrap 18 bootstrap-tokenizer 18 bootstrap-typeahead 18 react-component 17 stopword 17 δΈ­ζ–‡εˆ†θ― 17 εˆ†θ― 17 language 17 phrases 17 PanGuSegment 16 variant 16 η›€ε€εˆ†θ©ž 16 η›˜ε€εˆ†θ― 16 sync 16 segmentalization 16 PoS tagging 16 segmenter 16 opencc 16 node-segment 16 novel-segment 16 novel 16 node-novel 16 morphological analysis 16 loader 16 load 16 pangu 16 hanzi 16 file 16 word segmentation 16 css-value-parser 16 cssinjs 16 css-in-js 16 traverser 16 synonym 16 εˆ†θ©ž 16 δΈ­ζ–‡εˆ†θ©ž 16 dictionary 16 bredon 15 簑繁 15 analysis 14 bison 14 create-by-yarn-tool 14