Ecosyste.ms: Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

npmjs.org "tokenize" keyword

parse5-wxmp 3.0.3 💰
HTML parsing/serialization toolset for Node.js. WHATWG HTML Living Standard (aka HTML5)-compliant.
1 version - Latest release: almost 7 years ago - 2 dependent packages - 1 dependent repositories - 12 downloads last month - 3,564 stars on GitHub - 1 maintainer
dqtokenizer 1.0.6
Tokenize strings with double quotes, with embedded escaped, backslashed double quotes
7 versions - Latest release: almost 3 years ago - 1 dependent package - 7 downloads last month - 0 stars on GitHub - 1 maintainer
hx-tokenizer 1.0.0
Split text into tokens.
2 versions - Latest release: over 7 years ago - 1 dependent package - 1 dependent repositories - 5 downloads last month - 2 stars on GitHub - 1 maintainer
trustami-ai 1.0.4
A connector for the Trustami.AI API.
5 versions - Latest release: 9 months ago - 18 downloads last month - 0 stars on GitHub - 1 maintainer
stoke 0.1.2
Generate the Abstract Syntax Tree (AST) of a Bash command.
4 versions - Latest release: over 9 years ago - 1 dependent package - 1 dependent repositories - 20 downloads last month - 15 stars on GitHub - 1 maintainer
rn-braintree 0.2.0
React Native interface to Braintree's Android and iOS SDKs
2 versions - Latest release: over 7 years ago - 1 dependent package - 1 dependent repositories - 6 downloads last month - 1 stars on GitHub - 1 maintainer
Top 1.3% on npmjs.org
js-tokens 9.0.0
Tiny JavaScript tokenizer.
23 versions - Latest release: 4 months ago - 2,251 dependent packages - 1,940,799 dependent repositories - 205 million downloads last month - 482 stars on GitHub - 1 maintainer
html-tree 1.0.0
transform stream to parse html into parse-trees
1 version - Latest release: about 9 years ago - 1 dependent package - 1 dependent repositories - 6 downloads last month - 15 stars on GitHub - 1 maintainer
split-words 1.0.2
Easy way to split words
3 versions - Latest release: almost 9 years ago - 3 dependent packages - 5 dependent repositories - 10 downloads last month - 0 stars on GitHub - 1 maintainer
@liquicode/lib-tokenize 0.1.4
A NodeJS library to tokenize strings
5 versions - Latest release: over 3 years ago - 2 dependent packages - 1 dependent repositories - 9 downloads last month - 0 stars on GitHub - 1 maintainer
simple-token 0.1.0
Create a string representation of an object or primative
2 versions - Latest release: over 9 years ago - 1 dependent package - 4 dependent repositories - 2.67 thousand downloads last month - 1 stars on GitHub - 1 maintainer
generic-matcher 1.3.0
General purpose generic matcher
5 versions - Latest release: almost 3 years ago - 1 dependent package - 6 downloads last month - 0 stars on GitHub - 1 maintainer
y-scanner 0.7.9
Simple, but powerful lexical scanner that is a more minimal implementation of X-Scanner
12 versions - Latest release: over 1 year ago - 26 downloads last month - 1 maintainer
lexer-neo 1.0.1
convert a string to list of words
2 versions - Latest release: about 2 years ago - 1 dependent package - 7 downloads last month - 1 maintainer
fcf-framework-core 2.0.14
Basic functions of the fcf framework 2.0
15 versions - Latest release: 4 months ago - 2 dependent packages - 22 downloads last month - 1 stars on GitHub - 1 maintainer
@kessler/tokenize 2.0.0
personal tokenize util
4 versions - Latest release: about 4 years ago - 2 dependent packages - 1 dependent repositories - 6 downloads last month - 3 stars on GitHub - 1 maintainer
deconcat 0.0.1 💰
Tokenize a Concatenation Expression
1 version - Latest release: about 3 years ago - 2 dependent packages - 2 dependent repositories - 5 downloads last month - 0 stars on GitHub - 1 maintainer
jsx-to-hyperscript 2.2.1
A CLI command for converting JSX files into standard JS.
17 versions - Latest release: about 1 year ago - 1 dependent repositories - 16 downloads last month - 1 stars on GitHub - 2 maintainers
json-tokenizer 1.1.0
A streaming JSON tokenizer
3 versions - Latest release: almost 7 years ago - 6 dependent packages - 6 dependent repositories - 10 downloads last month - 5 stars on GitHub - 1 maintainer
Top 9.9% on npmjs.org
tokepi 0.0.3
Tokenizer that transforms a string of sentences into an array of white-space separated strings of...
3 versions - Latest release: about 10 years ago - 2 dependent packages - 10 dependent repositories - 438 downloads last month - 2 stars on GitHub - 1 maintainer
positional-tokenizer 1.0.6
Tokenizes a text using regex rules and returns the tokens with their positions in the text given.
6 versions - Latest release: over 1 year ago - 5 downloads last month - 0 stars on GitHub - 1 maintainer
Top 6.7% on npmjs.org
sql-limiter 3.0.0
Injects and enforces row limits in your SQL statements.
13 versions - Latest release: almost 2 years ago - 2 dependent packages - 11 dependent repositories - 615 downloads last month - 5 stars on GitHub - 1 maintainer
@parsing-kit/lexer 2.3.0-alpha.8
JavaScript library for tokenizing strings with regular expressions
9 versions - Latest release: over 1 year ago - 2 dependent packages - 3 downloads last month - 1 maintainer
prop-replace 0.0.3
Replaces spots in a string with their values in an object. Super simple templating engine.
3 versions - Latest release: almost 4 years ago - 1 dependent package - 3 dependent repositories - 7 downloads last month - 1 stars on GitHub - 1 maintainer
lexis 0.0.4
a very simple lexer
4 versions - Latest release: over 9 years ago - 1 dependent package - 1 dependent repositories - 5 downloads last month - 2 stars on GitHub - 1 maintainer
snapdragon-stack 2.1.0
Snapdragon utility for creating a stack.
4 versions - Latest release: over 6 years ago - 1 dependent package - 10 dependent repositories - 33 downloads last month - 4 stars on GitHub - 1 maintainer
kc-tokenize 1.0.3
[nodejs] Tokenize data.
4 versions - Latest release: about 3 years ago - 1 dependent package - 5 downloads last month - 0 stars on GitHub - 1 maintainer
separo 0.0.2 💰
A Smarter String Split. Preserve Separator and more!
2 versions - Latest release: about 3 years ago - 1 dependent package - 2 dependent repositories - 6 downloads last month - 0 stars on GitHub - 1 maintainer
Top 5.6% on npmjs.org
nlp-js-tools-french 1.0.9
POS Tagger and lemmatizer for javascript
11 versions - Latest release: about 7 years ago - 3 dependent packages - 16 dependent repositories - 428 downloads last month - 36 stars on GitHub - 1 maintainer
Top 1.7% on npmjs.org
mdast-util-from-markdown 2.0.0 💰
mdast utility to parse markdown
30 versions - Latest release: 11 months ago - 315 dependent packages - 144,051 dependent repositories - 30.8 million downloads last month - 188 stars on GitHub - 2 maintainers
ansi-tokenize 0.2.0
transform stream to tokenize ansi into escape codes and literals
2 versions - Latest release: almost 9 years ago - 4 dependent packages - 1 dependent repositories - 10 downloads last month - 0 stars on GitHub - 1 maintainer
selector-checker 0.9.2
Module for fast element to selector matching (with custom :hover, :focus etc. handing)
3 versions - Latest release: over 7 years ago - 1 dependent repositories - 22 downloads last month - 0 stars on GitHub - 1 maintainer
tinylex 0.7.4
A simple iterative lexer written in TypeScript
21 versions - Latest release: about 5 years ago - 1 dependent package - 1 dependent repositories - 25 downloads last month - 3 stars on GitHub - 1 maintainer
Top 7.7% on npmjs.org
complexion-js 0.2.2
Tokenize JavaScript
9 versions - Latest release: about 3 years ago - 2 dependent packages - 42 dependent repositories - 12.3 thousand downloads last month - 1 stars on GitHub - 1 maintainer
jsx2tokens 0.2.9
Tokenizer for js, jsx, (ts, tsx)
11 versions - Latest release: about 1 year ago - 2 dependent packages - 14 downloads last month - 2 stars on GitHub - 1 maintainer
jstransformer-megamark 0.1.0
megamark support for JSTransformers.
2 versions - Latest release: almost 9 years ago - 1 dependent package - 1 dependent repositories - 9 downloads last month - 1 stars on GitHub - 2 maintainers
Top 1.2% on npmjs.org
micromark-core-commonmark 2.0.1 💰
The CommonMark markdown constructs
14 versions - Latest release: about 2 months ago - 34 dependent packages - 49,395 dependent repositories - 20.2 million downloads last month - 1,576 stars on GitHub - 1 maintainer
poketh 0.6.11
string/array poking for pet params
11 versions - Latest release: over 9 years ago - 1 dependent package - 1 dependent repositories - 8 downloads last month - 1 maintainer
3h-ast 0.14.0
A simple AST generator.
25 versions - Latest release: over 2 years ago - 3 dependent packages - 1 dependent repositories - 15 downloads last month - 0 stars on GitHub - 1 maintainer
cljs-tokenizer 0.0.0
Clojurescript tokenizer
1 version - Latest release: over 10 years ago - 1 dependent package - 1 dependent repositories - 3 downloads last month - 0 stars on GitHub - 1 maintainer
Top 1.7% on npmjs.org
lex 1.7.9 deprecated 💰
An elegant armor-plated JavaScript lexer modelled after flex. Easily extensible to tailor to your...
21 versions - Latest release: almost 9 years ago - 53 dependent packages - 972 dependent repositories - 112 thousand downloads last month - 410 stars on GitHub - 1 maintainer
Top 8.9% on npmjs.org
nlp-toolkit 0.2.6
This module covers some basic nlp principles and implementations. Every implementation in this mo...
8 versions - Latest release: about 8 years ago - 2 dependent packages - 3 dependent repositories - 62 downloads last month - 49 stars on GitHub - 1 maintainer
Top 9.2% on npmjs.org
happynodetokenizer 7.1.0 💰
A simple, Twitter-aware tokenizer.
29 versions - Latest release: 3 months ago - 11 dependent packages - 26 dependent repositories - 236 downloads last month - 5 stars on GitHub - 1 maintainer
Top 1.5% on npmjs.org
snapdragon 0.12.1
Easy-to-use plugin system for creating powerful, fast and versatile parsers and compilers, with b...
33 versions - Latest release: about 3 years ago - 943 dependent packages - 3,779,990 dependent repositories - 47.9 million downloads last month - 219 stars on GitHub - 3 maintainers
Top 1.1% on npmjs.org
micromark-util-character 2.1.0 💰
micromark utility to handle character codes
10 versions - Latest release: 4 months ago - 82 dependent packages - 49,741 dependent repositories - 28 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.4% on npmjs.org
micromark-util-encode 2.0.0 💰
micromark utility to encode dangerous html characters
8 versions - Latest release: 12 months ago - 16 dependent packages - 49,516 dependent repositories - 21.5 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.0% on npmjs.org
micromark 4.0.0 💰
small commonmark compliant markdown parser with positional info and concrete tokens
40 versions - Latest release: 12 months ago - 144 dependent packages - 151,327 dependent repositories - 33 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.1% on npmjs.org
micromark-util-symbol 2.0.0 💰
micromark utility with symbols
8 versions - Latest release: 12 months ago - 86 dependent packages - 49,751 dependent repositories - 29.9 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.4% on npmjs.org
micromark-util-decode-string 2.0.0 💰
micromark utility to decode markdown strings
5 versions - Latest release: 12 months ago - 17 dependent packages - 48,161 dependent repositories - 20.6 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.2% on npmjs.org
micromark-util-chunked 2.0.0 💰
micromark utility to splice and push with giant arrays
7 versions - Latest release: 12 months ago - 33 dependent packages - 49,418 dependent repositories - 21 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.4% on npmjs.org
micromark-factory-destination 2.0.0 💰
micromark factory to parse destinations (found in resources, definitions)
7 versions - Latest release: 12 months ago - 15 dependent packages - 49,391 dependent repositories - 20.6 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.4% on npmjs.org
micromark-factory-label 2.0.0 💰
micromark factory to parse labels (found in media, definitions)
9 versions - Latest release: 12 months ago - 15 dependent packages - 49,391 dependent repositories - 20.6 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.4% on npmjs.org
micromark-util-subtokenize 2.0.1 💰
micromark utility to tokenize subtokens
10 versions - Latest release: about 2 months ago - 16 dependent packages - 49,392 dependent repositories - 20.8 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.3% on npmjs.org
micromark-util-resolve-all 2.0.0 💰
micromark utility to resolve subtokens
7 versions - Latest release: 12 months ago - 26 dependent packages - 49,418 dependent repositories - 20.7 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.4% on npmjs.org
micromark-util-decode-numeric-character-reference 2.0.1 💰
micromark utility to decode numeric character references
5 versions - Latest release: 7 months ago - 17 dependent packages - 49,462 dependent repositories - 20.8 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.3% on npmjs.org
micromark-util-sanitize-uri 2.0.0 💰
micromark utility to sanitize urls
8 versions - Latest release: 12 months ago - 26 dependent packages - 49,519 dependent repositories - 21.6 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.3% on npmjs.org
micromark-util-classify-character 2.0.0 💰
micromark utility to classify whether a character is whitespace or punctuation
7 versions - Latest release: 12 months ago - 22 dependent packages - 49,415 dependent repositories - 20.7 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.4% on npmjs.org
micromark-util-html-tag-name 2.0.0 💰
micromark utility with list of html tag names
8 versions - Latest release: 12 months ago - 14 dependent packages - 49,392 dependent repositories - 20.6 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.2% on npmjs.org
micromark-factory-whitespace 2.0.0 💰
micromark factory to parse markdown whitespace (found in lots of places)
7 versions - Latest release: 12 months ago - 31 dependent packages - 49,517 dependent repositories - 20.6 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.3% on npmjs.org
micromark-util-combine-extensions 2.0.0 💰
micromark utility to combine syntax or html extensions
7 versions - Latest release: 12 months ago - 29 dependent packages - 49,411 dependent repositories - 20.6 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.2% on npmjs.org
micromark-factory-space 2.0.0 💰
micromark factory to parse markdown space (found in lots of places)
7 versions - Latest release: 12 months ago - 57 dependent packages - 49,586 dependent repositories - 23.5 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.1% on npmjs.org
micromark-util-types 2.0.0 💰
micromark utility with a couple of typescript types
9 versions - Latest release: 12 months ago - 95 dependent packages - 49,745 dependent repositories - 25.3 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.2% on npmjs.org
micromark-util-normalize-identifier 2.0.0 💰
micromark utility normalize identifiers (as found in references, definitions)
7 versions - Latest release: 12 months ago - 28 dependent packages - 49,409 dependent repositories - 20.7 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 1.4% on npmjs.org
micromark-factory-title 2.0.0 💰
micromark factory to parse markdown titles (found in resources, definitions)
9 versions - Latest release: 12 months ago - 15 dependent packages - 49,391 dependent repositories - 20.6 million downloads last month - 1,576 stars on GitHub - 1 maintainer
Top 2.4% on npmjs.org
extract-comments 1.1.0 💰
Uses esprima to extract line and block comments from a string of JavaScript. Also optionally pars...
27 versions - Latest release: over 5 years ago - 88 dependent packages - 2,604 dependent repositories - 195 thousand downloads last month - 46 stars on GitHub - 1 maintainer
Top 5.5% on npmjs.org
complexion 0.2.2
A fast tokenizer/lexer core that is well tested.
7 versions - Latest release: about 3 years ago - 5 dependent packages - 44 dependent repositories - 12.4 thousand downloads last month - 4 stars on GitHub - 1 maintainer
Top 6.3% on npmjs.org
@stdlib/string-base-format-interpolate 0.2.1 💰
Generate string from a token array by interpolating values.
8 versions - Latest release: 4 months ago - 4 dependent packages - 801 dependent repositories - 2.49 million downloads last month - 1 stars on GitHub - 4 maintainers
Top 5.7% on npmjs.org
@stdlib/string-base-format-tokenize 0.2.1 💰
Tokenize a string into an array of string parts and format identifier objects.
8 versions - Latest release: 4 months ago - 5 dependent packages - 801 dependent repositories - 2.49 million downloads last month - 2 stars on GitHub - 4 maintainers
tokenize-sync 2.0.0 💰
Simple synchronous string tokenizer using Regex
2 versions - Latest release: over 6 years ago - 3 dependent packages - 2 dependent repositories - 29 downloads last month - 1 stars on GitHub - 1 maintainer
selector-tokenizer 0.9.5
Module for fast css selector tokenization from string
5 versions - Latest release: over 7 years ago - 1 dependent package - 1 dependent repositories - 11 downloads last month - 0 stars on GitHub - 1 maintainer
glsl-token-transpile-300 1.1.1
transpiles GLSL tokens from v100 to v300 es
6 versions - Latest release: over 8 years ago - 1 dependent package - 1 dependent repositories - 8 downloads last month - 39 stars on GitHub - 1 maintainer
c-tokenizer 1.0.0
tokenize c/c++ source code
2 versions - Latest release: over 7 years ago - 4 dependent packages - 3 dependent repositories - 8 downloads last month - 44 stars on - 1 maintainer
count-pieces 0.1.0
Gives you an inventory of tokens needed to write a given program
1 version - Latest release: over 7 years ago - 2 dependent packages - 1 dependent repositories - 14 downloads last month - 0 stars on GitHub - 1 maintainer
parse-regex 0.1.0-beta 💰
Parse regular expressions into an AST.
1 version - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 5 downloads last month - 8 stars on GitHub - 1 maintainer
js-akashcry-token 5.0.0
A regex that tokenizes JavaScript.
2 versions - Latest release: 3 months ago - 5 downloads last month - 0 stars on GitHub - 1 maintainer
Top 3.3% on npmjs.org
micromark-build 2.0.2 💰
Build production files from dev files
12 versions - Latest release: 12 months ago - 23 dependent packages - 30 dependent repositories - 166 downloads last month - 1,576 stars on GitHub - 1 maintainer
complexion-time-map 1.1.0
Time a JavaScript tokenizer
4 versions - Latest release: almost 7 years ago - 2 dependent packages - 4 dependent repositories - 13 downloads last month - 0 stars on GitHub - 1 maintainer
snapdragon-location 1.0.2
Adds a location object to snapdragon token or AST node.
3 versions - Latest release: over 6 years ago - 1 dependent repositories - 10 downloads last month - 2 stars on GitHub - 1 maintainer
extract-comment 1.1.1 💰
Uses esprima to extract line and block comments from a string of JavaScript. Also optionally pars...
2 versions - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 6 downloads last month - 46 stars on GitHub - 1 maintainer
json-tokenize 0.3.0
Splits a JSON string into an annotated list of tokens
3 versions - Latest release: about 7 years ago - 2 dependent packages - 1 dependent repositories - 14 downloads last month - 63 stars on GitHub - 1 maintainer
wat-tokenizer 0.3.0 💰
Tokenize WebAssembly Text-format into list of lists
2 versions - Latest release: over 6 years ago - 1 dependent repositories - 9 downloads last month - 8 stars on GitHub - 1 maintainer
token-sort 1.0.4
tokenize sort string
5 versions - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 12 downloads last month - 1 stars on GitHub - 1 maintainer
@p0n3/excel-formula-tokenizer 2.3.6
Tokenize Excel formulas
3 versions - Latest release: about 2 years ago - 2 dependent packages - 51 downloads last month - 0 stars on GitHub - 1 maintainer
overview-js-tokenizer 0.1.0
Splits text into tokens in as many languages as possible, using libicu
3 versions - Latest release: almost 9 years ago - 1 dependent package - 1 dependent repositories - 29 downloads last month - 3 stars on GitHub - 1 maintainer
@axontraining-sistemas/tokenize-card 1.0.2
Package for card's tokenization
3 versions - Latest release: almost 2 years ago - 4 downloads last month - 1 maintainer
prism-liquibase 0.0.4
Bash Language Extension for tokenizing Liquibase CLI command keywords.
4 versions - Latest release: about 4 years ago - 2 dependent packages - 8 downloads last month - 5 stars on GitHub - 1 maintainer
tokenize-monster 0.3.4
Yet another powerful tokenizer in js
8 versions - Latest release: almost 7 years ago - 1 dependent package - 2 dependent repositories - 12 downloads last month - 0 stars on GitHub - 1 maintainer
jsx-convert 1.1.7
A CLI tool for converting JSX files into standard JS.
21 versions - Latest release: over 1 year ago - 36 downloads last month - 1 maintainer
simple-text2word-tokens 0.0.2
simple text to word tokenizer. accepts a custom regex. returns an array of tokens
2 versions - Latest release: about 5 years ago - 1 dependent repositories - 8 downloads last month - 3 stars on GitHub - 1 maintainer
truly-unique 2.0.0
Simple utility to get unique / repeated words from strings or phrases
4 versions - Latest release: over 5 years ago - 2 dependent repositories - 17 downloads last month - 0 stars on GitHub - 1 maintainer
tokenize-insa-station-name 1.1.0 💰
Get search tokens from an INSA/NASA station name.
2 versions - Latest release: over 4 years ago - 1 dependent package - 5 downloads last month - 0 stars on GitHub - 1 maintainer
@laosb/dioscuri 1.1.0-laosbpatch.0 💰
A gemtext (`text/gemini`) parser with support for streaming, ASTs, and CSTs
1 version - Latest release: almost 3 years ago - 1 dependent package - 6 downloads last month - 39 stars on GitHub - 1 maintainer
tokenize-stache 0.1.0
transform stream to tokenize mustache
1 version - Latest release: about 9 years ago - 2 dependent packages - 2 dependent repositories - 18 downloads last month - 0 stars on GitHub - 1 maintainer
snapdragon-scanner 1.0.0
Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.
1 version - Latest release: over 5 years ago - 1 dependent repositories - 7 downloads last month - 7 stars on GitHub - 1 maintainer
tokenize-file 0.1.7
Read a file, tokenize it, and spit out a handy JSON.
8 versions - Latest release: over 6 years ago - 1 dependent package - 8 dependent repositories - 11 downloads last month - 1 stars on GitHub - 1 maintainer
dioscuri 1.3.0 💰
A gemtext (`text/gemini`) parser with support for streaming, ASTs, and CSTs
5 versions - Latest release: over 1 year ago - 1 dependent package - 2 dependent repositories - 15 downloads last month - 39 stars on GitHub - 1 maintainer
qb-json-tok 2.2.2
Fast (~300 MB/sec) and light (~1.3 kb) JSON/UTF-8 tokenizer for creating custom parsers
17 versions - Latest release: about 7 years ago - 2 dependent packages - 1 dependent repositories - 33 downloads last month - 0 stars on GitHub - 1 maintainer
sever-tokens 0.1.5
A moo compatible tokenizer/lexer generator, sacrificing some performance for features.
6 versions - Latest release: 9 months ago - 6 downloads last month - 2 stars on GitHub - 1 maintainer
sentence-tokenization 1.0.2
Tokenizer for tokenizing sentences, for BERT or other NLP preprocessing.
3 versions - Latest release: about 5 years ago - 2 dependent repositories - 23 downloads last month - 5 stars on GitHub - 1 maintainer
Top 9.4% on npmjs.org
tokenize 0.0.3
Tokenize a string with regex.
4 versions - Latest release: almost 10 years ago - 2 dependent packages - 10 dependent repositories - 344 downloads last month - 1 stars on GitHub - 1 maintainer
Related Keywords
parse 68 tokenizer 59 token 57 lexer 36 ast 35 parser 34 markdown 28 render 28 compile 28 gfm 24 cst 24 javascript 23 html 23 unified 22 commonmark 22 micromark 21 tokens 20 util 20 utility 19 string 19 lex 18 nodejs 16 nlp 15 node 14 text 13 scanner 11 transform 11 js 9 lexical 9 convert 9 code 8 regex 8 compiler 8 tokenization 8 split 8 analysis 8 parsing 8 css 7 syntax 7 comment 6 json 6 stream 6 snapdragon 6 fast 6 tokenise 6 jonschlinkert 6 scan 6 search 5 factory 5 tokeniser 5 replace 5 tags 5 htmlparser 5 html parser 5 serialize 5 html5 5 typescript 5 html5 parser 5 glslify 4 stackgl 4 glsl 4 mdast 4 jsx 4 character 4 natural-language-processing 4 word 4 words 4 normalize 4 JSON 4 language 4 cli 4 unist 4 NLP 4 extract 4 glob 4 es 4 regexp 4 whitespace 3 stemmer 3 types 3 sax 3 jsx-transform 3 tagger 3 html serializer 3 str 3 hyperscript 3 utilities 3 htmlserializer 3 block 3 comments 3 parse5 3 context 3 serializer 3 tokenized 3 keywords 3 flex 3 frequency 3 specification 3 stem 3 replacer 3