packagist.org "tokenizer" keyword
View the packages on the packagist.org package registry that are tagged with the "tokenizer" keyword.
megao-phpunit/php-token-stream 4.0.4 💰
Wrapper around PHP's tokenizer extension.36 versions - Latest release: about 5 years ago - 44 downloads total - 0 stars on GitHub - 1 maintainer
Top 0.5% on packagist.org
9 versions - Latest release: over 1 year ago - 15 dependent packages - 453,841 dependent repositories - 674 million downloads total - 5,119 stars on GitHub - 1 maintainer
theseer/tokenizer 1.2.3 💰
A small library for converting tokenized PHP source code into XML and potentially other formats9 versions - Latest release: over 1 year ago - 15 dependent packages - 453,841 dependent repositories - 674 million downloads total - 5,119 stars on GitHub - 1 maintainer
jundayw/tokenizer
Laravel Tokenizer provides a multi-role, multi-scenario authentication system for SPAs and APIs.Latest release: about 7 hours ago - 0 downloads total
Top 0.3% on packagist.org
15 versions - Latest release: over 7 years ago - 325 dependent packages - 5,002 dependent repositories - 59.5 million downloads total - 1,720 stars on GitHub - 1 maintainer
jeremeamia/superclosure 2.4.0
Serialize Closure objects, including their context and binding15 versions - Latest release: over 7 years ago - 325 dependent packages - 5,002 dependent repositories - 59.5 million downloads total - 1,720 stars on GitHub - 1 maintainer
Top 1.2% on packagist.org
46 versions - Latest release: 2 months ago - 15 dependent packages - 744 dependent repositories - 26.8 million downloads total - 187 stars on GitHub - 1 maintainer
mck89/peast v1.17.2
Peast is PHP library that generates AST for JavaScript code46 versions - Latest release: 2 months ago - 15 dependent packages - 744 dependent repositories - 26.8 million downloads total - 187 stars on GitHub - 1 maintainer
Top 2.5% on packagist.org
9 versions - Latest release: over 3 years ago - 5 dependent packages - 22 dependent repositories - 572 thousand downloads total - 101 stars on GitHub - 1 maintainer
chriskonnertz/string-calc v2.0.0
StringCalc is a PHP calculator library for mathematical terms (expressions) passed as strings.9 versions - Latest release: over 3 years ago - 5 dependent packages - 22 dependent repositories - 572 thousand downloads total - 101 stars on GitHub - 1 maintainer
Top 0.8% on packagist.org
4 versions - Latest release: about 11 years ago - 43 dependent packages - 2,022 dependent repositories - 1.75 million downloads total - 190 stars on GitHub - 2 maintainers
andrewsville/php-token-reflection 1.4.0
Library emulating the PHP internal reflection using just the tokenized source code.4 versions - Latest release: about 11 years ago - 43 dependent packages - 2,022 dependent repositories - 1.75 million downloads total - 190 stars on GitHub - 2 maintainers
Top 0.2% on packagist.org
37 versions - Latest release: about 5 years ago - 205 dependent packages - 382,265 dependent repositories - 382 million downloads total - 6,441 stars on GitHub - 1 maintainer
phpunit/php-token-stream 4.0.4 💰
Wrapper around PHP's tokenizer extension.37 versions - Latest release: about 5 years ago - 205 dependent packages - 382,265 dependent repositories - 382 million downloads total - 6,441 stars on GitHub - 1 maintainer
Top 1.0% on packagist.org
8 versions - Latest release: over 3 years ago - 103 dependent packages - 387 dependent repositories - 4.49 million downloads total - 140 stars on GitHub - 1 maintainer
nette/tokenizer v3.1.1 💰
Nette Tokenizer8 versions - Latest release: over 3 years ago - 103 dependent packages - 387 dependent repositories - 4.49 million downloads total - 140 stars on GitHub - 1 maintainer
Top 10.0% on packagist.org
6 versions - Latest release: about 9 years ago - 1 dependent package - 5 dependent repositories - 77.5 thousand downloads total - 2 stars on GitHub - 1 maintainer
popsul/php-token-reflection 1.5.8
Library emulating the PHP internal reflection using just the tokenized source code.6 versions - Latest release: about 9 years ago - 1 dependent package - 5 dependent repositories - 77.5 thousand downloads total - 2 stars on GitHub - 1 maintainer
attla/data-token
Turn everything into a token: randomized or predictable.Latest release: 2 days ago - 4 dependent repositories - 21 downloads total - 1 stars on GitHub - 1 maintainer
gorriecoe/silverstripe-sreg 1.2.2
Simple tokenizer that allows you to use .ss template like variables in a dataobjects string.7 versions - Latest release: over 7 years ago - 1 dependent package - 135 downloads total - 1 stars on GitHub - 1 maintainer
hexydec/htmldoc 1.8.7
A token based HTML document parser and minifier. Minify HTML documents including inline CSS, Java...30 versions - Latest release: about 2 months ago - 1 dependent package - 4 dependent repositories - 8.28 thousand downloads total - 22 stars on GitHub - 1 maintainer
ghostwriter/syntax removed 💰
Lexical Analysis library for PHPLatest release: 11 months ago - 0 downloads total - 2 stars on GitHub - 1 maintainer
chinleung/tokenizer 1.0.2
PHP Tokenizer3 versions - Latest release: about 9 years ago - 591 downloads total - 1 maintainer
dennis-de-swart/php-stanford-nlp-datastore 2.2.0
Stores NLP data from Stanford CoreNLP server.3 versions - Latest release: over 8 years ago - 70 downloads total - 5 stars on GitHub - 1 maintainer
phplrt/lexer-contracts 3.7.5
A set of phplrt lexer abstractions58 versions - Latest release: 12 months ago - 8 dependent packages - 9 dependent repositories - 147 thousand downloads total - 1 stars on GitHub - 1 maintainer
phplrt/lexer 3.7.5
The phplrt lexer library that can be used for scanners implementation using PCRE61 versions - Latest release: 10 months ago - 9 dependent packages - 11 dependent repositories - 136 thousand downloads total - 3 stars on GitHub - 1 maintainer
fenom/tokenizer 1.0.0
Stand-alone Fenom's tokenizer1 version - Latest release: over 9 years ago - 24 downloads total - 2 stars on GitHub - 1 maintainer
Top 2.5% on packagist.org
5 versions - Latest release: almost 10 years ago - 25 dependent packages - 69 dependent repositories - 984 thousand downloads total - 39 stars on GitHub - 1 maintainer
nilportugues/sql-query-formatter v1.2.2
A very lightweight PHP class that reformats unreadable and computer-generated SQL query statement...5 versions - Latest release: almost 10 years ago - 25 dependent packages - 69 dependent repositories - 984 thousand downloads total - 39 stars on GitHub - 1 maintainer
masnathan/sql-query-formatter v1.2.1
A very lightweight PHP class that reformats unreadable and computer-generated SQL query statement...4 versions - Latest release: over 10 years ago - 1 dependent package - 1.79 thousand downloads total - 0 stars on GitHub - 1 maintainer
thruster/tokenizer 1.0.0
Thruster Tokenizer Component1 version - Latest release: over 8 years ago - 6 downloads total - 0 stars on GitHub - 1 maintainer
apemsel/attributed-string v3.0.0
A collection of fast and easy to use attributed string classes for PHP. Attributed strings can ha...24 versions - Latest release: over 1 year ago - 555 downloads total - 1 stars on GitHub - 1 maintainer
hybridlogic/classifier v1.0.2
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize...3 versions - Latest release: over 12 years ago - 8 dependent repositories - 1.45 thousand downloads total - 80 stars on GitHub - 1 maintainer
dennis-de-swart/php-stanford-corenlp-adapter 7.1.0
PHP adapter for use with Stanford CoreNLP tools4 versions - Latest release: over 6 years ago - 1 dependent package - 1 dependent repositories - 2.86 thousand downloads total - 26 stars on GitHub - 1 maintainer
Top 2.4% on packagist.org
5 versions - Latest release: over 3 years ago - 4 dependent packages - 47 dependent repositories - 1.73 million downloads total - 206 stars on GitHub - 3 maintainers
netgen/query-translator 1.1.0
Query Translator is a search query translator with AST representation5 versions - Latest release: over 3 years ago - 4 dependent packages - 47 dependent repositories - 1.73 million downloads total - 206 stars on GitHub - 3 maintainers
nocika/gpt3-tokenizer
PHP package for Byte Pair Encoding (BPE) used by GPT-3.Latest release: 8 days ago - 15 downloads total - 0 stars on GitHub - 1 maintainer
yep/tokenizer
TokenizerLatest release: 8 days ago - 8 downloads total - 0 stars on GitHub - 1 maintainer
antto1/superclosure v2.4.1
Serialize Closure objects, including their context and binding16 versions - Latest release: almost 5 years ago - 1 dependent package - 160 downloads total - 0 stars on GitHub - 1 maintainer
tanuel/tokenizer v0.1.1
A tokenizer for PHP2 versions - Latest release: almost 6 years ago - 15 downloads total - 6 stars on GitHub - 1 maintainer
virusecks/objectron 0.0.4-alpha
Object creator from template4 versions - Latest release: over 7 years ago - 17 downloads total - 1 stars on GitHub - 1 maintainer
Top 9.3% on packagist.org
11 versions - Latest release: over 5 years ago - 2 dependent packages - 3 dependent repositories - 38.5 thousand downloads total - 12 stars on GitHub - 1 maintainer
funivan/php-tokenizer 0.3.0
Wrapper around token_get_all. Easy to extract and modify php code.11 versions - Latest release: over 5 years ago - 2 dependent packages - 3 dependent repositories - 38.5 thousand downloads total - 12 stars on GitHub - 1 maintainer
tacoberu/bnf v1.2.1
Framework for building BNF like parsers.8 versions - Latest release: 6 months ago - 1 dependent package - 1 dependent repositories - 2.76 thousand downloads total - 5 stars on GitHub - 1 maintainer
robinthehood/tokenizer 0.1.3
With Tokenizer you can easily convert a string to a stream or array of tokens.3 versions - Latest release: about 7 years ago - 2 dependent packages - 3 dependent repositories - 836 downloads total - 0 stars on GitHub - 1 maintainer
linguistic/ngramextractor
Extracts ngrams from a given text and does linguistic pre-processing like stopword removalLatest release: 10 days ago - 2 thousand downloads total - 3 stars on GitHub - 1 maintainer
sastrawi/tokenizer v0.4.0
PHP library that allows you to tokenize Bahasa Indonesia.4 versions - Latest release: over 10 years ago - 12 dependent repositories - 947 downloads total - 34 stars on GitHub - 1 maintainer
td7650/superclosure 2.1.3
Serialize Closure objects, including their context and binding15 versions - Latest release: almost 10 years ago - 1 dependent package - 58 downloads total - 0 stars on GitHub - 1 maintainer
Top 2.9% on packagist.org
95 versions - Latest release: 4 months ago - 39 dependent packages - 109 dependent repositories - 5.42 million downloads total - 17 stars on GitHub - 1 maintainer
spiral/tokenizer 3.15.8 💰
Static Analysis: Class and Invocation locators95 versions - Latest release: 4 months ago - 39 dependent packages - 109 dependent repositories - 5.42 million downloads total - 17 stars on GitHub - 1 maintainer
hike/hike v0.1.1
A consistent interface to tokenize, analyse and modify any text format. Programming languages, na...2 versions - Latest release: 5 months ago - 5 downloads total - 0 stars on GitHub - 1 maintainer
guttedgarden/tiktoken 1.2.1
PHP 7.4 version of tiktoken4 versions - Latest release: 5 months ago - 14.3 thousand downloads total - 4 stars on GitHub - 1 maintainer
your1/tokenizer-x 1.0.1
TokenizerX calculates required tokens for given prompt2 versions - Latest release: 5 months ago - 273 downloads total - 0 stars on GitHub - 1 maintainer
Top 2.0% on packagist.org
34 versions - Latest release: 4 months ago - 83 dependent packages - 180 dependent repositories - 1.03 million downloads total - 32 stars on GitHub - 1 maintainer
fig-r/psr2r-sniffer 2.2.1
Code-Sniffer, Auto-Fixer and Tokenizer for PSR2-R34 versions - Latest release: 4 months ago - 83 dependent packages - 180 dependent repositories - 1.03 million downloads total - 32 stars on GitHub - 1 maintainer
nexxes/tokenmatcher
PHP library to match tokenstreams (arrays of tokens) for later parsing.Latest release: 12 days ago - 4 dependent repositories - 9 downloads total - 0 stars on GitHub - 1 maintainer
didix16/php-grammar 1.0.1
A simple library for make Lexer and Parsers to build a language2 versions - Latest release: about 4 years ago - 2 dependent packages - 2 dependent repositories - 7.67 thousand downloads total - 1 stars on GitHub - 1 maintainer
muhfirdaus19/bahasata
Text Processing PHP bahasa indonesiaLatest release: 12 days ago - 6 downloads total - 1 stars on GitHub - 1 maintainer
phapp/phapp
The PHP Application for WebmastersLatest release: 12 days ago - 2 downloads total - 2 stars on GitHub - 1 maintainer
lalbert/tokenizer-engine 0.0.1
Tokenize string like Elasticsearch1 version - Latest release: over 8 years ago - 18 downloads total - 1 stars on GitHub - 1 maintainer
phpwatch/phptoken-polyfill v0.0.3
A polyfill for PHP 8.0's PhpToken class3 versions - Latest release: almost 5 years ago - 18 downloads total - 9 stars on GitHub - 1 maintainer
octava/code-generator
PHP code generatorLatest release: 14 days ago - 1 downloads total - 0 stars on GitHub - 1 maintainer
Top 8.2% on packagist.org
14 versions - Latest release: over 4 years ago - 5 dependent packages - 9 dependent repositories - 4.66 thousand downloads total - 8 stars on GitHub - 1 maintainer
mathematicator-core/tokenizer v3.0.1 💰
Math Tokenizer14 versions - Latest release: over 4 years ago - 5 dependent packages - 9 dependent repositories - 4.66 thousand downloads total - 8 stars on GitHub - 1 maintainer
affinity4/tokenizer 0.0.4
A zero-dedpendency tokenizer written in PHP. Returns an easily navigatable Stream object of Token...4 versions - Latest release: over 3 years ago - 7 downloads total - 0 stars on GitHub - 1 maintainer
djiele/autoloader 7.3.1-stable
Another class loader using PHP tokenizer1 version - Latest release: about 5 years ago - 11 downloads total - 0 stars on GitHub - 1 maintainer
apicart/fql 0.4.6
Filter Query Language20 versions - Latest release: about 2 months ago - 10.1 thousand downloads total - 11 stars on GitHub - 1 maintainer
html5/htmlreader v2.0.1
Html5 stream tokenizer/reader (not using libxml)4 versions - Latest release: almost 8 years ago - 1 dependent package - 2 dependent repositories - 3.3 thousand downloads total - 1 stars on GitHub - 1 maintainer
guanhui07/tiktoken 1.0.2
PHP version of tiktoken2 versions - Latest release: about 1 year ago - 6 downloads total - 1 stars on GitHub - 1 maintainer
Top 6.3% on packagist.org
6 versions - Latest release: over 11 years ago - 12 dependent packages - 893 dependent repositories - 128 thousand downloads total - 22 stars on GitHub - 1 maintainer
herrera-io/annotations 1.0.1
A tokenizer for Doctrine annotations.6 versions - Latest release: over 11 years ago - 12 dependent packages - 893 dependent repositories - 128 thousand downloads total - 22 stars on GitHub - 1 maintainer
attla/token
Turn everything into a unique encrypted JWT.Latest release: 15 days ago - 3 downloads total - 0 stars on GitHub - 1 maintainer
rahul900day/gpt-3-encoder 1.1.0 💰
GPT-3-Encoder is a PHP implementation of OpenAI's original python encoder/decoder.3 versions - Latest release: over 2 years ago - 2 dependent repositories - 30 thousand downloads total - 2 stars on GitHub - 1 maintainer
xp-framework/tokenize v9.1.0
Text tokenizing11 versions - Latest release: over 1 year ago - 9 dependent packages - 5 dependent repositories - 170 thousand downloads total - 0 stars on GitHub - 1 maintainer
mathematicator-core/engine v3.0.1 💰
Math computational Engine19 versions - Latest release: over 4 years ago - 5 dependent packages - 7 dependent repositories - 3.52 thousand downloads total - 2 stars on GitHub - 1 maintainer
Top 9.0% on packagist.org
16 versions - Latest release: 11 months ago - 4 dependent packages - 16 dependent repositories - 209 thousand downloads total - 2 stars on GitHub - 2 maintainers
phootwork/tokenizer v3.2.3
Tokenizer for PHP code16 versions - Latest release: 11 months ago - 4 dependent packages - 16 dependent repositories - 209 thousand downloads total - 2 stars on GitHub - 2 maintainers
scripturadesign/tokenizer
TokenizerLatest release: 16 days ago - 7 downloads total - 0 stars on GitHub - 1 maintainer
rudl/logger v0.8
Rudl UDP cluster logging library (PSR4 compliant)7 versions - Latest release: about 8 years ago - 2.71 thousand downloads total - 0 stars on GitHub - 1 maintainer
yuubit/tokenizer 1.0.0
PHP Library to tokenize various inputs.1 version - Latest release: over 6 years ago - 1 dependent package - 1 dependent repositories - 403 downloads total - 0 stars on GitHub - 1 maintainer
teepluss/wordbreaker v0.1
Word breaker is Thai wording separator for using with fulltext software.1 version - Latest release: almost 11 years ago - 4 dependent repositories - 26 downloads total - 4 stars on GitHub - 1 maintainer
mistralys/jtokenizer 1.0.1
Modernized, composer compatible version of Tim Whitlock's JTokenizer PHP Javascript parser and to...2 versions - Latest release: over 4 years ago - 1 dependent package - 6 dependent repositories - 6.77 thousand downloads total - 2 stars on GitHub - 1 maintainer
yurunsoft/tiktoken v0.1.4
PHP version of tiktoken2 versions - Latest release: almost 2 years ago - 1 dependent repositories - 241 downloads total - 0 stars on GitHub - 1 maintainer
hippophp/hippo v0.1.0
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.1 version - Latest release: over 9 years ago - 4 dependent repositories - 32 downloads total - 80 stars on GitHub - 1 maintainer
scheldejonas/phpunit-php-token-stream 4.0.5 💰
Wrapper around PHP's tokenizer extension.36 versions - Latest release: almost 5 years ago - 208 downloads total - 0 stars on GitHub - 1 maintainer
fi1a/tokenizer 1.1.7
Лексический анализатор строк (tokenizer). Разбор входной последовательности символов на распознан...11 versions - Latest release: over 2 years ago - 5 dependent packages - 5 dependent repositories - 7.81 thousand downloads total - 0 stars on GitHub - 1 maintainer
ze/tokenizer-gpt3 v1.1
PHP package for Byte Pair Encoding (BPE) used by GPT-3.2 versions - Latest release: over 2 years ago - 1.49 thousand downloads total - 1 stars on GitHub - 1 maintainer
dbeurive/lexer 1.0.4
This project implements a simple lexer.5 versions - Latest release: almost 9 years ago - 1 dependent package - 1 dependent repositories - 2.41 thousand downloads total - 2 stars on GitHub - 1 maintainer
brad-jones/superclosure v2.0.0
Doing interesting things with closures like serialization.9 versions - Latest release: almost 11 years ago - 2 dependent packages - 5 dependent repositories - 236 downloads total - 1 maintainer
devtronic/super-tokenizer 1.0.1 💰
A powerful dynamic tokenizer written in PHP2 versions - Latest release: over 8 years ago - 65 downloads total - 1 stars on GitHub - 1 maintainer
kevintweber/html-tokenizer v0.3.1
Will tokenize HTML.9 versions - Latest release: over 8 years ago - 1 dependent package - 2 dependent repositories - 47.6 thousand downloads total - 5 stars on GitHub - 1 maintainer
lahaxearnaud/laravel-token v0.5.1
Laravel 4 token management6 versions - Latest release: over 10 years ago - 4 dependent repositories - 395 downloads total - 10 stars on GitHub - 1 maintainer
secondtruth/lexer 💰
Base library for simple lexersLatest release: 22 days ago - 0 downloads total - 2 stars on GitHub - 1 maintainer
Top 7.0% on packagist.org
14 versions - Latest release: 24 days ago - 5 dependent packages - 1 dependent repositories - 1.73 million downloads total - 137 stars on GitHub - 1 maintainer
yethee/tiktoken 0.11.0
PHP version of tiktoken14 versions - Latest release: 24 days ago - 5 dependent packages - 1 dependent repositories - 1.73 million downloads total - 137 stars on GitHub - 1 maintainer
mathewberry/tokenizer 0.2.4
A very lightweight token system for micro-services6 versions - Latest release: about 8 years ago - 770 downloads total - 1 stars on GitHub - 1 maintainer
angkor/khmercut 0.2.0
Khmer Tokenization4 versions - Latest release: 4 months ago - 219 downloads total - 0 stars on GitHub - 1 maintainer
nexxes/tokenizer
Simple tokenizer to split a character steam (text) into character classes (tokens) written in PHPLatest release: 22 days ago - 6 dependent repositories - 40 downloads total - 1 stars on GitHub - 1 maintainer
interitty/tokenizer v1.0.9
Use regular expressions to split a given string into tokens.10 versions - Latest release: 4 months ago - 1 dependent package - 1 dependent repositories - 42 downloads total - 1 stars on gitlab.com - 1 maintainer
fourkitchens/tiktoken
PHP version of tiktokenLatest release: 23 days ago - 0 downloads total - 0 stars on GitHub - 1 maintainer
vanderlee/comprehend 1.0.2
Framework for building BNF LR(1) parsers3 versions - Latest release: almost 7 years ago - 17 downloads total - 7 stars on GitHub - 1 maintainer
everstu/gpt3-tokenizer v1.0.0
PHP package for Byte Pair Encoding (BPE) used by GPT-3.1 version - Latest release: over 2 years ago - 149 downloads total - 0 stars on GitHub - 1 maintainer
bermudaphp/tokenizer v1.0
PHP tokenizer for finding class, interface, trait, and enum declarations1 version - Latest release: 3 months ago - 0 downloads total - 0 stars on GitHub - 1 maintainer
kornrunner/php-token-reflection v1.5.6
Library emulating the PHP internal reflection using just the tokenized source code.11 versions - Latest release: over 9 years ago - 6.97 thousand downloads total - 1 stars on GitHub - 1 maintainer
Top 3.5% on packagist.org
21 versions - Latest release: over 4 years ago - 6 dependent packages - 7 dependent repositories - 70.8 thousand downloads total - 129 stars on GitHub - 1 maintainer
nicoswd/php-rule-parser 0.7.1
Rule Engine - Rule Parser & Evaluator21 versions - Latest release: over 4 years ago - 6 dependent packages - 7 dependent repositories - 70.8 thousand downloads total - 129 stars on GitHub - 1 maintainer
turanjanin/serbian-language-tools 1.0.3
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Ser...4 versions - Latest release: 24 days ago - 2.04 thousand downloads total - 17 stars on GitHub - 1 maintainer
taufik-nurrohman/minify v1.0.3 💰
Reduce CSS, HTML, JavaScript, JSON, PHP, and XML code size on the server side.4 versions - Latest release: 8 months ago - 5 downloads total - 3 stars on GitHub - 1 maintainer
danny50610/bpe-tokeniser 0.2.0
PHP port for openai/tiktoken (most)2 versions - Latest release: over 1 year ago - 1 dependent package - 286 thousand downloads total - 9 stars on GitHub - 1 maintainer
artemeon/tokenizer 2.0.1
Tokenizer for php4 versions - Latest release: about 3 years ago - 71.9 thousand downloads total - 0 stars on GitHub - 1 maintainer
railt/lexer 1.3.2
Fast implementation of the stateful and stateless lexers4 versions - Latest release: over 6 years ago - 4 dependent packages - 2 dependent repositories - 3.32 thousand downloads total - 6 stars on GitHub - 1 maintainer
legacyphpdoctor/tiktoken-php-7-4
PHP version of tiktokenLatest release: 28 days ago - 1 downloads total - 1 stars on GitHub - 1 maintainer
gossi/tokenizer v0.1
Tokenizer for PHP code1 version - Latest release: almost 11 years ago - 4 dependent repositories - 150 downloads total - 1 maintainer
vanry/tokenizer
支持 scws, phpanalysis 和 jieba 的中文分词器Latest release: 28 days ago - 2 downloads total - 0 stars on GitHub - 1 maintainer
infinityloop-dev/graphpinator-tokenizer v1.1.1
Tokenizer subproject for GraPHPinator: GraphQL server implementation for PHP.6 versions - Latest release: over 4 years ago - 2 dependent packages - 1 dependent repositories - 34 thousand downloads total - 3 stars on GitHub - 1 maintainer
xtreed/php-token-stream 4.0.4 💰
Wrapper around PHP's tokenizer extension.37 versions - Latest release: about 5 years ago - 1 dependent repositories - 3.53 thousand downloads total - 0 stars on GitHub - 1 maintainer
vchagin/superclosure
Serialize Closures, including their contextLatest release: about 1 month ago - 65 downloads total - 1 stars on GitHub - 1 maintainer
rajentrivedi/tokenizer-x v1.0.5
TokenizerX calculates required tokens for given prompt6 versions - Latest release: 7 months ago - 1 dependent package - 120 thousand downloads total - 88 stars on GitHub - 1 maintainer
Related Keywords
php
40
parser
33
token
20
openai
14
lexer
14
encode
12
decode
12
tiktoken
8
GPT-3
8
tokenization
7
bpe
7
laravel
7
function
7
closure
7
tokens
6
nlp
6
ast
6
gpt-2
6
library
5
html
5
serialize
5
serializable
5
parsing
5
query
5
php-library
5
javascript
5
language
4
generator
4
compressor
4
lambda
4
reflection
4
cl100k_base
3
php7
3
archiver
3
compiler
3
php8
3
sikessem
3
formatter
3
regex
3
stream
3
tokenize
3
xml
3
translator
3
search
3
math
3
class
3
ChatGpt
2
edismax
2
object
2
sql server
2
bnf
2
abnf
2
mysql
2
indonesia
2
sql
2
lexical-analysis
2
template
2
pcre
2
phplrt
2
ngram
2
chat-gpt
2
tokenizer-x
2
dall-e
2
codex
2
natural
2
r50k_base
2
processing
2
p50k_base
2
chat
2
p50k_edit
2
lex
2
json
2
solr
2
code-style
2
reformat
2
format
2
elasticsearch
2
mathematicator
2
css
2
ske
2
binary
2
minify
2
minifier
2
validator
2
cli-app
2
HTML5
2
console-application
2
calculator
2
stanford
2
installer
2
runtime
2
openie
2
script-compiler
2
corenlp-server
2
data-to-token
2
lemma
2
attla
2
jwt
2
string
2
setup
2