Ecosyste.ms: Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

packagist.org "tokenizer" keyword

masnathan/sql-query-formatter v1.2.1
A very lightweight PHP class that reformats unreadable and computer-generated SQL query statement...
4 versions - Latest release: almost 9 years ago - 1 dependent package - 1.62 thousand downloads total - 0 stars on GitHub - 1 maintainer
Top 2.5% on packagist.org
nilportugues/sql-query-formatter v1.2.2
A very lightweight PHP class that reformats unreadable and computer-generated SQL query statement...
5 versions - Latest release: over 8 years ago - 25 dependent packages - 69 dependent repositories - 650 thousand downloads total - 36 stars on GitHub - 1 maintainer
Top 1.2% on packagist.org
mck89/peast v1.16.2
Peast is PHP library that generates AST for JavaScript code
42 versions - Latest release: 2 months ago - 14 dependent packages - 744 dependent repositories - 10.8 million downloads total - 165 stars on GitHub - 1 maintainer
Top 8.2% on packagist.org
mathematicator-core/tokenizer v3.0.1 💰
Math Tokenizer
14 versions - Latest release: about 3 years ago - 5 dependent packages - 9 dependent repositories - 4.39 thousand downloads total - 8 stars on GitHub - 1 maintainer
nexxes/tokenizer
Simple tokenizer to split a character steam (text) into character classes (tokens) written in PHP
Latest release: 2 days ago - 6 dependent repositories - 40 downloads total - 1 stars on GitHub - 1 maintainer
html5/htmlreader v2.0.1
Html5 stream tokenizer/reader (not using libxml)
4 versions - Latest release: over 6 years ago - 1 dependent package - 2 dependent repositories - 3.23 thousand downloads total - 1 stars on GitHub - 1 maintainer
fi1a/tokenizer 1.1.7
Лексический анализатор строк (tokenizer). Разбор входной последовательности символов на распознан...
11 versions - Latest release: about 1 year ago - 5 dependent packages - 5 dependent repositories - 2.49 thousand downloads total - 0 stars on GitHub - 1 maintainer
scheldejonas/phpunit-php-token-stream 4.0.5 💰
Wrapper around PHP's tokenizer extension.
36 versions - Latest release: over 3 years ago - 205 downloads total - 0 stars on GitHub - 1 maintainer
artemeon/tokenizer 2.0.1
Tokenizer for php
4 versions - Latest release: over 1 year ago - 42.3 thousand downloads total - 0 stars on GitHub - 1 maintainer
yuubit/tokenizer 1.0.0
PHP Library to tokenize various inputs.
1 version - Latest release: about 5 years ago - 1 dependent package - 1 dependent repositories - 401 downloads total - 0 stars on GitHub - 1 maintainer
attla/data-token
Turn everything into a token: randomized or predictable.
Latest release: 4 days ago - 4 dependent repositories - 16 downloads total - 1 stars on GitHub - 1 maintainer
Top 1.0% on packagist.org
nette/tokenizer v3.1.1 💰
Nette Tokenizer
8 versions - Latest release: about 2 years ago - 100 dependent packages - 387 dependent repositories - 3.44 million downloads total - 140 stars on GitHub - 1 maintainer
kevintweber/html-tokenizer v0.3.1
Will tokenize HTML.
9 versions - Latest release: about 7 years ago - 1 dependent package - 2 dependent repositories - 46 thousand downloads total - 5 stars on GitHub - 1 maintainer
Top 0.3% on packagist.org
jeremeamia/superclosure 2.4.0
Serialize Closure objects, including their context and binding
15 versions - Latest release: about 6 years ago - 320 dependent packages - 5,002 dependent repositories - 56.9 million downloads total - 1,719 stars on GitHub - 1 maintainer
megao-phpunit/php-token-stream 4.0.4 💰
Wrapper around PHP's tokenizer extension.
36 versions - Latest release: almost 4 years ago - 33 downloads total - 0 stars on GitHub - 1 maintainer
mistralys/jtokenizer 1.0.1
Modernized, composer compatible version of Tim Whitlock's JTokenizer PHP Javascript parser and to...
2 versions - Latest release: about 3 years ago - 1 dependent package - 6 dependent repositories - 2.78 thousand downloads total - 2 stars on GitHub - 1 maintainer
teepluss/wordbreaker v0.1
Word breaker is Thai wording separator for using with fulltext software.
1 version - Latest release: over 9 years ago - 4 dependent repositories - 23 downloads total - 4 stars on GitHub - 1 maintainer
Top 0.8% on packagist.org
andrewsville/php-token-reflection 1.4.0
Library emulating the PHP internal reflection using just the tokenized source code.
4 versions - Latest release: almost 10 years ago - 43 dependent packages - 2,022 dependent repositories - 1.65 million downloads total - 189 stars on GitHub - 2 maintainers
infinityloop-dev/graphpinator-tokenizer v1.1.1
Tokenizer subproject for GraPHPinator: GraphQL server implementation for PHP.
6 versions - Latest release: almost 3 years ago - 2 dependent packages - 1 dependent repositories - 17.1 thousand downloads total - 3 stars on GitHub - 1 maintainer
nexxes/tokenmatcher
PHP library to match tokenstreams (arrays of tokens) for later parsing.
Latest release: 6 days ago - 4 dependent repositories - 9 downloads total - 0 stars on GitHub - 1 maintainer
robinthehood/tokenizer 0.1.3
With Tokenizer you can easily convert a string to a stream or array of tokens.
3 versions - Latest release: almost 6 years ago - 2 dependent packages - 3 dependent repositories - 792 downloads total - 0 stars on GitHub - 1 maintainer
Top 0.2% on packagist.org
phpunit/php-token-stream 4.0.4 💰
Wrapper around PHP's tokenizer extension.
37 versions - Latest release: almost 4 years ago - 204 dependent packages - 382,265 dependent repositories - 349 million downloads total - 6,461 stars on GitHub - 1 maintainer
Top 0.5% on packagist.org
theseer/tokenizer 1.2.3 💰
A small library for converting tokenized PHP source code into XML and potentially other formats
9 versions - Latest release: 2 months ago - 15 dependent packages - 453,841 dependent repositories - 473 million downloads total - 5,116 stars on GitHub - 1 maintainer
rahul900day/gpt-3-encoder 1.1.0 💰
GPT-3-Encoder is a PHP implementation of OpenAI's original python encoder/decoder.
3 versions - Latest release: about 1 year ago - 2 dependent repositories - 11.2 thousand downloads total - 2 stars on GitHub - 1 maintainer
taufik-nurrohman/minify 💰
Reduce CSS, HTML, JavaScript, JSON, PHP, and XML code size on the server side.
Latest release: 6 days ago - 1 downloads total - 1 stars on GitHub - 1 maintainer
patchwork/php-parser 💰
Fast and extensible PHP code analysis and transformation
Latest release: 6 days ago - 4 dependent repositories - 26 downloads total - 4 stars on GitHub - 1 maintainer
devtronic/super-tokenizer 1.0.1 💰
A powerful dynamic tokenizer written in PHP
2 versions - Latest release: about 7 years ago - 59 downloads total - 1 stars on GitHub - 1 maintainer
tacoberu/bnf v1.2.0
Framework for building BNF like parsers.
7 versions - Latest release: about 1 year ago - 1 dependent package - 1 dependent repositories - 2.13 thousand downloads total - 6 stars on GitHub - 1 maintainer
danny50610/bpe-tokeniser 0.1.0
PHP port for openai/tiktoken (most)
1 version - Latest release: 9 months ago - 20 thousand downloads total - 4 stars on GitHub - 1 maintainer
dennis-de-swart/php-stanford-corenlp-adapter 7.1.0
PHP adapter for use with Stanford CoreNLP tools
4 versions - Latest release: over 5 years ago - 1 dependent package - 1 dependent repositories - 2.58 thousand downloads total - 26 stars on GitHub - 1 maintainer
Top 2.9% on packagist.org
spiral/tokenizer 3.12.0 💰
Static Analysis: Class and Invocation locators
74 versions - Latest release: 4 months ago - 37 dependent packages - 109 dependent repositories - 2.64 million downloads total - 15 stars on GitHub - 1 maintainer
Top 10.0% on packagist.org
popsul/php-token-reflection 1.5.8
Library emulating the PHP internal reflection using just the tokenized source code.
6 versions - Latest release: almost 8 years ago - 1 dependent package - 5 dependent repositories - 76.4 thousand downloads total - 2 stars on GitHub - 1 maintainer
railt/lexer 1.3.2
Fast implementation of the stateful and stateless lexers
4 versions - Latest release: over 5 years ago - 4 dependent packages - 2 dependent repositories - 3.3 thousand downloads total - 6 stars on GitHub - 1 maintainer
interitty/tokenizer v1.0.5
Use regular expressions to split a given string into tokens.
6 versions - Latest release: 4 months ago - 1 dependent package - 1 dependent repositories - 17 downloads total - 1 stars on GitLab.com - 1 maintainer
apemsel/attributed-string v3.0.0
A collection of fast and easy to use attributed string classes for PHP. Attributed strings can ha...
24 versions - Latest release: about 1 month ago - 551 downloads total - 1 stars on GitHub - 1 maintainer
Top 7.0% on packagist.org
yethee/tiktoken 0.3.0
PHP version of tiktoken
5 versions - Latest release: 4 months ago - 5 dependent packages - 1 dependent repositories - 268 thousand downloads total - 82 stars on GitHub - 1 maintainer
Top 3.2% on packagist.org
gioni06/gpt3-tokenizer v1.2.0
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
4 versions - Latest release: about 1 year ago - 5 dependent packages - 10 dependent repositories - 191 thousand downloads total - 84 stars on GitHub - 1 maintainer
mathematicator-core/engine v3.0.1 💰
Math computational Engine
19 versions - Latest release: about 3 years ago - 5 dependent packages - 7 dependent repositories - 3.49 thousand downloads total - 2 stars on GitHub - 1 maintainer
phplrt/lexer 3.6.5
The phplrt lexer library that can be used for scanners implementation using PCRE
54 versions - Latest release: 3 months ago - 9 dependent packages - 11 dependent repositories - 76.9 thousand downloads total - 3 stars on GitHub - 1 maintainer
apicart/fql 0.4.5
Filter Query Language
19 versions - Latest release: 5 months ago - 8.59 thousand downloads total - 10 stars on GitHub - 1 maintainer
rajentrivedi/tokenizer-x 1.0.2
TokenizerX calculates required tokens for given prompt
3 versions - Latest release: 4 months ago - 19.9 thousand downloads total - 50 stars on GitHub - 1 maintainer
Top 9.0% on packagist.org
phootwork/tokenizer v3.2.2
Tokenizer for PHP code
15 versions - Latest release: over 1 year ago - 4 dependent packages - 16 dependent repositories - 205 thousand downloads total - 2 stars on GitHub - 2 maintainers
sastrawi/tokenizer v0.4.0
PHP library that allows you to tokenize Bahasa Indonesia.
4 versions - Latest release: over 9 years ago - 12 dependent repositories - 939 downloads total - 34 stars on GitHub - 1 maintainer
antto1/superclosure v2.4.1
Serialize Closure objects, including their context and binding
16 versions - Latest release: over 3 years ago - 1 dependent package - 144 downloads total - 0 stars on GitHub - 1 maintainer
yurunsoft/tiktoken v0.1.4
PHP version of tiktoken
2 versions - Latest release: 8 months ago - 1 dependent repositories - 189 downloads total - 0 stars on GitHub - 1 maintainer
fenom/tokenizer 1.0.0
Stand-alone Fenom's tokenizer
1 version - Latest release: over 8 years ago - 20 downloads total - 2 stars on GitHub - 1 maintainer
ghostwriter/syntax 💰
Lexical Analysis library for PHP
Latest release: 11 days ago - 0 downloads total - 1 stars on GitHub - 1 maintainer
Top 6.3% on packagist.org
herrera-io/annotations 1.0.1
A tokenizer for Doctrine annotations.
6 versions - Latest release: over 10 years ago - 12 dependent packages - 893 dependent repositories - 126 thousand downloads total - 22 stars on GitHub - 1 maintainer
Top 2.0% on packagist.org
fig-r/psr2r-sniffer 2.1.0
Code-Sniffer, Auto-Fixer and Tokenizer for PSR2-R
30 versions - Latest release: about 1 month ago - 78 dependent packages - 180 dependent repositories - 701 thousand downloads total - 32 stars on GitHub - 1 maintainer
phonyland/ngram
🧪 N-Gram Tools for 🙃 Phony Language Models with sanitizing, tokenization, n-gram extraction, freq...
Latest release: 13 days ago - 1 dependent repositories - 1.1 thousand downloads total - 2 stars on GitHub - 1 maintainer
xp-framework/tokenize v9.1.0
Text tokenizing
11 versions - Latest release: about 2 months ago - 9 dependent packages - 5 dependent repositories - 87.6 thousand downloads total - 0 stars on GitHub - 1 maintainer
gorriecoe/silverstripe-sreg 1.2.2
Simple tokenizer that allows you to use .ss template like variables in a dataobjects string.
7 versions - Latest release: about 6 years ago - 1 dependent package - 131 downloads total - 1 stars on GitHub - 1 maintainer
cxrxexo/tiktoken 1.0.1
PHP version of tiktoken. PHP 7.4+ required.
2 versions - Latest release: 11 months ago - 1.9 thousand downloads total - 2 stars on GitHub - 1 maintainer
chinleung/tokenizer 1.0.2
PHP Tokenizer
3 versions - Latest release: over 7 years ago - 591 downloads total - 1 maintainer
phpwatch/phptoken-polyfill v0.0.3
A polyfill for PHP 8.0's PhpToken class
3 versions - Latest release: over 3 years ago - 16 downloads total - 7 stars on GitHub - 1 maintainer
guttedgarden/tiktoken 1.0.0
PHP 7.4 version of tiktoken
1 version - Latest release: 7 months ago - 4.88 thousand downloads total - 3 stars on GitHub - 1 maintainer
rudl/logger v0.8
Rudl UDP cluster logging library (PSR4 compliant)
7 versions - Latest release: over 6 years ago - 2.25 thousand downloads total - 0 stars on GitHub - 1 maintainer
dennis-de-swart/php-stanford-nlp-datastore 2.2.0
Stores NLP data from Stanford CoreNLP server.
3 versions - Latest release: about 7 years ago - 66 downloads total - 5 stars on GitHub - 1 maintainer
Top 2.4% on packagist.org
netgen/query-translator 1.1.0
Query Translator is a search query translator with AST representation
5 versions - Latest release: over 2 years ago - 4 dependent packages - 47 dependent repositories - 1.26 million downloads total - 198 stars on GitHub - 3 maintainers
crazzy501/gpt3-tokenizer 1.0.1
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
2 versions - Latest release: 10 months ago - 6 downloads total - 1 maintainer
djiele/autoloader 7.3.1-stable
Another class loader using PHP tokenizer
1 version - Latest release: almost 4 years ago - 7 downloads total - 0 stars on GitHub - 1 maintainer
scripturadesign/tokenizer
Tokenizer
Latest release: 15 days ago - 7 downloads total - 0 stars on GitHub - 1 maintainer
kornrunner/php-token-reflection v1.5.6
Library emulating the PHP internal reflection using just the tokenized source code.
11 versions - Latest release: about 8 years ago - 6.97 thousand downloads total - 1 stars on GitHub - 1 maintainer
gossi/tokenizer v0.1
Tokenizer for PHP code
1 version - Latest release: over 9 years ago - 4 dependent repositories - 150 downloads total - 1 maintainer
everstu/gpt3-tokenizer v1.0.0
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
1 version - Latest release: about 1 year ago - 30 downloads total - 0 stars on GitHub - 1 maintainer
muhfirdaus19/bahasata
Text Processing PHP bahasa indonesia
Latest release: 16 days ago - 6 downloads total - 1 stars on GitHub - 1 maintainer
td7650/superclosure 2.1.3
Serialize Closure objects, including their context and binding
15 versions - Latest release: over 8 years ago - 1 dependent package - 54 downloads total - 0 stars on GitHub - 1 maintainer
yep/tokenizer
Tokenizer
Latest release: 17 days ago - 8 downloads total - 0 stars on GitHub - 1 maintainer
turanjanin/serbian-language-tools 1.0.2
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Ser...
3 versions - Latest release: about 2 years ago - 1.89 thousand downloads total - 12 stars on GitHub - 1 maintainer
mathewberry/tokenizer 0.2.4
A very lightweight token system for micro-services
6 versions - Latest release: over 6 years ago - 768 downloads total - 1 stars on GitHub - 1 maintainer
Top 2.5% on packagist.org
chriskonnertz/string-calc v2.0.0
StringCalc is a PHP calculator library for mathematical terms (expressions) passed as strings.
9 versions - Latest release: almost 2 years ago - 5 dependent packages - 22 dependent repositories - 384 thousand downloads total - 96 stars on GitHub - 1 maintainer
linguistic/ngramextractor
Extracts ngrams from a given text and does linguistic pre-processing like stopword removal
Latest release: 17 days ago - 750 downloads total - 3 stars on GitHub - 1 maintainer
secondtruth/lexer 💰
Base library for simple lexers
Latest release: 18 days ago - 0 downloads total - 2 stars on GitHub - 1 maintainer
vanry/tokenizer
支持 scws, phpanalysis 和 jieba 的中文分词器
Latest release: 18 days ago - 2 downloads total - 0 stars on GitHub - 1 maintainer
vchagin/superclosure
Serialize Closures, including their context
Latest release: 18 days ago - 64 downloads total - 1 stars on GitHub - 1 maintainer
netgen/query-translator-demo 1.0.0
A demo for the Query Translator library
1 version - Latest release: about 6 years ago - 122 downloads total - 8 stars on GitHub - 3 maintainers
lahaxearnaud/laravel-token v0.5.1
Laravel 4 token management
6 versions - Latest release: over 9 years ago - 4 dependent repositories - 392 downloads total - 10 stars on GitHub - 1 maintainer
nocika/gpt3-tokenizer
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
Latest release: 19 days ago - 15 downloads total - 0 stars on GitHub - 1 maintainer
phplrt/lexer-contracts 3.6.5
A set of phplrt lexer abstractions
51 versions - Latest release: 3 months ago - 8 dependent packages - 9 dependent repositories - 85.8 thousand downloads total - 1 stars on GitHub - 1 maintainer
virusecks/objectron 0.0.4-alpha
Object creator from template
4 versions - Latest release: about 6 years ago - 17 downloads total - 1 stars on GitHub - 1 maintainer
xtreed/php-token-stream 4.0.4 💰
Wrapper around PHP's tokenizer extension.
37 versions - Latest release: almost 4 years ago - 1 dependent repositories - 3.52 thousand downloads total - 0 stars on GitHub - 1 maintainer
chrismcmacken/phptools
Small utilities that assist in developing applications
Latest release: 20 days ago - 4 dependent repositories - 3.56 thousand downloads total - 26 stars on GitHub - 1 maintainer
thruster/tokenizer 1.0.0
Thruster Tokenizer Component
1 version - Latest release: over 7 years ago - 4 downloads total - 0 stars on GitHub - 1 maintainer
brad-jones/superclosure v2.0.0
Doing interesting things with closures like serialization.
9 versions - Latest release: over 9 years ago - 2 dependent packages - 5 dependent repositories - 236 downloads total - 1 maintainer
hippophp/hippo v0.1.0
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
1 version - Latest release: over 8 years ago - 4 dependent repositories - 32 downloads total - 80 stars on GitHub - 1 maintainer
tanuel/tokenizer v0.1.1
A tokenizer for PHP
2 versions - Latest release: over 4 years ago - 14 downloads total - 6 stars on GitHub - 1 maintainer
phapp/phapp
The PHP Application for Webmasters
Latest release: 20 days ago - 1 downloads total - 2 stars on GitHub - 1 maintainer
octava/code-generator
PHP code generator
Latest release: 21 days ago - 1 downloads total - 0 stars on GitHub - 1 maintainer
ze/tokenizer-gpt3 v1.1
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
2 versions - Latest release: about 1 year ago - 1.21 thousand downloads total - 1 stars on GitHub - 1 maintainer
lalbert/tokenizer-engine 0.0.1
Tokenize string like Elasticsearch
1 version - Latest release: about 7 years ago - 15 downloads total - 1 stars on GitHub - 1 maintainer
hexydec/htmldoc 1.8.3
A token based HTML document parser and minifier. Minify HTML documents including inline CSS, Java...
26 versions - Latest release: 4 months ago - 4 dependent repositories - 4.39 thousand downloads total - 21 stars on GitHub - 1 maintainer
basemax/jpophp v2.2.1
JPOPHP (JSON Parser Object PHP) is a library for parsing the data in JSON format.
2 versions - Latest release: about 5 years ago - 309 downloads total - 25 stars on GitHub - 1 maintainer
simpletools/functionparser 1.0.1
Function parser for PHP functions, methods, and closures
2 versions - Latest release: 3 months ago - 813 downloads total - 0 stars on GitHub - 1 maintainer
hybridlogic/classifier v1.0.2
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize...
3 versions - Latest release: over 11 years ago - 8 dependent repositories - 1.43 thousand downloads total - 82 stars on GitHub - 1 maintainer
remorhaz/php-unilex v0.5.3
Unilex: lexical analyzer generator with Unicode support written in PHP
27 versions - Latest release: 3 months ago - 2 dependent packages - 4 dependent repositories - 89.7 thousand downloads total - 2 stars on GitHub - 1 maintainer
sincco/tokenizer 1.1.2
Create and validate token for user data
4 versions - Latest release: about 1 year ago - 1 dependent package - 1 dependent repositories - 172 downloads total - 0 stars on GitHub - 1 maintainer
dbeurive/lexer 1.0.4
This project implements a simple lexer.
5 versions - Latest release: over 7 years ago - 1 dependent package - 1 dependent repositories - 2.24 thousand downloads total - 2 stars on GitHub - 1 maintainer
Top 9.3% on packagist.org
funivan/php-tokenizer 0.3.0
Wrapper around token_get_all. Easy to extract and modify php code.
11 versions - Latest release: over 4 years ago - 2 dependent packages - 3 dependent repositories - 37.1 thousand downloads total - 12 stars on GitHub - 1 maintainer
didix16/php-grammar 1.0.1
A simple library for make Lexer and Parsers to build a language
2 versions - Latest release: over 2 years ago - 2 dependent packages - 2 dependent repositories - 4.53 thousand downloads total - 1 stars on GitHub - 1 maintainer
vanderlee/comprehend 1.0.2
Framework for building BNF LR(1) parsers
3 versions - Latest release: over 5 years ago - 14 downloads total - 5 stars on GitHub - 1 maintainer
Related Keywords
php 37 parser 31 token 18 lexer 13 openai 10 decode 9 encode 9 function 7 GPT-3 7 closure 7 nlp 6 ast 6 gpt-2 6 html 5 tokenization 5 serialize 5 serializable 5 library 5 javascript 5 php-library 5 tiktoken 5 tokens 5 query 5 lambda 4 reflection 4 compressor 4 generator 4 laravel 4 parsing 4 bpe 4 php8 3 sikessem 3 archiver 3 language 3 tokenize 3 regex 3 xml 3 search 3 php7 3 stream 3 translator 3 math 3 formatter 3 compiler 3 bnf 2 indonesia 2 abnf 2 object 2 template 2 setup 2 minifier 2 minify 2 script-compiler 2 css 2 json 2 code-style 2 cl100k_base 2 reformat 2 stanford 2 ner 2 corenlp-server 2 lemma 2 openie 2 lex 2 pcre 2 format 2 phplrt 2 string 2 engine 2 sql server 2 mysql 2 sql 2 mathematicator 2 validator 2 ske 2 binary 2 calculator 2 cli-app 2 solr 2 elasticsearch 2 lexical-analysis 2 console-application 2 edismax 2 HTML5 2 ngram 2 installer 2 runtime 2 class 2 natural-language-processing 1 cyrillization 1 diacritics 1 diacritic restoration 1 term 1 expression 1 srpski 1 calculation 1 logger 1 injection 1 session 1 microservice 1