An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

packagist.org "tokenizer" keyword

View the packages on the packagist.org package registry that are tagged with the "tokenizer" keyword.

megao-phpunit/php-token-stream 4.0.4 💰
Wrapper around PHP's tokenizer extension.
36 versions - Latest release: about 5 years ago - 44 downloads total - 0 stars on GitHub - 1 maintainer
Top 0.5% on packagist.org
theseer/tokenizer 1.2.3 💰
A small library for converting tokenized PHP source code into XML and potentially other formats
9 versions - Latest release: over 1 year ago - 15 dependent packages - 453,841 dependent repositories - 674 million downloads total - 5,119 stars on GitHub - 1 maintainer
jundayw/tokenizer
Laravel Tokenizer provides a multi-role, multi-scenario authentication system for SPAs and APIs.
Latest release: about 7 hours ago - 0 downloads total
Top 0.3% on packagist.org
jeremeamia/superclosure 2.4.0
Serialize Closure objects, including their context and binding
15 versions - Latest release: over 7 years ago - 325 dependent packages - 5,002 dependent repositories - 59.5 million downloads total - 1,720 stars on GitHub - 1 maintainer
Top 1.2% on packagist.org
mck89/peast v1.17.2
Peast is PHP library that generates AST for JavaScript code
46 versions - Latest release: 2 months ago - 15 dependent packages - 744 dependent repositories - 26.8 million downloads total - 187 stars on GitHub - 1 maintainer
Top 2.5% on packagist.org
chriskonnertz/string-calc v2.0.0
StringCalc is a PHP calculator library for mathematical terms (expressions) passed as strings.
9 versions - Latest release: over 3 years ago - 5 dependent packages - 22 dependent repositories - 572 thousand downloads total - 101 stars on GitHub - 1 maintainer
Top 0.8% on packagist.org
andrewsville/php-token-reflection 1.4.0
Library emulating the PHP internal reflection using just the tokenized source code.
4 versions - Latest release: about 11 years ago - 43 dependent packages - 2,022 dependent repositories - 1.75 million downloads total - 190 stars on GitHub - 2 maintainers
Top 0.2% on packagist.org
phpunit/php-token-stream 4.0.4 💰
Wrapper around PHP's tokenizer extension.
37 versions - Latest release: about 5 years ago - 205 dependent packages - 382,265 dependent repositories - 382 million downloads total - 6,441 stars on GitHub - 1 maintainer
Top 1.0% on packagist.org
nette/tokenizer v3.1.1 💰
Nette Tokenizer
8 versions - Latest release: over 3 years ago - 103 dependent packages - 387 dependent repositories - 4.49 million downloads total - 140 stars on GitHub - 1 maintainer
Top 10.0% on packagist.org
popsul/php-token-reflection 1.5.8
Library emulating the PHP internal reflection using just the tokenized source code.
6 versions - Latest release: about 9 years ago - 1 dependent package - 5 dependent repositories - 77.5 thousand downloads total - 2 stars on GitHub - 1 maintainer
attla/data-token
Turn everything into a token: randomized or predictable.
Latest release: 2 days ago - 4 dependent repositories - 21 downloads total - 1 stars on GitHub - 1 maintainer
gorriecoe/silverstripe-sreg 1.2.2
Simple tokenizer that allows you to use .ss template like variables in a dataobjects string.
7 versions - Latest release: over 7 years ago - 1 dependent package - 135 downloads total - 1 stars on GitHub - 1 maintainer
hexydec/htmldoc 1.8.7
A token based HTML document parser and minifier. Minify HTML documents including inline CSS, Java...
30 versions - Latest release: about 2 months ago - 1 dependent package - 4 dependent repositories - 8.28 thousand downloads total - 22 stars on GitHub - 1 maintainer
ghostwriter/syntax removed 💰
Lexical Analysis library for PHP
Latest release: 11 months ago - 0 downloads total - 2 stars on GitHub - 1 maintainer
chinleung/tokenizer 1.0.2
PHP Tokenizer
3 versions - Latest release: about 9 years ago - 591 downloads total - 1 maintainer
dennis-de-swart/php-stanford-nlp-datastore 2.2.0
Stores NLP data from Stanford CoreNLP server.
3 versions - Latest release: over 8 years ago - 70 downloads total - 5 stars on GitHub - 1 maintainer
phplrt/lexer-contracts 3.7.5
A set of phplrt lexer abstractions
58 versions - Latest release: 12 months ago - 8 dependent packages - 9 dependent repositories - 147 thousand downloads total - 1 stars on GitHub - 1 maintainer
phplrt/lexer 3.7.5
The phplrt lexer library that can be used for scanners implementation using PCRE
61 versions - Latest release: 10 months ago - 9 dependent packages - 11 dependent repositories - 136 thousand downloads total - 3 stars on GitHub - 1 maintainer
fenom/tokenizer 1.0.0
Stand-alone Fenom's tokenizer
1 version - Latest release: over 9 years ago - 24 downloads total - 2 stars on GitHub - 1 maintainer
Top 2.5% on packagist.org
nilportugues/sql-query-formatter v1.2.2
A very lightweight PHP class that reformats unreadable and computer-generated SQL query statement...
5 versions - Latest release: almost 10 years ago - 25 dependent packages - 69 dependent repositories - 984 thousand downloads total - 39 stars on GitHub - 1 maintainer
masnathan/sql-query-formatter v1.2.1
A very lightweight PHP class that reformats unreadable and computer-generated SQL query statement...
4 versions - Latest release: over 10 years ago - 1 dependent package - 1.79 thousand downloads total - 0 stars on GitHub - 1 maintainer
thruster/tokenizer 1.0.0
Thruster Tokenizer Component
1 version - Latest release: over 8 years ago - 6 downloads total - 0 stars on GitHub - 1 maintainer
apemsel/attributed-string v3.0.0
A collection of fast and easy to use attributed string classes for PHP. Attributed strings can ha...
24 versions - Latest release: over 1 year ago - 555 downloads total - 1 stars on GitHub - 1 maintainer
hybridlogic/classifier v1.0.2
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize...
3 versions - Latest release: over 12 years ago - 8 dependent repositories - 1.45 thousand downloads total - 80 stars on GitHub - 1 maintainer
dennis-de-swart/php-stanford-corenlp-adapter 7.1.0
PHP adapter for use with Stanford CoreNLP tools
4 versions - Latest release: over 6 years ago - 1 dependent package - 1 dependent repositories - 2.86 thousand downloads total - 26 stars on GitHub - 1 maintainer
Top 2.4% on packagist.org
netgen/query-translator 1.1.0
Query Translator is a search query translator with AST representation
5 versions - Latest release: over 3 years ago - 4 dependent packages - 47 dependent repositories - 1.73 million downloads total - 206 stars on GitHub - 3 maintainers
nocika/gpt3-tokenizer
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
Latest release: 8 days ago - 15 downloads total - 0 stars on GitHub - 1 maintainer
yep/tokenizer
Tokenizer
Latest release: 8 days ago - 8 downloads total - 0 stars on GitHub - 1 maintainer
antto1/superclosure v2.4.1
Serialize Closure objects, including their context and binding
16 versions - Latest release: almost 5 years ago - 1 dependent package - 160 downloads total - 0 stars on GitHub - 1 maintainer
tanuel/tokenizer v0.1.1
A tokenizer for PHP
2 versions - Latest release: almost 6 years ago - 15 downloads total - 6 stars on GitHub - 1 maintainer
virusecks/objectron 0.0.4-alpha
Object creator from template
4 versions - Latest release: over 7 years ago - 17 downloads total - 1 stars on GitHub - 1 maintainer
Top 9.3% on packagist.org
funivan/php-tokenizer 0.3.0
Wrapper around token_get_all. Easy to extract and modify php code.
11 versions - Latest release: over 5 years ago - 2 dependent packages - 3 dependent repositories - 38.5 thousand downloads total - 12 stars on GitHub - 1 maintainer
tacoberu/bnf v1.2.1
Framework for building BNF like parsers.
8 versions - Latest release: 6 months ago - 1 dependent package - 1 dependent repositories - 2.76 thousand downloads total - 5 stars on GitHub - 1 maintainer
robinthehood/tokenizer 0.1.3
With Tokenizer you can easily convert a string to a stream or array of tokens.
3 versions - Latest release: about 7 years ago - 2 dependent packages - 3 dependent repositories - 836 downloads total - 0 stars on GitHub - 1 maintainer
linguistic/ngramextractor
Extracts ngrams from a given text and does linguistic pre-processing like stopword removal
Latest release: 10 days ago - 2 thousand downloads total - 3 stars on GitHub - 1 maintainer
sastrawi/tokenizer v0.4.0
PHP library that allows you to tokenize Bahasa Indonesia.
4 versions - Latest release: over 10 years ago - 12 dependent repositories - 947 downloads total - 34 stars on GitHub - 1 maintainer
td7650/superclosure 2.1.3
Serialize Closure objects, including their context and binding
15 versions - Latest release: almost 10 years ago - 1 dependent package - 58 downloads total - 0 stars on GitHub - 1 maintainer
Top 2.9% on packagist.org
spiral/tokenizer 3.15.8 💰
Static Analysis: Class and Invocation locators
95 versions - Latest release: 4 months ago - 39 dependent packages - 109 dependent repositories - 5.42 million downloads total - 17 stars on GitHub - 1 maintainer
hike/hike v0.1.1
A consistent interface to tokenize, analyse and modify any text format. Programming languages, na...
2 versions - Latest release: 5 months ago - 5 downloads total - 0 stars on GitHub - 1 maintainer
guttedgarden/tiktoken 1.2.1
PHP 7.4 version of tiktoken
4 versions - Latest release: 5 months ago - 14.3 thousand downloads total - 4 stars on GitHub - 1 maintainer
your1/tokenizer-x 1.0.1
TokenizerX calculates required tokens for given prompt
2 versions - Latest release: 5 months ago - 273 downloads total - 0 stars on GitHub - 1 maintainer
Top 2.0% on packagist.org
fig-r/psr2r-sniffer 2.2.1
Code-Sniffer, Auto-Fixer and Tokenizer for PSR2-R
34 versions - Latest release: 4 months ago - 83 dependent packages - 180 dependent repositories - 1.03 million downloads total - 32 stars on GitHub - 1 maintainer
nexxes/tokenmatcher
PHP library to match tokenstreams (arrays of tokens) for later parsing.
Latest release: 12 days ago - 4 dependent repositories - 9 downloads total - 0 stars on GitHub - 1 maintainer
didix16/php-grammar 1.0.1
A simple library for make Lexer and Parsers to build a language
2 versions - Latest release: about 4 years ago - 2 dependent packages - 2 dependent repositories - 7.67 thousand downloads total - 1 stars on GitHub - 1 maintainer
muhfirdaus19/bahasata
Text Processing PHP bahasa indonesia
Latest release: 12 days ago - 6 downloads total - 1 stars on GitHub - 1 maintainer
phapp/phapp
The PHP Application for Webmasters
Latest release: 12 days ago - 2 downloads total - 2 stars on GitHub - 1 maintainer
lalbert/tokenizer-engine 0.0.1
Tokenize string like Elasticsearch
1 version - Latest release: over 8 years ago - 18 downloads total - 1 stars on GitHub - 1 maintainer
phpwatch/phptoken-polyfill v0.0.3
A polyfill for PHP 8.0's PhpToken class
3 versions - Latest release: almost 5 years ago - 18 downloads total - 9 stars on GitHub - 1 maintainer
octava/code-generator
PHP code generator
Latest release: 14 days ago - 1 downloads total - 0 stars on GitHub - 1 maintainer
Top 8.2% on packagist.org
mathematicator-core/tokenizer v3.0.1 💰
Math Tokenizer
14 versions - Latest release: over 4 years ago - 5 dependent packages - 9 dependent repositories - 4.66 thousand downloads total - 8 stars on GitHub - 1 maintainer
affinity4/tokenizer 0.0.4
A zero-dedpendency tokenizer written in PHP. Returns an easily navigatable Stream object of Token...
4 versions - Latest release: over 3 years ago - 7 downloads total - 0 stars on GitHub - 1 maintainer
djiele/autoloader 7.3.1-stable
Another class loader using PHP tokenizer
1 version - Latest release: about 5 years ago - 11 downloads total - 0 stars on GitHub - 1 maintainer
apicart/fql 0.4.6
Filter Query Language
20 versions - Latest release: about 2 months ago - 10.1 thousand downloads total - 11 stars on GitHub - 1 maintainer
html5/htmlreader v2.0.1
Html5 stream tokenizer/reader (not using libxml)
4 versions - Latest release: almost 8 years ago - 1 dependent package - 2 dependent repositories - 3.3 thousand downloads total - 1 stars on GitHub - 1 maintainer
guanhui07/tiktoken 1.0.2
PHP version of tiktoken
2 versions - Latest release: about 1 year ago - 6 downloads total - 1 stars on GitHub - 1 maintainer
Top 6.3% on packagist.org
herrera-io/annotations 1.0.1
A tokenizer for Doctrine annotations.
6 versions - Latest release: over 11 years ago - 12 dependent packages - 893 dependent repositories - 128 thousand downloads total - 22 stars on GitHub - 1 maintainer
attla/token
Turn everything into a unique encrypted JWT.
Latest release: 15 days ago - 3 downloads total - 0 stars on GitHub - 1 maintainer
rahul900day/gpt-3-encoder 1.1.0 💰
GPT-3-Encoder is a PHP implementation of OpenAI's original python encoder/decoder.
3 versions - Latest release: over 2 years ago - 2 dependent repositories - 30 thousand downloads total - 2 stars on GitHub - 1 maintainer
xp-framework/tokenize v9.1.0
Text tokenizing
11 versions - Latest release: over 1 year ago - 9 dependent packages - 5 dependent repositories - 170 thousand downloads total - 0 stars on GitHub - 1 maintainer
mathematicator-core/engine v3.0.1 💰
Math computational Engine
19 versions - Latest release: over 4 years ago - 5 dependent packages - 7 dependent repositories - 3.52 thousand downloads total - 2 stars on GitHub - 1 maintainer
Top 9.0% on packagist.org
phootwork/tokenizer v3.2.3
Tokenizer for PHP code
16 versions - Latest release: 11 months ago - 4 dependent packages - 16 dependent repositories - 209 thousand downloads total - 2 stars on GitHub - 2 maintainers
scripturadesign/tokenizer
Tokenizer
Latest release: 16 days ago - 7 downloads total - 0 stars on GitHub - 1 maintainer
rudl/logger v0.8
Rudl UDP cluster logging library (PSR4 compliant)
7 versions - Latest release: about 8 years ago - 2.71 thousand downloads total - 0 stars on GitHub - 1 maintainer
yuubit/tokenizer 1.0.0
PHP Library to tokenize various inputs.
1 version - Latest release: over 6 years ago - 1 dependent package - 1 dependent repositories - 403 downloads total - 0 stars on GitHub - 1 maintainer
teepluss/wordbreaker v0.1
Word breaker is Thai wording separator for using with fulltext software.
1 version - Latest release: almost 11 years ago - 4 dependent repositories - 26 downloads total - 4 stars on GitHub - 1 maintainer
mistralys/jtokenizer 1.0.1
Modernized, composer compatible version of Tim Whitlock's JTokenizer PHP Javascript parser and to...
2 versions - Latest release: over 4 years ago - 1 dependent package - 6 dependent repositories - 6.77 thousand downloads total - 2 stars on GitHub - 1 maintainer
yurunsoft/tiktoken v0.1.4
PHP version of tiktoken
2 versions - Latest release: almost 2 years ago - 1 dependent repositories - 241 downloads total - 0 stars on GitHub - 1 maintainer
hippophp/hippo v0.1.0
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
1 version - Latest release: over 9 years ago - 4 dependent repositories - 32 downloads total - 80 stars on GitHub - 1 maintainer
scheldejonas/phpunit-php-token-stream 4.0.5 💰
Wrapper around PHP's tokenizer extension.
36 versions - Latest release: almost 5 years ago - 208 downloads total - 0 stars on GitHub - 1 maintainer
fi1a/tokenizer 1.1.7
Лексический анализатор строк (tokenizer). Разбор входной последовательности символов на распознан...
11 versions - Latest release: over 2 years ago - 5 dependent packages - 5 dependent repositories - 7.81 thousand downloads total - 0 stars on GitHub - 1 maintainer
ze/tokenizer-gpt3 v1.1
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
2 versions - Latest release: over 2 years ago - 1.49 thousand downloads total - 1 stars on GitHub - 1 maintainer
dbeurive/lexer 1.0.4
This project implements a simple lexer.
5 versions - Latest release: almost 9 years ago - 1 dependent package - 1 dependent repositories - 2.41 thousand downloads total - 2 stars on GitHub - 1 maintainer
brad-jones/superclosure v2.0.0
Doing interesting things with closures like serialization.
9 versions - Latest release: almost 11 years ago - 2 dependent packages - 5 dependent repositories - 236 downloads total - 1 maintainer
devtronic/super-tokenizer 1.0.1 💰
A powerful dynamic tokenizer written in PHP
2 versions - Latest release: over 8 years ago - 65 downloads total - 1 stars on GitHub - 1 maintainer
kevintweber/html-tokenizer v0.3.1
Will tokenize HTML.
9 versions - Latest release: over 8 years ago - 1 dependent package - 2 dependent repositories - 47.6 thousand downloads total - 5 stars on GitHub - 1 maintainer
lahaxearnaud/laravel-token v0.5.1
Laravel 4 token management
6 versions - Latest release: over 10 years ago - 4 dependent repositories - 395 downloads total - 10 stars on GitHub - 1 maintainer
secondtruth/lexer 💰
Base library for simple lexers
Latest release: 22 days ago - 0 downloads total - 2 stars on GitHub - 1 maintainer
Top 7.0% on packagist.org
yethee/tiktoken 0.11.0
PHP version of tiktoken
14 versions - Latest release: 24 days ago - 5 dependent packages - 1 dependent repositories - 1.73 million downloads total - 137 stars on GitHub - 1 maintainer
mathewberry/tokenizer 0.2.4
A very lightweight token system for micro-services
6 versions - Latest release: about 8 years ago - 770 downloads total - 1 stars on GitHub - 1 maintainer
angkor/khmercut 0.2.0
Khmer Tokenization
4 versions - Latest release: 4 months ago - 219 downloads total - 0 stars on GitHub - 1 maintainer
nexxes/tokenizer
Simple tokenizer to split a character steam (text) into character classes (tokens) written in PHP
Latest release: 22 days ago - 6 dependent repositories - 40 downloads total - 1 stars on GitHub - 1 maintainer
interitty/tokenizer v1.0.9
Use regular expressions to split a given string into tokens.
10 versions - Latest release: 4 months ago - 1 dependent package - 1 dependent repositories - 42 downloads total - 1 stars on gitlab.com - 1 maintainer
fourkitchens/tiktoken
PHP version of tiktoken
Latest release: 23 days ago - 0 downloads total - 0 stars on GitHub - 1 maintainer
vanderlee/comprehend 1.0.2
Framework for building BNF LR(1) parsers
3 versions - Latest release: almost 7 years ago - 17 downloads total - 7 stars on GitHub - 1 maintainer
everstu/gpt3-tokenizer v1.0.0
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
1 version - Latest release: over 2 years ago - 149 downloads total - 0 stars on GitHub - 1 maintainer
bermudaphp/tokenizer v1.0
PHP tokenizer for finding class, interface, trait, and enum declarations
1 version - Latest release: 3 months ago - 0 downloads total - 0 stars on GitHub - 1 maintainer
kornrunner/php-token-reflection v1.5.6
Library emulating the PHP internal reflection using just the tokenized source code.
11 versions - Latest release: over 9 years ago - 6.97 thousand downloads total - 1 stars on GitHub - 1 maintainer
Top 3.5% on packagist.org
nicoswd/php-rule-parser 0.7.1
Rule Engine - Rule Parser & Evaluator
21 versions - Latest release: over 4 years ago - 6 dependent packages - 7 dependent repositories - 70.8 thousand downloads total - 129 stars on GitHub - 1 maintainer
turanjanin/serbian-language-tools 1.0.3
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Ser...
4 versions - Latest release: 24 days ago - 2.04 thousand downloads total - 17 stars on GitHub - 1 maintainer
taufik-nurrohman/minify v1.0.3 💰
Reduce CSS, HTML, JavaScript, JSON, PHP, and XML code size on the server side.
4 versions - Latest release: 8 months ago - 5 downloads total - 3 stars on GitHub - 1 maintainer
danny50610/bpe-tokeniser 0.2.0
PHP port for openai/tiktoken (most)
2 versions - Latest release: over 1 year ago - 1 dependent package - 286 thousand downloads total - 9 stars on GitHub - 1 maintainer
artemeon/tokenizer 2.0.1
Tokenizer for php
4 versions - Latest release: about 3 years ago - 71.9 thousand downloads total - 0 stars on GitHub - 1 maintainer
railt/lexer 1.3.2
Fast implementation of the stateful and stateless lexers
4 versions - Latest release: over 6 years ago - 4 dependent packages - 2 dependent repositories - 3.32 thousand downloads total - 6 stars on GitHub - 1 maintainer
legacyphpdoctor/tiktoken-php-7-4
PHP version of tiktoken
Latest release: 28 days ago - 1 downloads total - 1 stars on GitHub - 1 maintainer
gossi/tokenizer v0.1
Tokenizer for PHP code
1 version - Latest release: almost 11 years ago - 4 dependent repositories - 150 downloads total - 1 maintainer
vanry/tokenizer
支持 scws, phpanalysis 和 jieba 的中文分词器
Latest release: 28 days ago - 2 downloads total - 0 stars on GitHub - 1 maintainer
infinityloop-dev/graphpinator-tokenizer v1.1.1
Tokenizer subproject for GraPHPinator: GraphQL server implementation for PHP.
6 versions - Latest release: over 4 years ago - 2 dependent packages - 1 dependent repositories - 34 thousand downloads total - 3 stars on GitHub - 1 maintainer
xtreed/php-token-stream 4.0.4 💰
Wrapper around PHP's tokenizer extension.
37 versions - Latest release: about 5 years ago - 1 dependent repositories - 3.53 thousand downloads total - 0 stars on GitHub - 1 maintainer
vchagin/superclosure
Serialize Closures, including their context
Latest release: about 1 month ago - 65 downloads total - 1 stars on GitHub - 1 maintainer
rajentrivedi/tokenizer-x v1.0.5
TokenizerX calculates required tokens for given prompt
6 versions - Latest release: 7 months ago - 1 dependent package - 120 thousand downloads total - 88 stars on GitHub - 1 maintainer