pypi.org "tokenizers" keyword
View the packages on the pypi.org package registry that are tagged with the "tokenizers" keyword.
tftokenizers 0.1.8
Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels.9 versions - Latest release: over 3 years ago - 1 dependent repositories - 40 downloads last month - 9 stars on GitHub - 1 maintainer
llm-magnet 0.3.17
the small distributed language model toolkit. fine-tune state-of-the-art LLMs anywhere, rapidly.37 versions - Latest release: about 1 year ago - 38 downloads last month - 31 stars on GitHub - 1 maintainer
transformers-domain-adaptation 0.3.1
Adapt Transformer-based language models to new text domains6 versions - Latest release: over 4 years ago - 1 dependent repositories - 686 downloads last month - 87 stars on GitHub - 1 maintainer
merge-tokenizers 0.0.6
Package to merge tokens from different tokenizers.3 versions - Latest release: over 1 year ago - 22 downloads last month - 11 stars on GitHub - 1 maintainer
autotiktokenizer 0.2.2
🧰 The AutoTokenizer that TikToken always needed -- Load any tokenizer with TikToken now! ✨7 versions - Latest release: 9 months ago - 5.09 thousand downloads last month - 39 stars on GitHub - 1 maintainer
Top 9.4% on pypi.org
7 versions - Latest release: about 3 years ago - 1 dependent package - 5 dependent repositories - 2.7 thousand downloads last month - 16 stars on GitHub - 1 maintainer
ginza-transformers 0.4.2
ginza-transformers7 versions - Latest release: about 3 years ago - 1 dependent package - 5 dependent repositories - 2.7 thousand downloads last month - 16 stars on GitHub - 1 maintainer
tokenizerchanger 1.0.5
Library for manipulating the existing tokenizer.20 versions - Latest release: 3 months ago - 211 downloads last month - 18 stars on GitHub - 1 maintainer
rs-bpe 0.1.0
A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust1 version - Latest release: 6 months ago - 475 downloads last month - 6 stars on GitHub - 1 maintainer
pgn-tokenizer 0.1.5 💰
A byte pair encoding tokenizer for chess portable game notation (PGN)4 versions - Latest release: 7 months ago - 21 downloads last month - 0 stars on GitHub - 1 maintainer
Related Keywords
transformers
6
huggingface
5
tokenizer
5
natural-language-processing
4
nlp
4
huggingface-tokenizers
2
tokens
2
tiktoken
2
machine-learning
2
llm
2
byte-pair-encoding
2
embeddings
2
bpe
2
pypi
1
openai
1
linguistics
1
cross-platform
1
fast
1
performance
1
research
1
data-science
1
text-generation
1
generative-ai
1
large-language-models
1
high-performance
1
efficient
1
ai
1
artificial-intelligence
1
deep-learning
1
accelerated
1
rapid
1
blazing-fast
1
optimized
1
speed
1
pgn
1
chess
1
byte pair encoding
1
python
1
pypi-package
1
byte-pair-tokenizer
1
bpe-tokenizer
1
backtracking
1
bpe-dropout
1
unigram
1
wordpiece
1
sentencepiece
1
tool
1
package
1
library
1
stable
1
production-ready
1
nlp-engineers
1
machine-learning-engineers
1
data-scientists
1
researchers
1
scientists
1
developers
1
tokenizers-library
1
tiktoken-compatible
1
tiktoken-alternative
1
domain-adaptation
1
pytorch
1
sentence-splitting
1
nats-streaming
1
nats-messaging
1
nats
1
mlx
1
mistral
1
milvus
1
llm-training
1
langchain
1
inference-api
1
gemini
1
finetuning-llms
1
fine-tuning
1
distributed-systems
1
distributed-computing
1
claude
1
apple-silicon
1
tensorflow-hub
1
sentencepie
1
bert
1
text
1
tensorflow
1
python-package
1
python-library
1
cpython-extension
1
rust-extension
1
rust
1
string-encoding
1
text-processing
1
text-encoding
1
vocab
1
vocabulary
1
tokenization
1
subword-units
1
subword-tokenization
1
delete
1
deletion
1
sudachitra
1