pypi.org "huggingface-tokenizers" keyword
View the packages on the pypi.org package registry that are tagged with the "huggingface-tokenizers" keyword.
transformers-domain-adaptation 0.3.1
Adapt Transformer-based language models to new text domains6 versions - Latest release: over 4 years ago - 1 dependent repositories - 686 downloads last month - 87 stars on GitHub - 1 maintainer
rs-bpe 0.1.0
A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust1 version - Latest release: 6 months ago - 475 downloads last month - 6 stars on GitHub - 1 maintainer
Related Keywords
transformers
2
tokenizers
2
huggingface
2
natural-language-processing
2
researchers
1
scientists
1
developers
1
tokenizers-library
1
tiktoken-compatible
1
tiktoken-alternative
1
tiktoken
1
openai
1
linguistics
1
research
1
data-science
1
embeddings
1
text-generation
1
generative-ai
1
large-language-models
1
llm
1
ai
1
rust-extension
1
python
1
pypi-package
1
byte-pair-tokenizer
1
bpe-tokenizer
1
bpe
1
backtracking
1
bpe-dropout
1
unigram
1
wordpiece
1
sentencepiece
1
tool
1
package
1
library
1
stable
1
production-ready
1
nlp-engineers
1
machine-learning-engineers
1
data-scientists
1
rust
1
string-encoding
1
text-processing
1
text-encoding
1
vocab
1
vocabulary
1
tokenizer
1
tokenization
1
subword-units
1
subword-tokenization
1
byte-pair-encoding
1
huggingface-transformers
1
deprecated
1
data-selection
1
transfer-learning
1
domain-adaptation
1
pytorch
1
artificial-intelligence
1
deep-learning
1
machine-learning
1
nlp
1
accelerated
1
rapid
1
blazing-fast
1
speed
1
optimized
1
efficient
1
high-performance
1
performance
1
fast
1
cross-platform
1
pypi
1
python-package
1
python-library
1
cpython-extension
1