nuget.org : fastberttokenizer
Fast and memory-efficient WordPiece tokenizer as it is used by BERT and others. Tokenizes text for further processing using NLP/language models.
Registry
-
Source
- JSON
purl: pkg:nuget/fastberttokenizer
Keywords:
bert
, tokenizer
, wordpiece
, llm
, semantic-kernel
, performance
, ai
, artificial-intelligence
, ml
, machine-learning
, bert-embeddings
, natural-language-processing
, nlp
, nlp-machine-learning
, tokenization
, tokens
, wordpiece-tokenization
License: MIT
Latest release: about 1 year ago
First release: over 125 years ago
Dependent packages: 2
Downloads: 435,650 total
Stars: 49 on GitHub
Forks: 11 on GitHub
Total Commits: 145
Committers: 2
Average commits per author: 72.5
Development Distribution Score (DDS): 0.034
More commit stats: commits.ecosyste.ms
See more repository details: repos.ecosyste.ms
Last synced: about 12 hours ago