nuget.org : portedfastberttokenizer
Fast and memory-efficient WordPiece tokenizer as it is used by BERT and others. Tokenizes text for further processing using NLP/language models. This fork only supports .NET Standard 2.1
Registry
-
Source
- JSON
purl: pkg:nuget/portedfastberttokenizer
Keywords:
bert
, tokenizer
, wordpiece
, llm
, semantic-kernel
, performance
, ai
, artificial-intelligence
, ml
, machine-learning
License: MIT
Latest release: almost 2 years ago
First release: almost 2 years ago
Stars: 1 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 22 days ago