npmjs.org : llm-tokenizer
A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package
Registry
-
Source
- Homepage
- JSON
- codemeta.json
purl: pkg:npm/llm-tokenizer
Keywords:
OpenAI
, GPT-3
, GPT-4
, GPT-3.5
, ChatGPT
, LLMs
, GPT
, tokenizer
, BPE
, encoder
, decoder
, Machine Learning
License: Apache-2.0
Latest release: about 2 years ago
First release: over 2 years ago
Dependent packages: 1
Downloads: 37 last month
Stars: 3 on GitHub
Forks: 1 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 26 days ago