Data

Tools

Indexes

Applications

Experiments

Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

hex.pm "tiktoken" keyword

tiktoken_ex 0.2.0
Pure-Elixir TikToken-style byte-level BPE tokenizer, with helpers for Kimi K2 encodings.
2 versions - Latest release: about 2 months ago - 96 downloads total - 1 stars on GitHub - 1 maintainer
Related Keywords
beam 1 bpe 1 byte-pair-encoding 1 elixir 1 encoding 1 erlang 1 hex-package 1 huggingface 1 kimi 1 llm 1 moonshotai 1 nlp 1 nshkr-ai-infra 1 otp 1 regex 1 tokenization 1 tokenizer 1 typespecs 1 unicode 1