Data

Tools

Indexes

Applications

Experiments

Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

tiktoken_ex

Pure-Elixir TikToken-style byte-level BPE tokenizer, with helpers for Kimi K2 encodings.

Ecosystem
hex.pm
Latest Release
0.2.0
about 2 months ago
Versions
2
Downloads
96 total
    Loading...
    Readme
    Loading...
    Links
    Registry hex.pm
    Source Repository
    Homepage Homepage
    Docs Documentation
    JSON API View JSON
    CodeMeta codemeta.json
    Package Details
    PURL pkg:hex/tiktoken_ex
    spec
    License MIT
    First Release 2 months ago
    Last Synced 7 days ago
    Keywords
    beam bpe byte-pair-encoding elixir encoding erlang hex-package huggingface kimi llm moonshotai nlp nshkr-ai-infra otp regex tiktoken tokenization tokenizer typespecs unicode
    Repository
    Stars 1 on GitHub
    Forks 0 on GitHub
    repos.ecosyste.ms