Data

Tools

Indexes

Applications

Experiments

Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

hex.pm "tokenizer" keyword

tiktoken_ex 0.2.0
Pure-Elixir TikToken-style byte-level BPE tokenizer, with helpers for Kimi K2 encodings.
2 versions - Latest release: about 2 months ago - 96 downloads total - 1 stars on GitHub - 1 maintainer
glex 1.0.1
Simple lexer generator for Gleam
2 versions - Latest release: almost 2 years ago - 246 downloads total - 3 stars on GitHub - 1 maintainer
Related Keywords
beam 1 bpe 1 byte-pair-encoding 1 elixir 1 encoding 1 erlang 1 hex-package 1 huggingface 1 kimi 1 llm 1 moonshotai 1 nlp 1 nshkr-ai-infra 1 otp 1 regex 1 tiktoken 1 tokenization 1 typespecs 1 unicode 1 gleam 1 gleam-lang 1 lex 1 lexer 1