proxy.golang.org : github.com/shadowy-pycoder/tinybpe : v1.0.0
Tiny implementation of the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. Ispired by karpathy/minbpe
Registry -
Documentation -
Download -
JSON
purl: pkg:golang/github.com/shadowy-pycoder/[email protected]
Published:
Indexed:
Loading...
Readme
Loading...