Data

Tools

Indexes

Applications

Experiments

Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

crates.io "tiktoken" keyword

View the packages on the crates.io package registry that are tagged with the "tiktoken" keyword.

chat-splitter 0.1.1 💰
Never exceed OpenAI's chat models' maximum number of tokens when using the async_openai Rust crate
2 versions - Latest release: almost 2 years ago - 2.5 thousand downloads total - 3 stars on GitHub - 1 maintainer
tokin 0.1.0
Experimental fast tokenizer.
1 version - Latest release: 9 months ago - 799 downloads total - 1 maintainer
Related Keywords
nlp 2 tokenizer 2 ai 1 artificial-intelligence 1 chat 1 chatgpt 1 gpt-4 1 openai 1 split 1 text 1 wordpiece 1 bpe 1