An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

pypi.org "llm-inference" keyword

View the packages on the pypi.org package registry that are tagged with the "llm-inference" keyword.

autogen-magentic
AutoGen Magentic
1 version - 97 downloads last month - 50,154 stars on GitHub - 1 maintainer
autogen-langchain
AutoGen Langchain
1 version - 139 downloads last month - 50,154 stars on GitHub - 1 maintainer
autogenstudio 0.4.2
AutoGen Studio
129 versions - Latest release: 7 months ago - 3 dependent packages - 61.3 thousand downloads last month - 18,528 stars on GitHub - 1 maintainer
flashinfer-python 0.3.1
FlashInfer: Kernel Library for LLM Serving
38 versions - Latest release: 28 days ago - 323 thousand downloads last month - 3,823 stars on GitHub - 2 maintainers
Top 0.7% on pypi.org
openvino-dev 2024.6.0
OpenVINO(TM) Development Tools
37 versions - Latest release: 10 months ago - 38 dependent packages - 498 dependent repositories - 358 thousand downloads last month - 6,310 stars on GitHub - 1 maintainer
Top 5.9% on pypi.org
dstack 0.19.31
dstack is an open-source orchestration engine for running AI workloads on any cloud or on-premises.
288 versions - Latest release: about 19 hours ago - 3 dependent repositories - 5.87 thousand downloads last month - 1,415 stars on GitHub - 1 maintainer
Top 6.0% on pypi.org
lmdeploy 0.10.1
A toolset for compressing, deploying and serving LLM
57 versions - Latest release: 7 days ago - 1 dependent package - 2 dependent repositories - 39 thousand downloads last month - 7,124 stars on GitHub - 1 maintainer
scalellm 0.2.6
A high-performance inference system for large language models.
19 versions - Latest release: 20 days ago - 594 downloads last month - 467 stars on GitHub - 1 maintainer
llama-trainer 0.2.1
Llama trainer utility
3 versions - Latest release: about 2 years ago - 25 downloads last month - 9 stars on GitHub - 1 maintainer
lorax-client 0.6.3
LoRAX Python Client
13 versions - Latest release: about 1 year ago - 4.38 thousand downloads last month - 3,433 stars on GitHub - 2 maintainers
periflow-sdk 0.1.2
PeriFlow SDK
4 versions - Latest release: over 2 years ago - 1 dependent repositories - 13 downloads last month - 18 stars on GitHub - 1 maintainer
edsl 1.0.4
Create and analyze LLM-based surveys
124 versions - Latest release: about 1 month ago - 2.16 thousand downloads last month - 20 stars on GitHub - 3 maintainers
Top 2.2% on pypi.org
openllm 0.6.30
OpenLLM: Self-hosting LLMs Made Easy.
195 versions - Latest release: 6 months ago - 3 dependent packages - 295 dependent repositories - 8.34 thousand downloads last month - 8,657 stars on GitHub - 3 maintainers
Top 3.5% on pypi.org
openllm-core 0.5.7
OpenLLM Core: Core components for OpenLLM.
85 versions - Latest release: over 1 year ago - 2 dependent packages - 8 dependent repositories - 1.91 thousand downloads last month - 8,657 stars on GitHub - 1 maintainer
Top 3.9% on pypi.org
openllm-client 0.5.7
OpenLLM Client: Interacting with OpenLLM HTTP/gRPC server, or any BentoML server.
85 versions - Latest release: over 1 year ago - 2 dependent packages - 8 dependent repositories - 1.05 thousand downloads last month - 8,657 stars on GitHub - 1 maintainer
Top 1.4% on pypi.org
gpt4all 2.8.2
Python bindings for GPT4All
54 versions - Latest release: about 1 year ago - 46 dependent packages - 168 dependent repositories - 96.4 thousand downloads last month - 70,285 stars on GitHub - 2 maintainers
picollmdemo 1.3.1
picoLLM Inference Engine demos
10 versions - Latest release: 5 months ago - 61 downloads last month - 154 stars on GitHub - 1 maintainer
picollm 1.3.1
picoLLM Inference Engine
10 versions - Latest release: 5 months ago - 226 downloads last month - 154 stars on GitHub - 1 maintainer
llm-inference 0.0.6 💰
Large Language Models Inference API and Applications
5 versions - Latest release: about 2 years ago - 37 downloads last month - 123 stars on GitHub - 1 maintainer
Top 8.9% on pypi.org
llama2-wrapper 0.1.14
Use llama2-wrapper as your local llama2 backend for Generative Agents / Apps
14 versions - Latest release: almost 2 years ago - 1 dependent repositories - 96 downloads last month - 1,958 stars on GitHub - 1 maintainer
unifyai 0.9.141
A Python package for interacting with the Unify API
160 versions - Latest release: 11 days ago - 3.66 thousand downloads last month - 183 stars on GitHub - 1 maintainer
friendli-client 1.5.8
Client of Friendli Suite.
45 versions - Latest release: 8 months ago - 4 dependent packages - 5.49 thousand downloads last month - 39 stars on GitHub - 1 maintainer
steadytext 2025.9.21
Deterministic text generation and embedding with zero configuration
34 versions - Latest release: 12 days ago - 547 downloads last month - 16 stars on GitHub - 1 maintainer
ant-ray-cpp-nightly 2.47.1
A subpackage of Ray which provides the Ray C++ API.
33 versions - Latest release: about 2 months ago - 7.16 thousand downloads last month - 39,095 stars on GitHub - 1 maintainer
ant-ray-nightly 2.47.1
Ray provides a simple, universal API for building distributed applications.
92 versions - Latest release: about 2 months ago - 7.76 thousand downloads last month - 39,095 stars on GitHub - 5 maintainers
exxa 0.6.4 💰
Exa - Pytorch
58 versions - Latest release: over 1 year ago - 2 dependent packages - 1 dependent repositories - 252 downloads last month - 23 stars on GitHub - 1 maintainer
libre-chat 0.0.6
Free and Open Source Large Language Model (LLM) chatbot web UI and API. Self-hosted, offline capa...
2 versions - Latest release: over 1 year ago - 53 downloads last month - 125 stars on GitHub - 1 maintainer
Top 1.2% on pypi.org
bentoml 1.4.25
BentoML: The easiest way to serve AI apps and models
205 versions - Latest release: 9 days ago - 13 dependent packages - 499 dependent repositories - 126 thousand downloads last month - 6,591 stars on GitHub - 3 maintainers
gpustack-runtime 0.1.5
GPUStack Runtime is library for detecting GPU resources and launching GPU workloads.
6 versions - Latest release: 1 day ago - 616 downloads last month - 3,753 stars on GitHub - 1 maintainer
vec-inf 0.7.0
Efficient LLM inference on Slurm clusters using vLLM.
11 versions - Latest release: about 1 month ago - 125 downloads last month - 58 stars on GitHub - 2 maintainers
lemonade-sdk 8.1.10
Lemonade SDK: Your LLM Aide for Validation and Deployment
23 versions - Latest release: 21 days ago - 2.66 thousand downloads last month - 1,254 stars on GitHub - 1 maintainer
archgw_modelserver 0.3.14
A model server for serving models
37 versions - Latest release: 3 days ago - 1.69 thousand downloads last month - 2,321 stars on GitHub - 1 maintainer
litgpt 0.5.11
Hackable implementation of state-of-the-art open-source LLMs
33 versions - Latest release: 23 days ago - 1 dependent package - 11.7 thousand downloads last month - 12,747 stars on GitHub - 2 maintainers
fpdb 0.0.0.dev2
Python package for debugging multi-processed code using PDB.
2 versions - Latest release: 4 months ago - 78 downloads last month - 12,747 stars on GitHub - 1 maintainer
mistral-inference 1.6.0
Official inference library for Mistral models
9 versions - Latest release: 7 months ago - 17.7 thousand downloads last month - 10,477 stars on GitHub - 2 maintainers
optillm 0.3.1
An optimizing inference proxy for LLMs.
72 versions - Latest release: 3 days ago - 3.92 thousand downloads last month - 2,929 stars on GitHub - 1 maintainer
asset-sentiment-analyzer 0.1.4
A sentiment analyzer package for financial assets and securities utilizing GPT models.
3 versions - Latest release: over 1 year ago - 38 downloads last month - 181 stars on GitHub - 1 maintainer
lazyllm-lmdeploy 0.7.1rc0
A toolset for compressing, deploying and serving LLM
1 version - Latest release: 7 months ago - 26 downloads last month - 6,298 stars on GitHub - 1 maintainer
talkingheads 0.5.4
A library to communicate with AI assistants such as ChatGPT, Claude, Copilot, Gemini, HuggingChat...
11 versions - Latest release: 11 months ago - 1 dependent repositories - 47 downloads last month - 323 stars on GitHub - 1 maintainer
sdap-os 0.1.2
Demo of usage of object storage for SDAP
3 versions - Latest release: over 3 years ago - 1 dependent repositories - 19 downloads last month - 36,881 stars on GitHub - 1 maintainer
kalavai-client 0.7.5
Client app for kalavai platform
57 versions - Latest release: 4 days ago - 630 downloads last month - 161 stars on GitHub - 1 maintainer
llmq 0.0.8
todo
8 versions - Latest release: 4 days ago - 159 downloads last month - 12 stars on GitHub - 1 maintainer
openvino-nvidia 2025.3.1
NVIDIA Plugin for OpenVINO Inference Engine Python* API
3 versions - Latest release: 14 days ago - 256 downloads last month - 8,915 stars on GitHub - 1 maintainer
openvino-arm 2022.1.0
OpenVINO(TM) Runtime
2 versions - Latest release: over 3 years ago - 1 dependent package - 2 dependent repositories - 51 downloads last month - 6,310 stars on GitHub - 1 maintainer
openvino-nightly 2024.5.0.dev20241105
OpenVINO(TM) Runtime
227 versions - Latest release: 11 months ago - 1 dependent repositories - 2.01 thousand downloads last month - 7,201 stars on GitHub - 1 maintainer
prompt-poet 0.0.49
Streamlines and simplifies prompt design for both developers and non-technical users with a low c...
50 versions - Latest release: 2 months ago - 7.29 thousand downloads last month - 1,104 stars on GitHub - 3 maintainers
dandy 1.0.0
Python Artificial Intelligence Framework
53 versions - Latest release: 4 days ago - 3.67 thousand downloads last month - 4 stars on GitHub - 1 maintainer
test-apptrace 0.5.7
package with monocle genAI tracing
5 versions - Latest release: about 1 month ago - 56 downloads last month - 39 stars on GitHub - 1 maintainer
kserve-mathking 0.10.0rc3
KServe Python SDK
3 versions - Latest release: over 2 years ago - 19 downloads last month - 4,593 stars on GitHub - 1 maintainer
gitleaks-py 0.3.1 💰
Find secrets with Gitleaks 🔑
4 versions - Latest release: almost 3 years ago - 561 downloads last month - 19,720 stars on GitHub - 1 maintainer
git-llm 0.1.3
The project integrates Git with a llm (OpenAI, LlamaCpp, and GPT-4-All) to extend the capabilitie...
3 versions - Latest release: over 2 years ago - 19 downloads last month - 70,285 stars on GitHub - 1 maintainer
local-llm-function-calling 0.1.23
A tool for generating function arguments and choosing what function to call with local LLMs
23 versions - Latest release: over 1 year ago - 98 downloads last month - 431 stars on GitHub - 1 maintainer
datatune 0.0.4
Your backend for LLM powered Big Data Apps
5 versions - Latest release: 26 days ago - 367 downloads last month - 109 stars on GitHub - 2 maintainers
datatune-client 0.0.2
A unified platform for ML data management and streaming
2 versions - Latest release: 8 months ago - 16 downloads last month - 109 stars on GitHub - 1 maintainer
eagle-llm 1.2.1
Accelerating LLMs by 3x with No Quality Loss
3 versions - Latest release: over 1 year ago - 41 downloads last month - 1,817 stars on GitHub - 1 maintainer
openai-assistants-api 1.0.0 💰
A Backend application leverging OpenAI AssistantAPI
1 version - Latest release: about 1 year ago - 33 downloads last month - 1 stars on GitHub - 1 maintainer
ray-for-mars 1.12.1
Ray provides a simple, universal API for building distributed applications.
2 versions - Latest release: over 3 years ago - 1 dependent repositories - 20 downloads last month - 39,095 stars on GitHub - 1 maintainer
ve-ray 2.46.0.1
Ray provides a simple, universal API for building distributed applications.
4 versions - Latest release: 7 days ago - 67 downloads last month - 39,095 stars on GitHub - 2 maintainers
Top 2.2% on pypi.org
ray-cpp 2.49.2
A subpackage of Ray which provides the Ray C++ API.
82 versions - Latest release: 13 days ago - 2 dependent packages - 4 dependent repositories - 19.3 thousand downloads last month - 39,095 stars on GitHub - 11 maintainers
ant-ray-cpp 3.0.0.dev0
A subpackage of Ray which provides the Ray C++ API.
1 version - Latest release: 2 months ago - 9 downloads last month - 39,095 stars on GitHub - 1 maintainer
ant-ray 2.44.1
Ray provides a simple, universal API for building distributed applications.
14 versions - Latest release: 6 months ago - 694 downloads last month - 39,095 stars on GitHub - 5 maintainers
Top 6.7% on pypi.org
secretflow-ray 2.2.0
Ray provides a simple, universal API for building distributed applications.
3 versions - Latest release: over 2 years ago - 3 dependent packages - 1 dependent repositories - 123 downloads last month - 39,095 stars on GitHub - 1 maintainer
fangyu-pypitest 0.8.0.dev4
A system for parallel and distributed Python that unifies the ML ecosystem.
1 version - Latest release: about 6 years ago - 1 dependent repositories - 17 downloads last month - 39,095 stars on GitHub - 1 maintainer
yunchang 0.6.3
a package for long context attention
21 versions - Latest release: 6 months ago - 40.7 thousand downloads last month - 570 stars on GitHub - 1 maintainer
vllm-cli 0.2.5
A CLI tool to conveniently serve LLMs with vLLM
13 versions - Latest release: about 1 month ago - 608 downloads last month - 414 stars on GitHub - 1 maintainer
superduper-vllm 0.10.0
Superduper allows users to work with self-hosted LLM models via [vLLM](https://github.com/vllm-pr...
9 versions - Latest release: about 1 month ago - 145 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-sklearn 0.10.0
superduper allows users to work with arbitrary sklearn estimators, with additional support for pr...
9 versions - Latest release: about 1 month ago - 151 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-dummy 0.5.0
Superduper: End-to-end framework for building custom AI applications and agents.
5 versions - Latest release: 9 months ago - 9 downloads last month - 5,213 stars on GitHub - 1 maintainer
kevj711-superduper 0.7.3
Build compositional and declarative AI applications and agents
1 version - Latest release: 3 months ago - 9 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-mongodb 0.10.0
SuperDuper MongoDB is a Python library that provides a high-level API for working with MongoDB. I...
20 versions - Latest release: about 1 month ago - 179 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-transformers 0.10.0
Transformers is a popular AI framework, and we have incorporated native support for Transformers ...
9 versions - Latest release: about 1 month ago - 128 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-framework 0.10.0
Build compositional and declarative AI applications and agents
60 versions - Latest release: about 1 month ago - 653 downloads last month - 5,213 stars on GitHub - 1 maintainer
kevj711-superduper-sql 0.7.2
`superduper_sql` is a plugin for SQL databases that allows you to use these databases as databack...
3 versions - Latest release: 3 months ago - 19 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-sentence-transformers 0.10.0
superduper allows users to work with self-hosted embedding models via [Sentence-Transformers](htt...
9 versions - Latest release: about 1 month ago - 154 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-qdrant 0.10.0
SuperDuper Lance is a Python library that provides a high-level API for working with Lance vector...
11 versions - Latest release: about 1 month ago - 161 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-openai 0.10.0
Superduper allows users to work with openai API models.
19 versions - Latest release: about 1 month ago - 167 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-torch 0.10.0
Superduper allows users to work with arbitrary `torch` models, with custom pre-, post-processing ...
9 versions - Latest release: about 1 month ago - 151 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-anthropic 0.10.0
Superduper allows users to work with anthropic API models. The key integration is the integration...
9 versions - Latest release: about 1 month ago - 135 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-snowflake 0.10.0
Superduper snowflake is a plugin for snowflake-framework that allows you to use Superduper as a b...
45 versions - Latest release: about 1 month ago - 338 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-chromadb 0.10.0
SuperDuper Lance is a Python library that provides a high-level API for working with Lance vector...
2 versions - Latest release: about 1 month ago - 221 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-sql 0.10.0
`superduper_sql` is a plugin for SQL databases that allows you to use these databases as databack...
8 versions - Latest release: about 1 month ago - 162 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-redis 0.10.0
superduper allows users to work with arbitrary sklearn estimators, with additional support for pr...
4 versions - Latest release: about 1 month ago - 128 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-cohere 0.10.0
Superduper allows users to work with cohere API models.
9 versions - Latest release: about 1 month ago - 150 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-lance 0.10.0
SuperDuper Lance is a Python library that provides a high-level API for working with Lance vector...
6 versions - Latest release: about 1 month ago - 124 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-pillow 0.10.0
SuperDuper Pillow is a plugin for SuperDuper that provides support for Pillow.
9 versions - Latest release: about 1 month ago - 131 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-ibis 0.5.4
Superduper ibis is a plugin for ibis-framework that allows you to use Superduper as a backend for...
17 versions - Latest release: 6 months ago - 37 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-sqlalchemy 0.5.8
Superduper sqlalchemy is a metadata plugin for Superduper that allows you to store metadata in a ...
21 versions - Latest release: 6 months ago - 31 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-jina 0.10.0
Superduper allows users to work with Jina Embeddings models through the Jina Embedding API.
9 versions - Latest release: about 1 month ago - 134 downloads last month - 5,213 stars on GitHub - 1 maintainer
superduper-llamacpp 0.10.0
Superduper allows users to work with self-hosted LLM models via [Llama.cpp](https://github.com/gg...
9 versions - Latest release: about 1 month ago - 127 downloads last month - 5,208 stars on GitHub - 1 maintainer
llmflows 0.2.1
LLMFlows - Simple, Explicit and Transparent LLM Apps
15 versions - Latest release: almost 2 years ago - 1 dependent repositories - 29 downloads last month - 701 stars on GitHub - 1 maintainer
deepsparse-ent 1.9.0
[DEPRECATED] An inference runtime offering GPU-class performance on CPUs and APIs to integrate ML...
15 versions - Latest release: 4 months ago - 2 dependent packages - 354 downloads last month - 3,158 stars on GitHub - 1 maintainer
okik 0.0.342
A Python package to serve python functions, classes, or .py files on a local server or cloud-base...
7 versions - Latest release: about 1 year ago - 44 downloads last month - 0 stars on GitHub - 1 maintainer
calculemus 0.0.2
Logical verification of probabilistic/language model 'intuitions'.
2 versions - Latest release: over 2 years ago - 22 downloads last month - 0 stars on GitHub - 1 maintainer
api4all 0.4.0
Easy-to-use LLM API from a state-of-the-art provider and comparison
12 versions - Latest release: over 1 year ago - 143 downloads last month - 3 stars on GitHub - 1 maintainer
autonomi-nos 0.0.10
Nitrous oxide system (NOS) for computer-vision.
21 versions - Latest release: about 2 years ago - 1 dependent repositories - 137 downloads last month - 144 stars on GitHub - 2 maintainers
swarm-squad-ep1 0.2.3
Swarm Squad Ep1: Surviving the jam
12 versions - Latest release: 4 months ago - 56 downloads last month - 0 stars on GitHub - 1 maintainer
torch-nos 0.3.0
Nitrous Oxide for your AI Infrastructure.
14 versions - Latest release: over 1 year ago - 39 downloads last month - 144 stars on GitHub - 2 maintainers
scaledp 0.2.2
ScaleDP is a library for processing documents using Apache Spark and LLMs
104 versions - Latest release: 7 months ago - 746 downloads last month - 15 stars on GitHub - 1 maintainer
sinapsis-chatbots 0.5.4
Mono repo with packages for text completion tasks
14 versions - Latest release: 23 days ago - 410 downloads last month - 24 stars on GitHub - 1 maintainer
orka-reasoning 0.9.3
Modular agent orchestrator for reasoning pipelines
46 versions - Latest release: 10 days ago - 947 downloads last month - 24 stars on GitHub - 1 maintainer