pypi.org "llm-serving" keyword
View the packages on the pypi.org package registry that are tagged with the "llm-serving" keyword.
Top 1.2% on pypi.org
190 versions - Latest release: 2 days ago - 13 dependent packages - 499 dependent repositories - 114 thousand downloads last month - 6,591 stars on GitHub - 3 maintainers
bentoml 1.4.10
BentoML: The easiest way to serve AI apps and models190 versions - Latest release: 2 days ago - 13 dependent packages - 499 dependent repositories - 114 thousand downloads last month - 6,591 stars on GitHub - 3 maintainers
superduper-sklearn 0.6.0
superduper allows users to work with arbitrary sklearn estimators, with additional support for pr...6 versions - Latest release: 24 days ago - 448 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-mongodb 0.6.2
SuperDuper MongoDB is a Python library that provides a high-level API for working with MongoDB. I...16 versions - Latest release: 17 days ago - 826 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-pillow 0.6.0
SuperDuper Pillow is a plugin for SuperDuper that provides support for Pillow.6 versions - Latest release: 24 days ago - 449 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-vllm 0.6.0
Superduper allows users to work with self-hosted LLM models via [vLLM](https://github.com/vllm-pr...6 versions - Latest release: 24 days ago - 371 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-sqlalchemy 0.5.8
Superduper sqlalchemy is a metadata plugin for Superduper that allows you to store metadata in a ...21 versions - Latest release: 12 days ago - 887 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-ibis 0.5.4
Superduper ibis is a plugin for ibis-framework that allows you to use Superduper as a backend for...17 versions - Latest release: 6 days ago - 865 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-sql 0.6.3
`superduper_sql` is a plugin for SQL databases that allows you to use these databases as databack...4 versions - Latest release: 16 days ago - 580 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-llamacpp 0.6.0
Superduper allows users to work with self-hosted LLM models via [Llama.cpp](https://github.com/gg...6 versions - Latest release: 24 days ago - 402 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-dummy 0.5.0
Superduper: End-to-end framework for building custom AI applications and agents.5 versions - Latest release: 3 months ago - 236 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-snowflake 0.6.3
Superduper snowflake is a plugin for snowflake-framework that allows you to use Superduper as a b...34 versions - Latest release: 5 days ago - 1.94 thousand downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-lance 0.6.0
SuperDuper Lance is a Python library that provides a high-level API for working with Lance vector...3 versions - Latest release: 24 days ago - 219 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-qdrant 0.6.0
SuperDuper Lance is a Python library that provides a high-level API for working with Lance vector...3 versions - Latest release: 24 days ago - 240 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-framework 0.6.7
Build compositional and declarative AI applications and agents52 versions - Latest release: 16 days ago - 3.05 thousand downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-transformers 0.6.0
Transformers is a popular AI framework, and we have incorporated native support for Transformers ...6 versions - Latest release: 24 days ago - 445 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-jina 0.6.0
Superduper allows users to work with Jina Embeddings models through the Jina Embedding API.6 versions - Latest release: 24 days ago - 368 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-redis 0.6.0
superduper allows users to work with arbitrary sklearn estimators, with additional support for pr...1 version - Latest release: 24 days ago - 166 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-sentence-transformers 0.6.0
superduper allows users to work with self-hosted embedding models via [Sentence-Transformers](htt...6 versions - Latest release: 24 days ago - 471 downloads last month - 5,032 stars on GitHub - 1 maintainer
superduper-torch 0.6.0
Superduper allows users to work with arbitrary `torch` models, with custom pre-, post-processing ...6 versions - Latest release: 24 days ago - 374 downloads last month - 5,022 stars on GitHub - 1 maintainer
superduper-cohere 0.6.0
Superduper allows users to work with cohere API models.6 versions - Latest release: 24 days ago - 303 downloads last month - 5,022 stars on GitHub - 1 maintainer
superduper-openai 0.6.0
Superduper allows users to work with openai API models.16 versions - Latest release: 24 days ago - 620 downloads last month - 5,022 stars on GitHub - 1 maintainer
superduper-anthropic 0.6.0
Superduper allows users to work with anthropic API models. The key integration is the integration...6 versions - Latest release: 24 days ago - 321 downloads last month - 5,022 stars on GitHub - 1 maintainer
periflow-sdk 0.1.2
PeriFlow SDK4 versions - Latest release: about 2 years ago - 1 dependent repositories - 102 downloads last month - 18 stars on GitHub - 1 maintainer
lorax-client 0.6.3
LoRAX Python Client13 versions - Latest release: 7 months ago - 4.93 thousand downloads last month - 2,165 stars on GitHub - 2 maintainers
friendli-client 1.5.8
Client of Friendli Suite.40 versions - Latest release: 3 months ago - 4 dependent packages - 59.4 thousand downloads last month - 39 stars on GitHub - 1 maintainer
Top 3.4% on pypi.org
56 versions - Latest release: 5 days ago - 46 dependent packages - 5 dependent repositories - 2.31 million downloads last month - 25,904 stars on GitHub - 4 maintainers
vllm 0.8.4 💰
A high-throughput and memory-efficient inference and serving engine for LLMs56 versions - Latest release: 5 days ago - 46 dependent packages - 5 dependent repositories - 2.31 million downloads last month - 25,904 stars on GitHub - 4 maintainers
byzerllm 0.1.181 💰
ByzerLLM: Byzer LLM177 versions - Latest release: 13 days ago - 1 dependent package - 2 dependent repositories - 15.4 thousand downloads last month - 25,904 stars on GitHub - 1 maintainer
hive-vllm 0.0.1 💰
a1 version - Latest release: about 1 year ago - 36 downloads last month - 25,551 stars on GitHub - 1 maintainer
vllm-xft 0.5.5.3 💰
A high-throughput and memory-efficient inference and serving engine for LLMs11 versions - Latest release: about 1 month ago - 358 downloads last month - 25,904 stars on GitHub - 2 maintainers
moe-kernels 0.8.2 💰
MoE kernels15 versions - Latest release: 3 months ago - 319 downloads last month - 25,904 stars on GitHub - 1 maintainer
llm_math 0.2.0 💰
A tool designed to evaluate the performance of large language models on mathematical tasks.5 versions - Latest release: 6 months ago - 104 downloads last month - 44,312 stars on GitHub - 1 maintainer
nextai-vllm 0.0.7 💰
A high-throughput and memory-efficient inference and serving engine for LLMs6 versions - Latest release: 12 months ago - 147 downloads last month - 25,904 stars on GitHub - 1 maintainer
ai-dynamo-vllm 0.7.2 💰
A high-throughput and memory-efficient inference and serving engine for LLMs2 versions - Latest release: about 1 month ago - 1.85 thousand downloads last month - 44,312 stars on GitHub - 1 maintainer
tilearn-test01 0.1 💰
A high-throughput and memory-efficient inference and serving engine for LLMs1 version - Latest release: about 1 year ago - 25 downloads last month - 25,700 stars on GitHub - 1 maintainer
wxy-test 0.8.1 💰
A high-throughput and memory-efficient inference and serving engine for LLMs1 version - Latest release: 2 months ago - 38 downloads last month - 44,312 stars on GitHub - 1 maintainer
vllm-rocm 0.6.3 💰
A high-throughput and memory-efficient inference and serving engine for LLMs with AMD GPU support1 version - Latest release: 6 months ago - 44 downloads last month - 44,312 stars on GitHub - 1 maintainer
vllm-consul 0.2.1
A high-throughput and memory-efficient inference and serving engine for LLMs5 versions - Latest release: over 1 year ago - 152 downloads last month - 12,665 stars on GitHub - 1 maintainer
vllm-acc 0.4.1 💰
A high-throughput and memory-efficient inference and serving engine for LLMs8 versions - Latest release: 12 months ago - 226 downloads last month - 25,904 stars on GitHub - 1 maintainer
vllm-emissary 0.1.0 💰
A high-throughput and memory-efficient inference and serving engine for LLMs2 versions - Latest release: 12 days ago - 211 downloads last month - 44,312 stars on GitHub - 1 maintainer
llm_atc 0.1.7 💰
Tools for fine tuning and serving LLMs6 versions - Latest release: over 1 year ago - 238 downloads last month - 25,904 stars on GitHub - 1 maintainer
vllm-online 0.4.2 💰
A high-throughput and memory-efficient inference and serving engine for LLMs2 versions - Latest release: 12 months ago - 54 downloads last month - 25,904 stars on GitHub - 1 maintainer
marlin-kernels 0.3.7 💰
Marlin quantization kernels11 versions - Latest release: 3 months ago - 244 downloads last month - 25,904 stars on GitHub - 1 maintainer
vllm-npu 0.4.2 💰
A high-throughput and memory-efficient inference and serving engine for LLMs3 versions - Latest release: 3 months ago - 167 downloads last month - 44,312 stars on GitHub - 1 maintainer
llm-engines 0.0.23 💰
A unified inference engine for large language models (LLMs) including open-source models (VLLM, S...22 versions - Latest release: about 1 month ago - 774 downloads last month - 25,904 stars on GitHub - 1 maintainer
tilearn-infer 0.3.3 💰
A high-throughput and memory-efficient inference and serving engine for LLMs3 versions - Latest release: 12 months ago - 68 downloads last month - 25,904 stars on GitHub - 1 maintainer
llm-swarm 0.1.1 💰
A high-throughput and memory-efficient inference and serving engine for LLMs2 versions - Latest release: about 1 year ago - 89 downloads last month - 25,904 stars on GitHub - 1 maintainer
superlaser 0.0.6 💰
An MLOps library for LLM deployment w/ the vLLM engine on RunPod's infra.6 versions - Latest release: about 1 year ago - 229 downloads last month - 25,904 stars on GitHub - 1 maintainer
yatai 0.0.1
Model and deployment management for BentoML1 version - Latest release: over 3 years ago - 1 dependent repositories - 51 downloads last month - 7,597 stars on GitHub - 1 maintainer
sentencebertservice 20211205152102
BentoML generated model module1 version - Latest release: over 3 years ago - 1 dependent repositories - 29 downloads last month - 7,597 stars on GitHub - 1 maintainer
bentoml-unsloth 0.1.2
BentoML: The easiest way to serve AI apps and models3 versions - Latest release: 7 months ago - 139 downloads last month - 7,597 stars on GitHub - 1 maintainer
bentoml-core 0.1.0
The rust core of BentoML: The Unified Model Serving Framework2 versions - Latest release: almost 2 years ago - 98 downloads last month - 7,597 stars on GitHub - 2 maintainers
sgl-kernel 0.0.9
Kernel Library for SGLang48 versions - Latest release: 4 days ago - 230 thousand downloads last month - 13,233 stars on GitHub - 3 maintainers
skypilot-nightly 1.0.0.dev20250417
SkyPilot: An intercloud broker for the clouds639 versions - Latest release: 2 days ago - 36.7 thousand downloads last month - 7,680 stars on GitHub - 3 maintainers
vllm-cpm 0.2.2
A high-throughput and memory-efficient inference and serving engine for LLMs1 version - Latest release: about 1 year ago - 25 downloads last month - 7,680 stars on GitHub - 1 maintainer
skypilot-im 0.7.0
SkyPilot: An intercloud broker for the clouds1 version - Latest release: 2 months ago - 70 downloads last month - 7,680 stars on GitHub - 1 maintainer
nextai-prism 1.0.20
Prism: An intercloud broker for the clouds21 versions - Latest release: about 1 year ago - 589 downloads last month - 7,680 stars on GitHub - 1 maintainer
Top 4.1% on pypi.org
39 versions - Latest release: 11 days ago - 11 dependent packages - 3 dependent repositories - 76.5 thousand downloads last month - 7,680 stars on GitHub - 3 maintainers
skypilot 0.8.1
SkyPilot: An intercloud broker for the clouds39 versions - Latest release: 11 days ago - 11 dependent packages - 3 dependent repositories - 76.5 thousand downloads last month - 7,680 stars on GitHub - 3 maintainers
skypilot-impala 0.7.0
SkyPilot: An intercloud broker for the clouds1 version - Latest release: 2 months ago - 69 downloads last month - 7,680 stars on GitHub - 1 maintainer
trainy-skypilot-nightly 1.0.0.dev20250304
SkyPilot: An intercloud broker for the clouds172 versions - Latest release: about 2 months ago - 3.46 thousand downloads last month - 7,680 stars on GitHub - 1 maintainer
nextai-star 0.2.33
An open platform for training, serving, and evaluating large language model based chatbots by nex...9 versions - Latest release: over 1 year ago - 212 downloads last month - 7,680 stars on GitHub - 1 maintainer
Top 2.2% on pypi.org
194 versions - Latest release: 3 days ago - 3 dependent packages - 295 dependent repositories - 14.7 thousand downloads last month - 8,657 stars on GitHub - 3 maintainers
openllm 0.6.29
OpenLLM: Self-hosting LLMs Made Easy.194 versions - Latest release: 3 days ago - 3 dependent packages - 295 dependent repositories - 14.7 thousand downloads last month - 8,657 stars on GitHub - 3 maintainers
Top 3.9% on pypi.org
85 versions - Latest release: 10 months ago - 2 dependent packages - 8 dependent repositories - 2.94 thousand downloads last month - 8,657 stars on GitHub - 1 maintainer
openllm-client 0.5.7
OpenLLM Client: Interacting with OpenLLM HTTP/gRPC server, or any BentoML server.85 versions - Latest release: 10 months ago - 2 dependent packages - 8 dependent repositories - 2.94 thousand downloads last month - 8,657 stars on GitHub - 1 maintainer
Top 3.5% on pypi.org
85 versions - Latest release: 10 months ago - 2 dependent packages - 8 dependent repositories - 5.31 thousand downloads last month - 8,657 stars on GitHub - 1 maintainer
openllm-core 0.5.7
OpenLLM Core: Core components for OpenLLM.85 versions - Latest release: 10 months ago - 2 dependent packages - 8 dependent repositories - 5.31 thousand downloads last month - 8,657 stars on GitHub - 1 maintainer
Top 0.2% on pypi.org
124 versions - Latest release: 23 days ago - 310 dependent packages - 3,641 dependent repositories - 6.98 million downloads last month - 33,648 stars on GitHub - 26 maintainers
ray 2.44.1
Ray provides a simple, universal API for building distributed applications.124 versions - Latest release: 23 days ago - 310 dependent packages - 3,641 dependent repositories - 6.98 million downloads last month - 33,648 stars on GitHub - 26 maintainers
vllm-ascend 0.7.3rc2 💰
vLLM Ascend backend plugin4 versions - Latest release: 22 days ago - 327 downloads last month - 477 stars on GitHub - 1 maintainer
sglang 0.4.5
SGLang is yet another fast serving framework for large language models and vision language models.90 versions - Latest release: 12 days ago - 71.6 thousand downloads last month - 13,055 stars on GitHub - 3 maintainers
dblcsgen 0.2.11
DBLC Fast Structured Generation17 versions - Latest release: 11 months ago - 598 downloads last month - 13,055 stars on GitHub - 1 maintainer
ant-ray-nightly 3.0.0.dev20250405
Ray provides a simple, universal API for building distributed applications.69 versions - Latest release: 14 days ago - 14 thousand downloads last month - 36,453 stars on GitHub - 3 maintainers
fangyu-pypitest 0.8.0.dev4
A system for parallel and distributed Python that unifies the ML ecosystem.1 version - Latest release: over 5 years ago - 1 dependent repositories - 40 downloads last month - 36,453 stars on GitHub - 1 maintainer
Top 2.2% on pypi.org
74 versions - Latest release: 23 days ago - 2 dependent packages - 4 dependent repositories - 35.6 thousand downloads last month - 36,453 stars on GitHub - 11 maintainers
ray-cpp 2.44.1
A subpackage of Ray which provides the Ray C++ API.74 versions - Latest release: 23 days ago - 2 dependent packages - 4 dependent repositories - 35.6 thousand downloads last month - 36,453 stars on GitHub - 11 maintainers
Top 6.7% on pypi.org
3 versions - Latest release: over 2 years ago - 3 dependent packages - 1 dependent repositories - 234 downloads last month - 36,453 stars on GitHub - 1 maintainer
secretflow-ray 2.2.0
Ray provides a simple, universal API for building distributed applications.3 versions - Latest release: over 2 years ago - 3 dependent packages - 1 dependent repositories - 234 downloads last month - 36,453 stars on GitHub - 1 maintainer
myray 0.1.1
my ray desc2 versions - Latest release: about 4 years ago - 1 dependent repositories - 82 downloads last month - 36,453 stars on GitHub - 1 maintainer
ray-for-mars 1.12.1
Ray provides a simple, universal API for building distributed applications.2 versions - Latest release: almost 3 years ago - 1 dependent repositories - 81 downloads last month - 36,453 stars on GitHub - 1 maintainer
ant-ray 2.44.1
Ray provides a simple, universal API for building distributed applications.14 versions - Latest release: 21 days ago - 4.8 thousand downloads last month - 36,453 stars on GitHub - 3 maintainers
sdap-os 0.1.2
Demo of usage of object storage for SDAP3 versions - Latest release: almost 3 years ago - 1 dependent repositories - 104 downloads last month - 34,339 stars on GitHub - 1 maintainer
horizon-takeoff 0.0.4.4
Auto-deploy the Takeoff Server on AWS for LLM inference5 versions - Latest release: over 1 year ago - 238 downloads last month - 0 stars on GitHub - 1 maintainer
chitu 0.1.2
A high-performance inference framework for large language models, focusing on efficiency, flexibi...3 versions - Latest release: 18 days ago - 253 downloads last month - 1,086 stars on GitHub - 1 maintainer
quick-llama 0.0.8 💰
Run Ollama models easily, anywhere – including online platforms like Google Colab6 versions - Latest release: 4 months ago - 248 downloads last month - 4 stars on GitHub - 1 maintainer
Top 7.4% on pypi.org
49 versions - Latest release: about 2 months ago - 1 dependent package - 1 dependent repositories - 11.2 thousand downloads last month - 836 stars on GitHub - 2 maintainers
mosec 0.9.2
Model Serving made Efficient in the Cloud49 versions - Latest release: about 2 months ago - 1 dependent package - 1 dependent repositories - 11.2 thousand downloads last month - 836 stars on GitHub - 2 maintainers
mosec-tiinfer 0.0.7
Model Serving made Efficient in the Cloud.1 version - Latest release: about 2 years ago - 112 downloads last month - 836 stars on GitHub - 1 maintainer
tokenswift 0.1.1
Framework for Accelerating LLM Generation3 versions - Latest release: about 2 months ago - 126 downloads last month - 85 stars on GitHub - 1 maintainer
okik 0.0.342
A Python package to serve python functions, classes, or .py files on a local server or cloud-base...7 versions - Latest release: 9 months ago - 294 downloads last month - 0 stars on GitHub - 1 maintainer
superduperdb 0.2.4
🔮 Bring AI to your favourite database 🔮29 versions - Latest release: 8 months ago - 1 dependent repositories - 581 downloads last month - 5,022 stars on GitHub - 1 maintainer
webapp-builder 1.0.0
Webapp Builder -- LLM/LVM Code Generator2 versions - Latest release: over 1 year ago - 32 downloads last month - 3 stars on GitHub - 1 maintainer
periflow-client 0.2.2
Client of PeriFlow, the fastest generative AI serving available.17 versions - Latest release: about 1 year ago - 441 downloads last month - 43 stars on GitHub - 1 maintainer
benchmark-llm-serving 1.0.3
A library to benchmark LLMs via their API exposure5 versions - Latest release: 9 months ago - 191 downloads last month - 6 stars on GitHub - 1 maintainer
happy-vllm 1.2.5
happy_vllm is a REST API for vLLM, production ready24 versions - Latest release: 22 days ago - 837 downloads last month - 18 stars on GitHub - 1 maintainer
llmailbot 0.4.3
A service for automatically replying to emails using LLMs.5 versions - Latest release: 29 days ago - 0 downloads last month - 0 stars on GitHub - 1 maintainer
faster-outlines 2024.11.14
Faster, lazy backend for the `Outlines` library5 versions - Latest release: 5 months ago - 679 downloads last month - 4 stars on GitHub - 1 maintainer
hami-core 1.0.16
A Python library for batched backend scheduling7 versions - Latest release: about 1 month ago - 1.99 thousand downloads last month - 142 stars on GitHub - 1 maintainer
cortecs-py 0.1.2
Lightweight wrapper for cortecs.ai enabling ⚡️ instant provisioning7 versions - Latest release: about 1 month ago - 299 downloads last month - 6 stars on GitHub - 1 maintainer
Related Keywords
pytorch
61
mlops
58
llmops
58
inference
53
machine-learning
47
llm
47
python
39
data-science
39
llm-inference
38
model-serving
33
llama
31
ai
29
tpu
29
transformer
28
ml
27
deep-learning
25
gpt
25
transformers
24
cuda
24
torch
22
semantic-search
22
rag
22
pretrained-models
22
distributed-ml
22
database
22
data
22
chatbot
22
vector-database
22
mongodb
22
databases
22
vector-search
22
inferentia
21
rocm
21
trainium
21
xpu
21
amd
21
serving
14
deployment
12
gpu
11
deepseek
11
tensorflow
11
ray
10
rllib
9
reinforcement-learning
9
optimization
9
model-selection
9
java
9
hyperparameter-search
9
hyperparameter-optimization
9
automl
9
parallel
9
distributed
9
cloud-computing
8
spot-instances
8
cloud-management
8
cost-management
8
cost-optimization
8
distributed-training
8
multicloud
8
ml-platform
8
MLOps
8
ml-infrastructure
8
generative-ai
8
llm-training
8
finops
8
job-scheduler
8
hyperparameter-tuning
8
job-queue
8
qwen
7
hyperparameter-tuningreinforcement-learning
6
Model Deployment
6
Model Serving
6
hpu
6
BentoML
6
llama2
6
LLMOps
5
AI
5
inference-platform
5
ml-engineering
5
model-inference-service
5
multimodal
5
llms
5
llm-ops
5
mistral
5
mpt
4
vllm
4
fine-tuning
4
ai-inference
4
Vicuna
3
Transformers
3
StableLM
3
Serverless
3
PyTorch
3
Llama 2
3
bentoml
3
falcon
3
model-inference
3
open-source-llm
3
openllm
3
stablelm
3