serving

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

A flexible, high-performance serving system for machine learning models

A Cloud Native Batch System (Project under CNCF)

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models

In this repository, I will share some useful notes and references about deploying deep learning-based models in production.

Serve, optimize and scale PyTorch models in production

The easiest way to deploy agents, MCP servers, models, RAG, pipelines and more. No MLOps. No YAML.

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

A course of learning LLM inference serving on Apple Silicon for systems engineers: build a tiny vLLM + Qwen.

Database system for AI-powered apps

TensorFlow template application for deep learning

A comprehensive guide to building RAG-based LLM applications for production.

A multi-modal vector database that supports upserts and vector queries using unified SQL (MySQL-Compatible) on structured and unstructured data, while meeting the requirements of high concurrency and ultra-low latency.

DELTA is a deep learning based natural language and speech processing platform. LF AI & DATA Projects: https://lfaidata.foundation/projects/delta/

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

A scalable inference server for models optimized with OpenVINO™

Generic and easy-to-use serving service for machine learning models

Python + Inference - Model Deployment library in Python. Simplest model inference server ever.

A unified end-to-end machine intelligence platform

A high-performance inference system for large language models, designed for production environments.