ggml

LLM inference in C/C++

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.

[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models

Diffusion model(SD,Flux,Wan,...) inference in pure C/C++

llama and other large language models on iOS and MacOS offline using GGML library.

INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model

Go manage your Ollama models

Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization

Suno AI's Bark model in C/C++ for fast text-to-speech generation

Whisper Dart is a cross platform library for dart and flutter that allows converting audio to text / speech to text / inference from Open AI models

Self-evaluating interview for AI coders

Run inference on MPT-30B using CPU

Port of MiniGPT4 in C++ (4bit, 5bit, 6bit, 8bit, 16bit CPU inference with GGML)

This custom_node for ComfyUI adds one-click "Virtual VRAM" for any UNet and CLIP loader as well MultiGPU integration in WanVideoWrapper, managing the offload/Block Swap of layers to DRAM *or* VRAM to maximize the latent space of your card. Also includes nodes for directly loading entire components (UNet, CLIP, VAE) onto the device you choose

Running any GGUF SLMs/LLMs locally, on-device in Android

CLIP inference in plain C/C++ with no extra dependencies

Large Language Models for All, 🦙 Cult and More, Stay in touch !

WIP Library Text To Speech From Suno AI's Bark in C/C++ for fast inference

GENERAL Ai Library For DART & Flutter

Inference Vision Transformer (ViT) in plain C/C++ with ggml