llama-cpp

A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device. New: Code Llama support!

A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.

Maid is a cross-platform Flutter app for interfacing with GGUF / llama.cpp models locally, and with Ollama and OpenAI models remotely.

Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level

llama.go is like llama.cpp in pure Golang!

Create characters in Unity with LLMs!

prima.cpp: Speeding up 70B-scale LLM inference on low-resource everyday home clusters

React Native binding of llama.cpp

Self-evaluating interview for AI coders

Run AI ✨ assistant locally! with simple API for Node.js 🚀

Build and run AI agents using Docker Compose. A collection of ready-to-use examples for orchestrating open-source LLMs, tools, and agent runtimes.

LLama.cpp rust bindings

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

This repo is to showcase how you can run a model locally and offline, free of OpenAI dependencies.

Review/Check GGUF files and estimate the memory usage and maximum tokens per second.

Local ML voice chat using high-end models.

Run LLMs locally. A clojure wrapper for llama.cpp.

Booster - open accelerator for LLM models. Better inference and debugging for AI hackers

A Pure Rust based LLM (Any LLM based MLLM such as Spark-TTS) Inference Engine, powering by Candle framework.

Making offline AI models accessible to all types of edge devices.