evaluation

🤘 awesome-semantic-segmentation

🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with OpenTelemetry, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23

Supercharge Your LLM Application Evaluations 🚀

Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

Arbitrary expression evaluation for golang

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

Python package for the evaluation of odometry and SLAM

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

The easiest tool for fine-tuning LLM models, synthetic data generation, and collaborating on datasets.

Building a modern functional compiler from first principles. (http://dev.stephendiehl.com/fun/)

SuperCLUE: 中文通用大模型综合性基准 | A Benchmark for Foundation Models in Chinese

Klipse is a JavaScript plugin for embedding interactive code snippets in tech blogs.

End-to-end Automatic Speech Recognition for Madarian and English in Tensorflow

A unified evaluation framework for large language models

An open-source visual programming environment for battle-testing prompts to LLMs.

Accelerating the development of large multimodal models (LMMs) with one-click evaluation module - lmms-eval.

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.