llm-as-a-judge

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

Evaluate your LLM's response with Prometheus and GPT4 💯

⚖️ The First Coding Agent-as-a-Judge

Inference-time scaling for LLMs-as-a-judge.

[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

xVerify: Efficient Answer Verifier for Reasoning Model Evaluations

CodeUltraFeedback: aligning large language models to coding preferences

This is the repo for the survey of Bias and Fairness in IR with LLMs.

Official implementation for "MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?"

First-of-its-kind AI benchmark for evaluating the protection capabilities of large language model (LLM) guard systems (guardrails and safeguards)

Solving Inequality Proofs with Large Language Models.

A set of tools to create synthetically-generated data from documents

Code and data for "Timo: Towards Better Temporal Reasoning for Language Models" (COLM 2024)

Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"

The official repository for our EMNLP 2024 paper, Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability.

Harnessing Large Language Models for Curated Code Reviews

Root Signals SDK

A set of examples demonstrating how to evaluate Generative AI augmented systems using traditional information retrieval and LLM-As-A-Judge validation techniques

LLM-as-judge evals as Semantic Kernel Plugins

A comprehensive study of the LLM-as-a-judge paradigm in a controlled setup that reveals new results about its strengths and weaknesses.