Repository navigation

llm-evaluation-framework

Website
Wikipedia

Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

大语言模型 prompt-engineering prompts llmops prompt-testing Testing rag evaluation evaluation-framework llm-eval llm-evaluation llm-evaluation-framework 持续集成 CI/CD pentesting red-teaming vulnerability-scanners

TypeScript

6232

511

17 小时前

confident-ai / deepeval

The LLM Evaluation Framework

evaluation-metrics evaluation-framework llm-evaluation llm-evaluation-framework llm-evaluation-metrics

Python

6008

523

5 小时前

msoedov / agentic_security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

llm-security ai-red-team llm-evaluation llm-evaluation-framework prompt-testing agent-framework

Python

1296

204

4 天前

JinjieNi / MixEval

The official evaluation suite and dynamic data release for MixEval.

benchmark evaluation evaluation-framework foundation-models large-language-model large-language-models large-multimodal-models llm-evaluation llm-evaluation-framework llm-inference

Python

235

5 个月前

cvs-health / langfair

LangFair is a Python library for conducting use-case level LLM bias and fairness assessments

人工智能 bias bias-detection fairness fairness-ai fairness-ml fairness-testing large-language-models 大语言模型 responsible-ai Python ai-safety llm-evaluation llm-evaluation-framework llm-evaluation-metrics

Python

201

3 天前

parea-ai / parea-sdk-py

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

大语言模型 llm-evaluation llm-tools llmops llm-eval llm-evaluation-framework prompt-engineering generative-ai good-first-issue 监控

Python

2 个月前

Addepto / contextcheck

MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.

大语言模型 llm-evaluation rag Testing chatbot-framework Open Source ai-chat ai-testing-tool large-language-models 持续集成 llm-evaluation-framework

Python

4 个月前

zhuohaoyu / KIEval

[ACL'24] A Knowledge-grounded Interactive Evaluation Framework for Large Language Models

explainable-ai 大语言模型 llm-evaluation llm-evaluation-framework llm-evaluation-metrics 机器学习

Python

9 个月前

multinear / multinear

Develop reliable AI apps

evaluation 大语言模型 llms reliability llm-eval llm-evaluation llm-evaluation-framework

Svelte

2 天前

flexpa / llm-fhir-eval

Benchmarking Large Language Models for FHIR

evals fhir 大语言模型 llm-evaluation-framework

5 个月前

aws-samples / fm-leaderboarder

FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts

llm-evaluation llm-evaluation-framework

Python

6 个月前

honeyhiveai / realign

Realign is a testing and simulation framework for AI applications.

人工智能 alignment evaluation llms prompt-engineering red-teaming Simulation llm-eval llm-evaluation llm-evaluation-framework llmops rag

Python

4 个月前

Networks-Learning / prediction-powered-ranking

Code for "Prediction-Powered Ranking of Large Language Models", NeurIPS 2024.

llm-eval llm-evaluation llm-evaluation-framework ranking-algorithm

Jupyter Notebook

6 个月前

pyladiesams / eval-llm-based-apps-jan2025

Create an evaluation framework for your LLM based app. Incorporate it into your test suite. Lay the monitoring foundation.

大语言模型 llmops llms workshop llm-eval llm-evaluation-framework llm-evaluation-metrics llm-monitoring

Jupyter Notebook

3 个月前

parea-ai / parea-sdk-ts

TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

大语言模型 llm-evaluation llm-evaluation-framework llm-tools llms llm-eval prompt-engineering

TypeScript

3 个月前

yukinagae / genkitx-promptfoo

Community Plugin for Genkit to use Promptfoo

人工智能 evaluation evaluation-framework Firebase genkit 大语言模型 llm-eval llm-evaluation llm-evaluation-framework llmops 插件 prompt prompt-testing prompts Testing

TypeScript

4 个月前

yuzu-ai / ShinRakuda

Shin Rakuda is a comprehensive framework for evaluating and benchmarking Japanese large language models, offering researchers and developers a flexible toolkit for assessing LLM performance across diverse datasets.

大语言模型 llm-eval llm-evaluation llm-evaluation-framework japanese

Python

7 个月前