ai-alignment

Control GenAI interactions with power, precision, and consistency using Conversation Modeling paradigms

A curated list of trustworthy deep learning papers. Daily updating...

PromptInject is a framework that assembles prompts in a modular fashion to provide a quantitative analysis of the robustness of LLMs to adversarial prompt attacks. 🏆 Best Paper Awards @ NeurIPS ML Safety Workshop 2022

Code accompanying the paper Pretraining Language Models with Human Preferences

📚 A curated list of papers & technical articles on AI Quality & Safety

How to Make Safe AI? Let's Discuss! 💡|💬|🙌|📚

[AAAI'25 Oral] "MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector".

Reading list for adversarial perspective and robustness in deep reinforcement learning.

A curated list of awesome academic research, books, code of ethics, data sets, institutes, maturity models, newsletters, principles, podcasts, reports, tools, regulations and standards related to Responsible, Trustworthy, and Human-Centered AI.

A curated list of awesome resources for Artificial Intelligence Alignment research

Directional Preference Alignment

Sparse probing paper full code.

Website to track people, organizations, and products (tools, websites, etc.) in AI safety

[TMLR 2024] Official implementation of "Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics"

Official Implementation of Nabla-GFlowNet (ICLR 2025)

An initiative to create concise and widely shareable educational resources, infographics, and animated explainers on the latest contributions to the community AI alignment effort. Boosting the signal and moving the community towards finding and building solutions.

Code and materials for the paper S. Phelps and Y. I. Russell, Investigating Emergent Goal-Like Behaviour in Large Language Models Using Experimental Economics, working paper, arXiv:2305.07970, May 2023

Scan your AI/ML models for problems before you put them into production.

Community list of awesome projects, apps, tools and more related to Polis.

IDA with RL and overseer failures