Repository navigation
flash-attention
- Website
- Wikipedia
The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud.
中文LLaMA-2 & Alpaca-2大模型二期项目 + 64K超长上下文模型 (Chinese LLaMA-2 & Alpaca-2 LLMs with 64K long context models)
Official release of InternLM series (InternLM, InternLM2, InternLM2.5, InternLM3).
📚LeetCUDA: Modern CUDA Learn Notes with PyTorch for Beginners🐑, 200+ CUDA Kernels, Tensor Cores, HGEMM, FA-2 MMA.🎉
📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉
MoBA: Mixture of Block Attention for Long-Context LLMs
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
[CVPR 2025 Highlight] The official CLIP training codebase of Inf-CL: "Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss". A super memory-efficiency CLIP training scheme.
🤖FFPA: Extend FlashAttention-2 with Split-D, ~O(1) SRAM complexity for large headdim, 1.8x~3x↑🎉 vs SDPA EA.
Triton implementation of FlashAttention2 that adds Custom Masks.
Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.
Performance of the C++ interface of flash attention and flash attention v2 in large language model (LLM) inference scenarios.
Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using CUDA core for the decoding stage of LLM inference.
Fast and memory efficient PyTorch implementation of the Perceiver with FlashAttention.
Python package for rematerialization-aware gradient checkpointing
A flexible and efficient implementation of Flash Attention 2.0 for JAX, supporting multiple backends (GPU/TPU/CPU) and platforms (Triton/Pallas/JAX).
Utilities for efficient fine-tuning, inference and evaluation of code generation models
An simple pytorch implementation of Flash MultiHead Attention
