starcoder2

Finetuning Starcoder2-3B for Code Completion on a single A100 GPU

Fine-tune StarCoder2-3b for SQL tasks on limited resources with LORA. LORA reduces model size for faster training on smaller datasets. StarCoder2 is a family of code generation models (3B, 7B, and 15B), trained on 600+ programming languages from The Stack v2 and some natural language text such as Wikipedia, Arxiv, and GitHub issues.

Replication package for the paper: "How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning"