BigCode reposted this
Self-Instruct for CodeLLMs Transparent and Permissive! 👀 BigCode released a new StarCoder2-Instruct, the first entirely self-aligned code LLM trained with a transparent and permissive pipeline. 🧑🏻💻 It used itself to generate thousands of instruction-response pairs, which were then used to fine-tune—achieving 72.6 on HumanEval without relying on human annotations. 🤯 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 1️⃣ Collect Seed Code Snippets, e.g., functions with docstrings. 2️⃣ Apply type checking, decontamination (benchmarks), Quality Filtering & Near-Deduplication 3️⃣ Employ in-context learning to self-generate coding tasks from these snippets. 4️⃣ For each instruction, generate answers and tests using in-context learning. 5️⃣ Execute these tests in a sandbox environment and select responses that pass for training. 6️⃣ Create a Training Dataset with the validated responses 7️⃣ Fine-Tune StarCoder2-15B on the generated self-instruct dataset 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 🧮 15B parameter version with 8192 context 🔓 Fully open-source datasets and pipeline for distillation 📝 Fully self-aligned without human annotation 🏆 Outperforms CodeLlama-70B-Instruct (72.0) and GPT-4 (march) on HumanEval 🥇 Outperforming other open Models like Grok-1, Command-R+, and DBRX, and closely matching Snowflake Arctic 480B and Mixtral-8x22B-Instruct Blog: https://lnkd.in/eiPgEY7k Model: https://lnkd.in/e9astScY Dataset: https://lnkd.in/ewMYbuAj Code: https://lnkd.in/ed_wcZnJ Kudos to Leandro von Werra, Harm de Vries, and the whole BigCode team for paving the way for more transparent, permissive, and powerful language models! 🤗