type
status
date
slug
summary
tags
category
icon
password
URL
1.Highlight2.Motivation & Background3.Contribution4.Method4.1 Evol-Instruct(Instruction Data Evolution)4.1.1 What is Data Evolution?4.1.2 Automatic Instruction Data Evolution4.2 Fine-tuning LLM on Evolved Instruction4.2.1 Evol Instruction Dataset4.2.2 WizardLM4.3 Test4.3.1 Testset4.3.2 Human Evaluation4.2.2 GPT-4 Automatic Evaluation5.Conclusion6. Extension:6.1 WizardMath:6.2 WizardCoder
WizardLM: Empowering Large Language Models to Follow Complex Instructions
Microsoft & PKU
arxiv
1.Highlight
![https://tatsu-lab.github.io/alpaca_eval/
Up to 2023/8/26](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F74126857-d2e6-4aef-9476-a30b7532f209%2FUntitled.png?table=block&id=9f1262fa-790f-4b49-9dfb-bc941160ddc1)
2.Motivation & Background
目前LLMs在open domain instructions 上 pretrain 是目前大模型训练的主流方法,也取得了很好的效果。
创建这些Instruction data需要消耗大量的人力和时间,而生成有些难度和较为复杂的instruction更是这样。
ShareGPT
用户生成的open domain instruction data, 但是其中大多数问题难度处于简单或者中等, 一方面可能是用户中expert比例较少, 另一方面可能是创建较为困难的instruction需要耗费更多的时间和精力
有没有一种方法能够借助LLM而不是人工来批量生成不同难度和复杂性等级的Instruction data?
3.Contribution
- 提出Evol-instruct:一种利用大模型进行open domain instruction数据生成的方法
- 构建了一个Evol-instruct 测试数据集:包含多种skill和各个level难度Instruction的测试数据集
- 提出了WizardLM模型:一个基于LLaMA的利用Evol-instruct方法fine-tuning的更为强大的LLM
4.Method
4.1 Evol-Instruct(Instruction Data Evolution)
4.1.1 What is Data Evolution?
- 初始instruction dataset
- 每次进化时:
- 利用LLM instruction evolution prompt, 将进化为, 然后利用LLM生成对应回应, 从而获得
- M次迭代后, 可以得到dataset
4.1.2 Automatic Instruction Data Evolution
Pipeline
:- 指令进化
- 生成回复
- 终止进化(过滤掉不能进化的指令)
指令进化器
: 通过特定prompts进化instruction的LLM两类进化Prompts
:- In-depth Evolving Prompts: 强化指令, 让指令进化得更为困难和复杂
- 五种prompt:
- add constraints
- deepening
- concretizing
- increased reasoning steps
- complicating input
- 难度增长控制:每次增加一点难度
add constraints example:
I want you act as a Prompt Rewriter.
Your objective is to rewrite a given prompt into a more complex version to make those famous AI systems (e.g., ChatGPT and GPT4) a bit harder to handle.
But the rewritten prompt must be reasonable and must be understood and responded by humans.
Your rewriting cannot omit the non-text parts such as the table and code in #Given Prompt#:. Also, please do not omit the input in #Given Prompt#.
You SHOULD complicate the given prompt using the following method:
Please add one more constraints/requirements into #Given Prompt#
You should try your best not to make the #Rewritten Prompt# become verbose, #Rewritten Prompt# can only add 10 to 20 words into #Given Prompt#.
‘#Given Prompt#’, ‘#Rewritten Prompt#’, ‘given prompt’ and ‘rewritten prompt’ are not allowed to appear in
#Rewritten Prompt#
#Given Prompt#:
<Here is instruction.>
#Rewritten Prompt#:
- In-Breadth Evolving Prompts:增强主题覆盖、技能覆盖和整体数据集多样性。设计一个prompt去根据给定的指令生成一个全新的指令
In Breadth Prompts example:
I want you act as a Prompt Creator.
Your goal is to draw inspiration from the #Given Prompt# to create a brand new prompt.
This new prompt should belong to the same domain as the #Given Prompt# but be even more rare.
The LENGTH and difficulty level of the #Created Prompt# should be similar to that of the #Given Prompt#.
The #Created Prompt# must be reasonable and must be understood and responded by humans.
‘#Given Prompt#’, ‘#Created Prompt#’, ‘given prompt’ and ‘created prompt’ are not allowed to appear in
#Created Prompt#.
#Given Prompt#:
<Here is instruction.>
#Created Prompt#:
A case from ‘1+1=?’
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fb3d28bea-773c-4700-86d2-88ebf9815fc9%2FUntitled.png?table=block&id=445c2a41-f423-4a43-a203-07d08b5ce9dd)
生成Response
:- 利用生成Instruction的LLM生成对应的Response
进化中止
:判别进化失败的四种条件:- 与原始指令相比,进化后的指令没有提供任何信息增益。(使用ChatGPT进行确定)
- 进化的指令使得LLM很难产生响应。
eg:
当生成的响应包含“sorry”并且长度相对较短(即少于80个单词)时,LLM通常难以响应进化后的指令。
- LLM生成的响应只包含标点和终止词。
- 进化的指令中明显地复制了一些Evolution prompt中的词
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F275d93ba-202c-43ce-89c1-4977b1e4588e%2FUntitled.png?table=block&id=1f8979f1-e04f-466a-9f23-a570449e94e9)
4.2 Fine-tuning LLM on Evolved Instruction
4.2.1 Evol Instruction Dataset
将初始数据集和每轮进化后的数据融合在一起, 并对样本进行random shuffle 获得fine-tuing 数据集。
确保了数据集中不同难度级别的指令均匀分布,最大限度地提高了模型微调的平滑度。
initial dataset
: Alpaca 52K进化轮次M
:4共计250K instructions
4.2.2 WizardLM
利用上述生成的Evol instruction数据在LLaMA-7B上进行微调,并获得WizardLM
具体训练参数和更多细节可以参考论文4.2部分
4.3 Test
4.3.1 Testset
Evol-Instruct测试集包括来自各种来源(如在线开源项目、平台和论坛)的真实人类指令。
Testset包含29种不同的技能,这些技能代表了人类的主要需求,如代码生成和调试、数学、推理、写作等。
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Ff095931d-d54a-4a27-a9dd-6f73b027d41c%2FUntitled.png?table=block&id=1ab2e706-0c76-49a6-a58b-2d0106a3bf6b)
4.3.2 Human Evaluation
在Evol-instruct 测试集, Vicuna 测试集, Evol-instruct 高难度数据集上, 通过聘请10名受过良好教育的标记员, 对来自Alpaca、Vicuna-7b、WizardLM和ChatGPT的四种响应(隐藏来源)进行打分(1-5),
![- WizardLM的效果明显优于Alpaca和Vicuna-7b,证明了evolo - instruct的有效性。
- 在高难度指令(难度等级>= 8)中,WizardLM比ChatGPT更受人类标注者的青睐。](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F29a737dd-5d0c-4129-8192-322d622daa52%2FUntitled.png?table=block&id=8716070a-7e9a-45df-9b06-b30c95c757dd)
4.2.2 GPT-4 Automatic Evaluation
采用Vicuna提出的基于GPT-4的自动评估框架来评估LLM的性能。
- 遵循与Vicuna相同的GPT-4超参数,提示设置和评估方法。
- 为了减轻顺序偏差,在配对比较中交替放置WizardLM和其他模型:WizardLM是奇数id的第一个,偶数id的第二个。
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F581ef719-ed59-4271-bf67-e9e414dfb400%2FUntitled.png?table=block&id=c1a2ea6c-eccd-4daa-a3a5-2819d4a9e285)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fb159bc43-b0a1-4b03-accb-fd5a0210eaf8%2FUntitled.png?table=block&id=15f1dc13-622a-4ee9-96b0-d4d5dcfc7ae0)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F64a8f205-d264-4685-b213-90958cc67738%2FUntitled.png?table=block&id=85f763bc-f185-411b-965d-d6876a3d032c)
- WizardLM在所有难度水平超过Vicuna,在Easy、Hard技能上都超过Alpaca,在Hard技能上几乎达到了ChatGPT的88%。这表明WizardLM有潜力解决复杂的问题,并减少为LLM训练收集复杂数据的人力。
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F26c6479c-091a-4ff0-bb69-79965edc87f6%2FUntitled.png?table=block&id=e226a98b-2c7a-4c25-8d02-896cc2cc3d30)
- Evol-instruct 数据集在数据难度和复杂度上优于ShareGPT和Alpaca, 并且难度分布上也有更多比例的复杂和困难数据
5.Conclusion
论文提供了一种用于生成更为多样和复杂的指令进化算法Evol-Instruct, 并且利用这种方法生成数据训练的WizardLM模型在高复杂性任务上取得了SOTA的结果,在其他指标上取得了具有竞争力的结果。
6. Extension:
6.1 WizardMath:
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
结合强化学习和Evol-Instruct的数学LLM
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fffd9fdcd-1e1d-476c-98a6-911a10f2835a%2FUntitled.png?table=block&id=422ef605-36c1-465b-83d4-5e194bbf6d99)
what is RLEIF(Reinforcement Learning from Evol-Instruct Feedback)?
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F0bab5ab2-4f4d-450d-800b-48f0179e7caf%2FUntitled.png?table=block&id=a32451a8-5c2c-4ef5-82de-59a32fedf368)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F9c296f42-165a-4824-886e-54360fb147aa%2FUntitled.png?table=block&id=3352b933-d115-4baf-9dd6-2ffcb9388b9b)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fc8f10db9-069b-4799-af3a-4f246356d290%2FUntitled.png?table=block&id=8625343c-ed0a-4011-92c0-2f9f7283c727)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fb501cba1-e554-4e53-acbc-c1dcd55ad49b%2FUntitled.png?table=block&id=bc1267bd-7b9b-4423-bee7-4312eb141e6c)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F5ada9694-6f32-4d00-9810-9a6e74d3c0be%2FUntitled.png?table=block&id=ce3d6a8b-bb9c-44d6-811f-0c2d7f8bb5d3)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F0f9a3827-7c33-4ce2-9cb5-f3381b21b045%2FUntitled.png?table=block&id=e31a43db-d754-44f3-86d9-417a6df7af8d)
6.2 WizardCoder
WizardCoder: Empowering Code Large Language
Models with Evol-Instruct
将Evol Instruct的能力延申到代码生成领域
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fd66adf6a-1c78-48a1-8b2e-d7622b897ba6%2FUntitled.png?table=block&id=486085ae-9bf3-4fb8-bcf1-5916ad0253ac)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fc258ca2f-998f-40b4-b185-8051a7041564%2FUntitled.png?table=block&id=ee833ed8-cbbd-4697-b504-9801f3940aac)
- 作者:GTX
- 链接:https://blog.gongtx.org/article/94579547-fba1-4a17-a1c3-3d9fe559708e
- 声明:本文采用 CC BY-NC-SA 4.0 许可协议,转载请注明出处。
相关文章
2023-11-03
Woodpecker: Hallucination Correction for Multimodal Large Language Models
2023-10-25
Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies
2023-10-20
RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
2023-09-06
Review of Agentbench
2023-09-01
Review of Learning to Retrieve In-Context Examples for Large Language Models
2023-08-24
LLM for Science — An Attempt to Solve Partial Differential Equations