mirror of https://github.com/InternLM/InternLM
[Doc] Update Evaluation (#588)
* [Doc] Update Evaluation * Update performance * Update performance * Update README.md * Update README_zh-CN.md --------- Co-authored-by: Kai Chen <chenkaidev@gmail.com>pull/598/head
Songyang Zhang
GitHub
parent
13cd9d9b21
commit
3ebe24d92c
70
README.md
70
README.md
|
|
@ -19,7 +19,7 @@
|
|||
<!-- [](https://internlm.readthedocs.io/zh_CN/latest/?badge=latest) -->
|
||||
[📘Chat](./chat) |
|
||||
[🛠️Agent](./agent) |
|
||||
[📊Evaluation](./evaluation) |
|
||||
[📊Evaluation](#evaluation) |
|
||||
[👀Model](./model_cards) |
|
||||
[🤗HuggingFace](https://huggingface.co/spaces/internlm/internlm2-Chat-7B) |
|
||||
[🆕Update News](#news) |
|
||||
|
|
@ -77,6 +77,48 @@ The release of InternLM2 series contains two model sizes: 7B and 20B. 7B models
|
|||
|
||||
**Limitations:** Although we have made efforts to ensure the safety of the model during the training process and to encourage the model to generate text that complies with ethical and legal requirements, the model may still produce unexpected outputs due to its size and probabilistic generation paradigm. For example, the generated responses may contain biases, discrimination, or other harmful content. Please do not propagate such content. We are not responsible for any consequences resulting from the dissemination of harmful information.
|
||||
|
||||
## Performance
|
||||
|
||||
### Objective Evaluation
|
||||
| Dataset | Baichuan2-7B-Chat | Mistral-7B-Instruct-v0.2 | Qwen-7B-Chat | InternLM2-Chat-7B | ChatGLM3-6B | Baichuan2-13B-Chat | Mixtral-8x7B-Instruct-v0.1 | Qwen-14B-Chat | InternLM2-Chat-20B |
|
||||
|-----------------------|-------------------|--------------------------|--------------|-------------------|-------------|---------------------|--------------------------------|---------------|---------------------|
|
||||
| MMLU | 50.1 | 59.2 | 57.1 | 63.7 | 58.0 | 56.6 | 70.3 | 66.7 | 65.1 |
|
||||
| CMMLU | 53.4 | 42.0 | 57.9 | 63.0 | 57.8 | 54.8 | 50.6 | 68.1 | 65.1 |
|
||||
| AGIEval | 35.3 | 34.5 | 39.7 | 47.2 | 44.2 | 40.0 | 41.7 | 46.5 | 50.3 |
|
||||
| C-Eval | 53.9 | 42.4 | 59.8 | 60.8 | 59.1 | 56.3 | 54.0 | 71.5 | 63.0 |
|
||||
| TrivialQA | 37.6 | 35.0 | 46.1 | 50.8 | 38.1 | 40.3 | 57.7 | 54.5 | 53.9 |
|
||||
| NaturalQuestions | 12.8 | 8.1 | 18.6 | 24.1 | 14.0 | 12.7 | 22.5 | 22.9 | 25.9 |
|
||||
| C3 | 78.5 | 66.9 | 84.4 | 91.5 | 79.3 | 84.4 | 82.1 | 91.5 | 93.5 |
|
||||
| CMRC | 8.1 | 5.6 | 14.6 | 63.8 | 43.2 | 27.8 | 5.3 | 13.0 | 50.4 |
|
||||
| WinoGrande | 49.9 | 50.8 | 54.2 | 65.8 | 61.7 | 50.9 | 60.9 | 55.7 | 74.8 |
|
||||
| BBH | 35.9 | 46.5 | 45.5 | 61.2 | 56.0 | 42.5 | 57.3 | 55.8 | 68.3 |
|
||||
| GSM-8K | 32.4 | 48.3 | 44.1 | 70.7 | 53.8 | 56.0 | 71.7 | 57.7 | 79.6 |
|
||||
| Math | 5.7 | 8.6 | 12.0 | 23.0 | 20.4 | 4.3 | 22.5 | 27.6 | 31.9 |
|
||||
| HumanEval | 17.7 | 35.4 | 36.0 | 59.8 | 52.4 | 19.5 | 37.8 | 40.9 | 67.1 |
|
||||
| MBPP | 37.7 | 25.7 | 33.9 | 51.4 | 55.6 | 40.9 | 40.9 | 30.0 | 65.8 |
|
||||
|
||||
- Performance of MBPP is reported with MBPP(Sanitized)
|
||||
|
||||
### Alignment Evaluation
|
||||
|
||||
- We have evaluated our model on [AlpacaEval 2.0](https://tatsu-lab.github.io/alpaca_eval/) and InternLM2-Chat-20B surpass Claude 2, GPT-4(0613) and Gemini Pro.
|
||||
|
||||
| Model Name | Win Rate | Length |
|
||||
| ----------------------- | -------- | ------ |
|
||||
| GPT-4 Turbo | 50.00% | 2049 |
|
||||
| GPT-4 | 23.58% | 1365 |
|
||||
| GPT-4 0314 | 22.07% | 1371 |
|
||||
| Mistral Medium | 21.86% | 1500 |
|
||||
| XwinLM 70b V0.1 | 21.81% | 1775 |
|
||||
| InternLM2 Chat 20B | 21.75% | 2373 |
|
||||
| Mixtral 8x7B v0.1 | 18.26% | 1465 |
|
||||
| Claude 2 | 17.19% | 1069 |
|
||||
| Gemini Pro | 16.85% | 1315 |
|
||||
| GPT-4 0613 | 15.76% | 1140 |
|
||||
| Claude 2.1 | 15.73% | 1096 |
|
||||
|
||||
* According to the released performance of 2024-01-17.
|
||||
|
||||
## Usages
|
||||
|
||||
We briefly show the usages with [Transformers](#import-from-transformers), [ModelScope](#import-from-modelscope), and [Web demos](#dialogue).
|
||||
|
|
@ -154,6 +196,32 @@ Please refer to [finetune docs](./finetune/) for fine-tuning with InternLM.
|
|||
|
||||
**Note:** We have migrated the whole training functionality in this project to [InternEvo](https://github.com/InternLM/InternEvo) for easier user experience, which provides efficient pre-training and fine-tuning infra for training InternLM.
|
||||
|
||||
|
||||
## Evaluation
|
||||
|
||||
We utilize [OpenCompass](https://github.com/open-compass/opencompass) for model evaluation. In InternLM-2, we primarily focus on standard objective evaluation, long-context evaluation (needle in a haystack), data contamination assessment, agent evaluation, and subjective evaluation.
|
||||
|
||||
### Objective Evaluation
|
||||
|
||||
To evaluate the InternLM model, please follow the guidelines in the [OpenCompass tutorial](https://opencompass.readthedocs.io/en/latest/get_started/installation.html). Typically, we use `ppl` for multiple-choice questions on the **Base** model and `gen` for all questions on the **Chat** model.
|
||||
|
||||
### Long-Context Evaluation (Needle in a Haystack)
|
||||
|
||||
For the `Needle in a Haystack` evaluation, refer to the tutorial provided in the [documentation](https://github.com/open-compass/opencompass/blob/main/docs/en/advanced_guides/needleinahaystack_eval.md). Feel free to try it out.
|
||||
|
||||
### Data Contamination Assessment
|
||||
|
||||
To learn more about data contamination assessment, please check the [contamination eval](https://opencompass.readthedocs.io/en/latest/advanced_guides/contamination_eval.html).
|
||||
|
||||
### Agent Evaluation
|
||||
|
||||
- To evaluate tool utilization, please refer to [T-Eval](https://github.com/open-compass/T-Eval).
|
||||
- For code interpreter evaluation, use the [gsm-8k-agent](https://github.com/open-compass/opencompass/blob/main/configs/datasets/gsm8k/gsm8k_agent_gen_be1606.py) provided in the repository. Additionally, you need to install [Lagent](https://github.com/InternLM/lagent).
|
||||
|
||||
### Subjective Evaluation
|
||||
|
||||
- Please follow the [tutorial](https://opencompass.readthedocs.io/en/latest/advanced_guides/subjective_evaluation.html) for subjective evaluation.
|
||||
|
||||
## Contribution
|
||||
|
||||
We appreciate all the contributors for their efforts to improve and enhance InternLM. Community users are highly encouraged to participate in the project. Please refer to the contribution guidelines for instructions on how to contribute to the project.
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@
|
|||
|
||||
[📘对话教程](./chat) |
|
||||
[🛠️智能体教程](./agent) |
|
||||
[📊评测](./evaluation) |
|
||||
[📊评测](#评测) |
|
||||
[👀模型库](./model_cards) |
|
||||
[🤗HuggingFace](https://huggingface.co/spaces/internlm/internlm2-Chat-7B) |
|
||||
[🆕Update News](#news) |
|
||||
|
|
@ -75,6 +75,48 @@ InternLM2 系列模型在本仓库正式发布，具有如下特性：
|
|||
|
||||
**局限性：** 尽管在训练过程中我们非常注重模型的安全性，尽力促使模型输出符合伦理和法律要求的文本，但受限于模型大小以及概率生成范式，模型可能会产生各种不符合预期的输出，例如回复内容包含偏见、歧视等有害内容，请勿传播这些内容。由于传播不良信息导致的任何后果，本项目不承担责任。
|
||||
|
||||
## 性能
|
||||
|
||||
### 客观评测
|
||||
| Dataset | Baichuan2-7B-Chat | Mistral-7B-Instruct-v0.2 | Qwen-7B-Chat | InternLM2-Chat-7B | ChatGLM3-6B | Baichuan2-13B-Chat | Mixtral-8x7B-Instruct-v0.1 | Qwen-14B-Chat | InternLM2-Chat-20B |
|
||||
|-----------------------|-------------------|--------------------------|--------------|-------------------|-------------|---------------------|--------------------------------|---------------|---------------------|
|
||||
| MMLU | 50.1 | 59.2 | 57.1 | 63.7 | 58.0 | 56.6 | 70.3 | 66.7 | 65.1 |
|
||||
| CMMLU | 53.4 | 42.0 | 57.9 | 63.0 | 57.8 | 54.8 | 50.6 | 68.1 | 65.1 |
|
||||
| AGIEval | 35.3 | 34.5 | 39.7 | 47.2 | 44.2 | 40.0 | 41.7 | 46.5 | 50.3 |
|
||||
| C-Eval | 53.9 | 42.4 | 59.8 | 60.8 | 59.1 | 56.3 | 54.0 | 71.5 | 63.0 |
|
||||
| TrivialQA | 37.6 | 35.0 | 46.1 | 50.8 | 38.1 | 40.3 | 57.7 | 54.5 | 53.9 |
|
||||
| NaturalQuestions | 12.8 | 8.1 | 18.6 | 24.1 | 14.0 | 12.7 | 22.5 | 22.9 | 25.9 |
|
||||
| C3 | 78.5 | 66.9 | 84.4 | 91.5 | 79.3 | 84.4 | 82.1 | 91.5 | 93.5 |
|
||||
| CMRC | 8.1 | 5.6 | 14.6 | 63.8 | 43.2 | 27.8 | 5.3 | 13.0 | 50.4 |
|
||||
| WinoGrande | 49.9 | 50.8 | 54.2 | 65.8 | 61.7 | 50.9 | 60.9 | 55.7 | 74.8 |
|
||||
| BBH | 35.9 | 46.5 | 45.5 | 61.2 | 56.0 | 42.5 | 57.3 | 55.8 | 68.3 |
|
||||
| GSM-8K | 32.4 | 48.3 | 44.1 | 70.7 | 53.8 | 56.0 | 71.7 | 57.7 | 79.6 |
|
||||
| Math | 5.7 | 8.6 | 12.0 | 23.0 | 20.4 | 4.3 | 22.5 | 27.6 | 31.9 |
|
||||
| HumanEval | 17.7 | 35.4 | 36.0 | 59.8 | 52.4 | 19.5 | 37.8 | 40.9 | 67.1 |
|
||||
| MBPP | 37.7 | 25.7 | 33.9 | 51.4 | 55.6 | 40.9 | 40.9 | 30.0 | 65.8 |
|
||||
|
||||
- MBPP性能使用的是MBPP(Sanitized)版本数据集
|
||||
|
||||
### 主观评测
|
||||
|
||||
- 我们评测了InternLM2-Chat在[AlpacaEval 2.0](https://tatsu-lab.github.io/alpaca_eval/) 上的性能，结果表明InternLM2-Chat在AlpacaEval上已经超过了 Claude 2, GPT-4(0613) 和 Gemini Pro.
|
||||
|
||||
| Model Name | Win Rate | Length |
|
||||
| ----------------------- | -------- | ------ |
|
||||
| GPT-4 Turbo | 50.00% | 2049 |
|
||||
| GPT-4 | 23.58% | 1365 |
|
||||
| GPT-4 0314 | 22.07% | 1371 |
|
||||
| Mistral Medium | 21.86% | 1500 |
|
||||
| XwinLM 70b V0.1 | 21.81% | 1775 |
|
||||
| InternLM2 Chat 20B | 21.75% | 2373 |
|
||||
| Mixtral 8x7B v0.1 | 18.26% | 1465 |
|
||||
| Claude 2 | 17.19% | 1069 |
|
||||
| Gemini Pro | 16.85% | 1315 |
|
||||
| GPT-4 0613 | 15.76% | 1140 |
|
||||
| Claude 2.1 | 15.73% | 1096 |
|
||||
|
||||
* 性能数据截止2024-01-17
|
||||
|
||||
## 使用案例
|
||||
|
||||
接下来我们展示使用 [Transformers](#import-from-transformers)，[ModelScope](#import-from-modelscope) 和 [Web demo](#dialogue) 进行推理。
|
||||
|
|
@ -148,6 +190,31 @@ print(response)
|
|||
|
||||
**注意：** 本项目中的全量训练功能已经迁移到了 [InternEvo](https://github.com/InternLM/InternEvo) 以便用户使用。InternEvo 提供了高效的预训练和微调基建用于训练 InternLM 系列模型。
|
||||
|
||||
## 评测
|
||||
|
||||
我们使用 [OpenCompass](https://github.com/open-compass/opencompass) 进行模型评估。在 InternLM-2 中，我们主要标准客观评估、长文评估（大海捞针）、数据污染评估、智能体评估和主观评估。
|
||||
|
||||
### 标准客观评测
|
||||
|
||||
请按照 [OpenCompass 教程](https://opencompass.readthedocs.io/zh-cn/latest/get_started/installation.html) 进行客观评测。我们通常在 Base 模型上使用 ppl 进行多项选择题评测，在 Chat 模型上使用 gen 进行所有问题的答案生成和评测。
|
||||
|
||||
### 长文评估（大海捞针）
|
||||
|
||||
有关 `大海捞针` 评估的教程，请参阅 [文档](https://github.com/open-compass/opencompass/blob/main/docs/en/advanced_guides/needleinahaystack_eval.md) 中的教程。
|
||||
|
||||
### 数据污染评估
|
||||
|
||||
要了解更多关于数据污染评估的信息，请查看 [污染评估](https://opencompass.readthedocs.io/en/latest/advanced_guides/contamination_eval.html)。
|
||||
|
||||
### 智能体评估
|
||||
|
||||
- 要评估大模型的工具利用能力，请使用 [T-Eval](https://github.com/open-compass/T-Eval) 进行评测。
|
||||
- 对于代码解释器评估，请使用 [gsm-8k-agent](https://github.com/open-compass/opencompass/blob/main/configs/datasets/gsm8k/gsm8k_agent_gen_be1606.py) 提供的配置进行评估。此外，您还需要安装 [Lagent](https://github.com/InternLM/lagent)。
|
||||
|
||||
### 主观评估
|
||||
|
||||
- 请按照 [教程](https://opencompass.readthedocs.io/en/latest/advanced_guides/subjective_evaluation.html) 进行主观评估。
|
||||
|
||||
## 贡献
|
||||
|
||||
我们感谢所有的贡献者为改进和提升 InternLM 所作出的努力。非常欢迎社区用户能参与进项目中来。请参考贡献指南来了解参与项目贡献的相关指引。
|
||||
|
|
|
|||
Loading…
Reference in New Issue