mirror of https://github.com/InternLM/InternLM
add README_npu
liutongtong27
parent
1759c4b9b4
commit
ad035eb8bd
|
|
@ -0,0 +1,298 @@
|
|||
# InternLM-NPU
|
||||
|
||||
<div align="center">
|
||||
|
||||
<img src="./assets/logo.svg" width="200"/>
|
||||
<div> </div>
|
||||
<div align="center">
|
||||
<b><font size="5">InternLM</font></b>
|
||||
<sup>
|
||||
<a href="https://internlm.intern-ai.org.cn/">
|
||||
<i><font size="4">HOT</font></i>
|
||||
</a>
|
||||
</sup>
|
||||
<div> </div>
|
||||
</div>
|
||||
|
||||
[](./LICENSE)
|
||||
[](https://github.com/internLM/OpenCompass/)
|
||||
|
||||
<!-- [](https://internlm.readthedocs.io/zh_CN/latest/?badge=latest) -->
|
||||
|
||||
[📘Commercial Application](#license) |
|
||||
[🤗HuggingFace](https://huggingface.co/internlm) |
|
||||
[🆕Update News](#news) |
|
||||
[🤔Reporting Issues](https://github.com/InternLM/InternLM/issues/new) |
|
||||
[📜Technical Report](https://arxiv.org/abs/2403.17297)<br>
|
||||
[💬Chat Web](https://internlm-chat.intern-ai.org.cn/) |
|
||||
[🔗API](https://internlm.intern-ai.org.cn/api/document) |
|
||||
[🧩Modelers](https://modelers.cn/spaces/MindSpore-Lab/INTERNLM2-20B-PLAN)
|
||||
|
||||
[English](./README_npu.md) |
|
||||
[简体中文](./README_npu_zh-CN.md)
|
||||
|
||||
</div>
|
||||
|
||||
## Introduction
|
||||
This is a guide to using Ascend NPU to train and infer the InternLM series models.
|
||||
|
||||
## News
|
||||
\[2025.01.15\] InternLM3-8B-Instruct can be used in Xtuner, LLaMa-Factory and transformers.
|
||||
|
||||
## Model Zoo
|
||||
|
||||
### InternLM3
|
||||
| Model | Transformers(HF) | ModelScope(HF) | Release Date |
|
||||
|---------------------------| ------------------------------------------ | ---------------------------------------- |--------------|
|
||||
| **InternLM3-8B-Instruct** | [🤗internlm3-8b-instruct](https://huggingface.co/internlm/internlm3-8b-instruct) | [<img src="./assets/modelscope_logo.png" width="20px" /> internlm3-8b-instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct) | 2025-01-15 |
|
||||
|
||||
## Environment Setup
|
||||
|
||||
### Installing Ascend CANN Toolkit and Kernels
|
||||
|
||||
For details about the installation method, see [Installation Scheme](https://gitee.com/link?target=https%3A%2F%2Fwww.hiascend.com%2Fdocument%2Fdetail%2Fzh%2FCANNCommunityEdition%2F80RC2alpha002%2Fquickstart%2Fquickstart%2Fquickstart_18_0004.html) or run the following commands:
|
||||
|
||||
```shell
|
||||
# Replace the URL with the URL corresponding to the CANN version and device model.
|
||||
# Install CANN Toolkit.
|
||||
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Milan-ASL/Milan-ASL%20V100R001C17SPC701/Ascend-cann-toolkit_8.0.RC1.alpha001_linux-"$(uname -i)".run
|
||||
bash Ascend-cann-toolkit_8.0.RC1.alpha001_linux-"$(uname -i)".run --install
|
||||
|
||||
# Install CANN Kernels.
|
||||
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Milan-ASL/Milan-ASL%20V100R001C17SPC701/Ascend-cann-kernels-910b_8.0.RC1.alpha001_linux.run
|
||||
bash Ascend-cann-kernels-910b_8.0.RC1.alpha001_linux.run --install
|
||||
|
||||
# Set environment variables.
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
```
|
||||
|
||||
## Xtuner
|
||||
|
||||
### Installing Xtuner
|
||||
|
||||
```shell
|
||||
git clone https://github.com/InternLM/xtuner.git
|
||||
cd xtuner
|
||||
```
|
||||
|
||||
Modify `requirements/runtime.txt` with the following changes:
|
||||
|
||||
```text
|
||||
bitsandbytes==0.42.0
|
||||
mmengine==0.10.5
|
||||
torchvision==0.19.0
|
||||
numpy==1.26.4
|
||||
```
|
||||
|
||||
Use the following command for installation:
|
||||
|
||||
```shell
|
||||
pip install -e '.[all]'
|
||||
```
|
||||
|
||||
**Note**:
|
||||
|
||||
- The default installation version of `torch` is the latest version. Please pay attention to match it with the version of `torch_npu`.
|
||||
|
||||
### LoRA Fine-tuning
|
||||
|
||||
Use the following commands to copy and rename the file to `internlm3_8b_instruct_lora_oasst1_e10.py`:
|
||||
|
||||
```shell
|
||||
xtuner copy-cfg internlm2_5_chat_7b_qlora_oasst1_e3 .
|
||||
mv internlm2_5_chat_7b_qlora_oasst1_e3_copy.py internlm3_8b_instruct_lora_oasst1_e10.py
|
||||
```
|
||||
|
||||
The modifications to the configuration file `internlm3_8b_instruct_lora_oasst1_e10.py` are as follows:
|
||||
|
||||
```python
|
||||
pretrained_model_name_or_path = 'internlm/internlm3-8b-instruct'
|
||||
|
||||
max_epochs = 10
|
||||
|
||||
model = dict(
|
||||
type=SupervisedFinetune,
|
||||
use_varlen_attn=use_varlen_attn,
|
||||
llm=dict(
|
||||
type=AutoModelForCausalLM.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
torch_dtype=torch.float16),
|
||||
# quantization_config=dict(
|
||||
# type=BitsAndBytesConfig,
|
||||
# load_in_4bit=True,
|
||||
# load_in_8bit=False,
|
||||
# llm_int8_threshold=6.0,
|
||||
# llm_int8_has_fp16_weight=False,
|
||||
# bnb_4bit_compute_dtype=torch.float16,
|
||||
# bnb_4bit_use_double_quant=True,
|
||||
# bnb_4bit_quant_type='nf4')),
|
||||
lora=dict(
|
||||
type=LoraConfig,
|
||||
r=64,
|
||||
lora_alpha=16,
|
||||
lora_dropout=0.1,
|
||||
bias='none',
|
||||
task_type='CAUSAL_LM'))
|
||||
|
||||
custom_hooks = [
|
||||
dict(type=DatasetInfoHook, tokenizer=tokenizer),
|
||||
# dict(
|
||||
# type=EvaluateChatHook,
|
||||
# tokenizer=tokenizer,
|
||||
# every_n_iters=evaluation_freq,
|
||||
# evaluation_inputs=evaluation_inputs,
|
||||
# system=SYSTEM,
|
||||
# prompt_template=prompt_template)
|
||||
]
|
||||
|
||||
randomness = dict(seed=123, deterministic=True)
|
||||
```
|
||||
|
||||
Run the following commands to start single-machine eight-card fine-tuning:
|
||||
|
||||
```shell
|
||||
NPROC_PER_NODE=8 xtuner train internlm3_8b_instruct_lora_oasst1_e10.py --deepspeed deepspeed_zero2
|
||||
```
|
||||
|
||||
The fine-tuning results are saved in the directory `./work_dirs/internlm3_8b_instruct_lora_oasst1_e10/iter_xxx.pth`.
|
||||
|
||||
### Model Convert
|
||||
|
||||
Convert the model weight file obtained from fine-tuning into the Hugging Face format, which facilitates subsequent deployment and usage.
|
||||
Use the following command for the conversion:
|
||||
|
||||
```shell
|
||||
xtuner convert pth_to_hf internlm3_8b_instruct_lora_oasst1_e10.py ./work_dirs/internlm3_8b_instruct_lora_oasst1_e10/iter_xxx.pth ./work_dirs/convert_output
|
||||
```
|
||||
|
||||
### Model Merge
|
||||
|
||||
LoRA or QLoRA fine-tuning generates an additional `Adapter` layer, which needs to be merged with the original model to
|
||||
create a complete model. Use the following command for model merging, where `$model_path` is the local path where the
|
||||
original model is stored, and `--max-shard-size` 2GB limits the maximum size of each weight file to 2GB:
|
||||
|
||||
```shell
|
||||
xtuner convert merge $model_path ./work_dirs/convert_output ./work_dirs/merge_output --max-shard-size 2GB
|
||||
```
|
||||
|
||||
### Chat
|
||||
|
||||
Chat with the merged model weights:
|
||||
|
||||
|
||||
```shell
|
||||
cp path_to_your_model/modeling_internlm3.py ./work_dirs/merge_output
|
||||
xtuner chat ./work_dirs/merge_output --prompt-template internlm2_chat
|
||||
```
|
||||
|
||||
## LLama-Factory
|
||||
|
||||
### Installing LLaMa-Factory
|
||||
|
||||
```shell
|
||||
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
|
||||
cd LLaMA-Factory
|
||||
pip install -e ".[torch-npu,metrics]"
|
||||
```
|
||||
|
||||
### Inference
|
||||
|
||||
Create the `examples/inference/internlm2_5_7b_chat.yaml` inference configuration file in the LLaMa-Factory directory:
|
||||
|
||||
```yaml
|
||||
model_name_or_path: xxx # Support only local loading. Set this parameter to the local weight path of InternLM2.5-7B-Chat.
|
||||
template: intern2
|
||||
```
|
||||
|
||||
Run the following command to interact with the model:
|
||||
|
||||
```shell
|
||||
llamafactory-cli chat examples/inference/internlm2_5_7b_chat.yaml
|
||||
```
|
||||
|
||||
### Fine-tuning
|
||||
|
||||
Create the `examples/train_lora/internlm2_5_7b_chat_lora_sft.yaml` configuration file in the LLaMa-Factory directory. The fine-tuning configuration file is as follows:
|
||||
|
||||
```yaml
|
||||
### model
|
||||
model_name_or_path: xxx # Support only local loading. Set this parameter to the local weight path of InternLM2.5-7B-Chat.
|
||||
|
||||
### method
|
||||
stage: sft
|
||||
do_train: true
|
||||
finetuning_type: lora
|
||||
lora_target: all
|
||||
|
||||
### dataset
|
||||
dataset: identity
|
||||
template: intern2
|
||||
cutoff_len: 128
|
||||
preprocessing_num_workers: 16
|
||||
|
||||
### output
|
||||
output_dir: saves/internlm2_5_7b_chat/lora/sft
|
||||
logging_steps: 5
|
||||
save_steps: 20
|
||||
plot_loss: true
|
||||
overwrite_output_dir: true
|
||||
|
||||
### train
|
||||
per_device_train_batch_size: 8
|
||||
gradient_accumulation_steps: 1
|
||||
learning_rate: 1.0e-4
|
||||
num_train_epochs: 5.0
|
||||
lr_scheduler_type: cosine
|
||||
warmup_ratio: 0.1
|
||||
bf16: true
|
||||
ddp_timeout: 180000000
|
||||
```
|
||||
|
||||
Run the following commands to start fine-tuning:
|
||||
|
||||
```shell
|
||||
export ASCEND_RT_VISIBLE_DEVICES=0
|
||||
llamafactory-cli train examples/train_lora/internlm2_5_7b_chat_lora_sft.yaml
|
||||
```
|
||||
|
||||
### Accuracy
|
||||
|
||||
The loss curve obtained after finetuning is as follows:
|
||||
|
||||
|
||||
|
||||
### Performance
|
||||
|
||||
| Chip Type | train_samples_per_second |
|
||||
|-------------------|--------------------------|
|
||||
| Atlas 900 A2 PODc | 49.662 |
|
||||
|
||||
## Transformers
|
||||
|
||||
### Inference
|
||||
|
||||
Create the inference script `inference_internlm2_5_7b_chat.py`:
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
# 若模型已下载，可替换成模型本地路径
|
||||
tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2_5-7b-chat", trust_remote_code=True)
|
||||
# `torch_dtype=torch.float16`可以令模型以float16精度加载，否则transformers会将模型加载为float32，导致显存不足
|
||||
model = AutoModelForCausalLM.from_pretrained("internlm/internlm2_5-7b-chat", torch_dtype=torch.float16, trust_remote_code=True).npu()
|
||||
model = model.eval()
|
||||
response, history = model.chat(tokenizer, "你好，请提供三个管理时间的建议。", history=[])
|
||||
print(response)
|
||||
```
|
||||
|
||||
Execute the inference script:
|
||||
|
||||
```shell
|
||||
python inference_internlm2_5_7b_chat.py
|
||||
```
|
||||
|
||||
|
||||
## License
|
||||
The code is licensed under Apache-2.0, while model weights are fully open for academic research and also allow **free** commercial usage. To apply for a commercial license, please fill in the [application form (English)](https://wj.qq.com/s2/12727483/5dba/)/[申请表（中文）](https://wj.qq.com/s2/12725412/f7c1/). For other questions or collaborations, please contact <internlm@pjlab.org.cn>.
|
||||
|
|
@ -0,0 +1,295 @@
|
|||
# InternLM-NPU
|
||||
|
||||
<div align="center">
|
||||
|
||||
<img src="./assets//logo.svg" width="200"/>
|
||||
<div> </div>
|
||||
<div align="center">
|
||||
<b><font size="5">书生·浦语 官网</font></b>
|
||||
<sup>
|
||||
<a href="https://internlm.intern-ai.org.cn/">
|
||||
<i><font size="4">HOT</font></i>
|
||||
</a>
|
||||
</sup>
|
||||
<div> </div>
|
||||
</div>
|
||||
|
||||
[](https://github.com/open-mmlab/mmdetection/blob/main/LICENSE)
|
||||
[](https://github.com/internLM/OpenCompass/)
|
||||
|
||||
<!-- [](https://internlm.readthedocs.io/zh_CN/latest/?badge=latest) -->
|
||||
|
||||
[📘商业授权](#开源许可证) |
|
||||
[🤗HuggingFace](https://huggingface.co/internlm) |
|
||||
[🆕最新消息](#更新) |
|
||||
[🤔提交反馈](https://github.com/InternLM/InternLM/issues/new)|
|
||||
[📜技术报告](https://arxiv.org/abs/2403.17297)<br>
|
||||
[💬聊天应用](https://internlm-chat.intern-ai.org.cn/) |
|
||||
[🔗API](https://internlm.intern-ai.org.cn/api/document) |
|
||||
[🧩魔乐社区](https://modelers.cn/spaces/MindSpore-Lab/INTERNLM2-20B-PLAN)
|
||||
|
||||
[English](./README_npu.md) |
|
||||
[简体中文](./README_npu_zh-CN.md)
|
||||
|
||||
</div>
|
||||
|
||||
## 介绍
|
||||
这是一份使用 Ascend NPU 对 InternLM 系列模型进行训练和推理的指南。
|
||||
|
||||
## News
|
||||
\[2025.01.15\] InternLM3-8B-Instruct 可用于 Xtuner、LLaMa-Factory 和 transformers 中。
|
||||
|
||||
## Model Zoo
|
||||
|
||||
### InternLM3
|
||||
| Model | Transformers(HF) | ModelScope(HF) | Release Date |
|
||||
|---------------------------| ------------------------------------------ | ---------------------------------------- |--------------|
|
||||
| **InternLM3-8B-Instruct** | [🤗internlm3-8b-instruct](https://huggingface.co/internlm/internlm3-8b-instruct) | [<img src="./assets/modelscope_logo.png" width="20px" /> internlm3-8b-instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct) | 2025-01-15 |
|
||||
|
||||
## 环境准备l
|
||||
|
||||
### 安装Ascend CANN Toolkit和Kernels
|
||||
|
||||
安装方法请参考[安装教程](https://gitee.com/link?target=https%3A%2F%2Fwww.hiascend.com%2Fdocument%2Fdetail%2Fzh%2FCANNCommunityEdition%2F80RC2alpha002%2Fquickstart%2Fquickstart%2Fquickstart_18_0004.html)或使用以下命令
|
||||
|
||||
```shell
|
||||
# 请替换URL为CANN版本和设备型号对应的URL
|
||||
# 安装CANN Toolkit
|
||||
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Milan-ASL/Milan-ASL%20V100R001C17SPC701/Ascend-cann-toolkit_8.0.RC1.alpha001_linux-"$(uname -i)".run
|
||||
bash Ascend-cann-toolkit_8.0.RC1.alpha001_linux-"$(uname -i)".run --install
|
||||
|
||||
# 安装CANN Kernels
|
||||
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Milan-ASL/Milan-ASL%20V100R001C17SPC701/Ascend-cann-kernels-910b_8.0.RC1.alpha001_linux.run
|
||||
bash Ascend-cann-kernels-910b_8.0.RC1.alpha001_linux.run --install
|
||||
|
||||
# 设置环境变量
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
```
|
||||
|
||||
## Xtuner
|
||||
|
||||
### 安装 Xtuner
|
||||
|
||||
```shell
|
||||
git clone https://github.com/InternLM/xtuner.git
|
||||
cd xtuner
|
||||
```
|
||||
|
||||
修改`requirements/runtime.txt`，修改点如下：
|
||||
|
||||
```text
|
||||
bitsandbytes==0.42.0
|
||||
mmengine==0.10.5
|
||||
torchvision==0.19.0
|
||||
numpy==1.26.4
|
||||
```
|
||||
|
||||
使用以下命令进行安装：
|
||||
|
||||
```shell
|
||||
pip install -e '.[all]'
|
||||
```
|
||||
|
||||
**注意**:
|
||||
|
||||
- 默认安装`torch`为最新版，请注意与`torch_npu`版本相匹配
|
||||
|
||||
### LoRA 微调
|
||||
|
||||
使用以下命令复制并重命名文件为`internlm3_8b_instruct_lora_oasst1_e10.py`，
|
||||
|
||||
```shell
|
||||
xtuner copy-cfg internlm2_5_chat_7b_qlora_oasst1_e3 .
|
||||
mv internlm2_5_chat_7b_qlora_oasst1_e3_copy.py internlm3_8b_instruct_lora_oasst1_e10.py
|
||||
```
|
||||
|
||||
`internlm3_8b_instruct_lora_oasst1_e10.py`配置文件的修改点如下：
|
||||
|
||||
```python
|
||||
pretrained_model_name_or_path = 'internlm/internlm3-8b-instruct'
|
||||
|
||||
max_epochs = 10
|
||||
|
||||
model = dict(
|
||||
type=SupervisedFinetune,
|
||||
use_varlen_attn=use_varlen_attn,
|
||||
llm=dict(
|
||||
type=AutoModelForCausalLM.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
torch_dtype=torch.float16),
|
||||
# quantization_config=dict(
|
||||
# type=BitsAndBytesConfig,
|
||||
# load_in_4bit=True,
|
||||
# load_in_8bit=False,
|
||||
# llm_int8_threshold=6.0,
|
||||
# llm_int8_has_fp16_weight=False,
|
||||
# bnb_4bit_compute_dtype=torch.float16,
|
||||
# bnb_4bit_use_double_quant=True,
|
||||
# bnb_4bit_quant_type='nf4')),
|
||||
lora=dict(
|
||||
type=LoraConfig,
|
||||
r=64,
|
||||
lora_alpha=16,
|
||||
lora_dropout=0.1,
|
||||
bias='none',
|
||||
task_type='CAUSAL_LM'))
|
||||
|
||||
custom_hooks = [
|
||||
dict(type=DatasetInfoHook, tokenizer=tokenizer),
|
||||
# dict(
|
||||
# type=EvaluateChatHook,
|
||||
# tokenizer=tokenizer,
|
||||
# every_n_iters=evaluation_freq,
|
||||
# evaluation_inputs=evaluation_inputs,
|
||||
# system=SYSTEM,
|
||||
# prompt_template=prompt_template)
|
||||
]
|
||||
|
||||
randomness = dict(seed=123, deterministic=True)
|
||||
```
|
||||
|
||||
通过下列命令启动单机8卡微调：
|
||||
|
||||
```shell
|
||||
NPROC_PER_NODE=8 xtuner train internlm3_8b_instruct_lora_oasst1_e10.py --deepspeed deepspeed_zero2
|
||||
```
|
||||
|
||||
微调后结果保存在`./work_dirs/internlm3_8b_instruct_lora_oasst1_e10/iter_xxx.pth`下。
|
||||
|
||||
### 模型转换
|
||||
|
||||
将训练得到的模型权重文件转换为 Hugging Face 格式的模型文件，便于后续的部署和使用。使用以下命令进行转换：
|
||||
|
||||
```shell
|
||||
xtuner convert pth_to_hf internlm3_8b_instruct_lora_oasst1_e10.py ./work_dirs/internlm3_8b_instruct_lora_oasst1_e10/iter_xxx.pth ./work_dirs/convert_output
|
||||
```
|
||||
|
||||
### 模型合并
|
||||
|
||||
LoRA或QLoRA微调生成的是一个额外的 `Adapter` 层，需要与原模型合并才能生成一个完整的模型。使用以下命令进行模型合并，其中`$model_path`
|
||||
为原模型存储的本地路径, `--max-shard-size 2GB` 限制每个权重文件最大为2GB：
|
||||
|
||||
```shell
|
||||
xtuner convert merge $model_path ./work_dirs/convert_output ./work_dirs/merge_output --max-shard-size 2GB
|
||||
```
|
||||
|
||||
### 对话
|
||||
|
||||
使用合并后的模型权重进行对话：
|
||||
|
||||
```shell
|
||||
cp path_to_your_model/modeling_internlm3.py ./work_dirs/merge_output
|
||||
xtuner chat ./work_dirs/merge_output --prompt-template internlm2_chat
|
||||
```
|
||||
|
||||
## LLama-Factory
|
||||
|
||||
### 安装 LLaMa-Factory
|
||||
|
||||
```shell
|
||||
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
|
||||
cd LLaMA-Factory
|
||||
pip install -e ".[torch-npu,metrics]"
|
||||
```
|
||||
|
||||
### 推理
|
||||
|
||||
在 LLaMa-Factory 路径下新建`examples/inference/internlm2_5_7b_chat.yaml`推理配置文件，文件内容为：
|
||||
|
||||
```yaml
|
||||
model_name_or_path: xxx # Support only local loading. Set this parameter to the local weight path of InternLM2.5-7B-Chat.
|
||||
template: intern2
|
||||
```
|
||||
|
||||
使用以下命令与模型进行交互：
|
||||
|
||||
```shell
|
||||
llamafactory-cli chat examples/inference/internlm2_5_7b_chat.yaml
|
||||
```
|
||||
|
||||
### 微调
|
||||
|
||||
在 LLaMa-Factory 路径下新建`examples/train_lora/internlm2_5_7b_chat_lora_sft.yaml`微调配置文件，微调配置文件如下：
|
||||
|
||||
```yaml
|
||||
### model
|
||||
model_name_or_path: xxx # Support only local loading. Set this parameter to the local weight path of InternLM2.5-7B-Chat.
|
||||
|
||||
### method
|
||||
stage: sft
|
||||
do_train: true
|
||||
finetuning_type: lora
|
||||
lora_target: all
|
||||
|
||||
### dataset
|
||||
dataset: identity
|
||||
template: intern2
|
||||
cutoff_len: 128
|
||||
preprocessing_num_workers: 16
|
||||
|
||||
### output
|
||||
output_dir: saves/internlm2_5_7b_chat/lora/sft
|
||||
logging_steps: 5
|
||||
save_steps: 20
|
||||
plot_loss: true
|
||||
overwrite_output_dir: true
|
||||
|
||||
### train
|
||||
per_device_train_batch_size: 8
|
||||
gradient_accumulation_steps: 1
|
||||
learning_rate: 1.0e-4
|
||||
num_train_epochs: 5.0
|
||||
lr_scheduler_type: cosine
|
||||
warmup_ratio: 0.1
|
||||
bf16: true
|
||||
ddp_timeout: 180000000
|
||||
```
|
||||
|
||||
通过下面的命令启动微调：
|
||||
|
||||
```shell
|
||||
export ASCEND_RT_VISIBLE_DEVICES=0
|
||||
llamafactory-cli train examples/train_lora/internlm2_5_7b_chat_lora_sft.yaml
|
||||
```
|
||||
|
||||
### 精度
|
||||
|
||||
微调后得到的loss曲线如下：
|
||||
|
||||
|
||||
|
||||
### 性能
|
||||
|
||||
| 芯片型号 | train_samples_per_second |
|
||||
|-------------------|--------------------------|
|
||||
| Atlas 900 A2 PODc | 49.662 |
|
||||
|
||||
## Transformers
|
||||
|
||||
### 推理
|
||||
|
||||
新建推理脚本`inference_internlm2_5_7b_chat.py`，推理脚本内容为：
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
# 若模型已下载，可替换成模型本地路径
|
||||
tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2_5-7b-chat", trust_remote_code=True)
|
||||
# `torch_dtype=torch.float16`可以令模型以float16精度加载，否则transformers会将模型加载为float32，导致显存不足
|
||||
model = AutoModelForCausalLM.from_pretrained("internlm/internlm2_5-7b-chat", torch_dtype=torch.float16, trust_remote_code=True).npu()
|
||||
model = model.eval()
|
||||
response, history = model.chat(tokenizer, "你好，请提供三个管理时间的建议。", history=[])
|
||||
print(response)
|
||||
```
|
||||
|
||||
执行推理脚本：
|
||||
|
||||
```shell
|
||||
python inference_internlm2_5_7b_chat.py
|
||||
```
|
||||
|
||||
## 开源许可证
|
||||
|
||||
本仓库的代码依照 Apache-2.0 协议开源。模型权重对学术研究完全开放，也可申请免费的商业使用授权（[申请表](https://wj.qq.com/s2/12725412/f7c1/)）。其他问题与合作请联系 <internlm@pjlab.org.cn>。
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
Loading…
Reference in New Issue