mirror of https://github.com/THUDM/ChatGLM-6B
Update english README
duzx16
parent
5dee917a67
commit
c95b6b9d0d
85
README_en.md
85
README_en.md
|
|
@ -1,6 +1,5 @@
|
|||
# ChatGLM-6B
|
||||
|
||||
|
||||
<p align="center">
|
||||
🌐 <a href="https://chatglm.cn/blog" target="_blank">Blog</a> • 🤗 <a href="https://huggingface.co/THUDM/chatglm-6b" target="_blank">HF Repo</a> • 🐦 <a href="https://twitter.com/thukeg" target="_blank">Twitter</a> • 📃 <a href="https://arxiv.org/abs/2103.10360" target="_blank">[GLM@ACL 22]</a> <a href="https://github.com/THUDM/GLM" target="_blank">[GitHub]</a> • 📃 <a href="https://arxiv.org/abs/2210.02414" target="_blank">[GLM-130B@ICLR 23]</a> <a href="https://github.com/THUDM/GLM-130B" target="_blank">[GitHub]</a> <br>
|
||||
</p>
|
||||
|
|
@ -14,21 +13,29 @@ ChatGLM-6B is an open bilingual language model based on [General Language Model
|
|||
|
||||
ChatGLM-6B uses technology similar to ChatGPT, optimized for Chinese QA and dialogue. The model is trained for about 1T tokens of Chinese and English corpus, supplemented by supervised fine-tuning, feedback bootstrap, and reinforcement learning wit human feedback. With only about 6.2 billion parameters, the model is able to generate answers that are in line with human preference.
|
||||
|
||||
In order to facilitate downstream developers to customize the model for their own application scenarios, we also implements an parameter-efficient tuning method based on [P-Tuning v2](https://github.com/THUDM/P-tuning-v2)[(Guidelines)](ptuning/README_en.md). Tuning requires at least 7GB of GPU memory at INT4 quantization level.
|
||||
|
||||
Try the [online demo](https://huggingface.co/spaces/ysharma/ChatGLM-6b_Gradio_Streaming) on Huggingface Spaces.
|
||||
|
||||
## Update
|
||||
**[2023/03/31]** Added a parameter-efficient tuning implementation based on [P-Tuning-v2](https://github.com/THUDM/P-tuning-v2). The minimum INT4 quantization level only needs 7GB GPU memory is enough for model tuning. See [Parameter-efficient tuning method](ptuning/README.md) for details.
|
||||
|
||||
**[2023/03/23]** Add API deployment, thanks to [@LemonQu-GIT](https://github.com/LemonQu-GIT). Add embedding-quantized model [ChatGLM-6B-INT4-QE](https://huggingface.co/THUDM/chatglm-6b-int4-qe). Add support for GPU inference on Mac with Apple Silicon.
|
||||
|
||||
**[2023/03/19]** Add streaming output function `stream_chat`, already applied in web and CLI demo. Fix Chinese punctuations in output. Add quantized model [ChatGLM-6B-INT4](https://huggingface.co/THUDM/chatglm-6b-int4).
|
||||
|
||||
## Projects
|
||||
The following are some open source projects developed based on this repository:
|
||||
* [ChatGLM-MNN](https://github.com/wangzhaode/ChatGLM-MNN): An [MNN](https://github.com/alibaba/MNN)-based implementation of ChatGLM-6B C++ inference, which supports automatic allocation of computing tasks to GPU and CPU according to the size of GPU memory
|
||||
* [ChatGLM-Tuning](https://github.com/mymusise/ChatGLM-Tuning): Fine-tuning ChatGLM-6B based on LoRA
|
||||
Open source projects that accelerate ChatGLM:
|
||||
* [ChatGLM-MNN](https://github.com/wangzhaode/ChatGLM-MNN): An MNN-based implementation of ChatGLM-6B C++ inference, which supports automatic allocation of computing tasks to GPU and CPU according to the size of GPU memory
|
||||
* [JittorLLMs](https://github.com/Jittor/JittorLLMs): Running ChatGLM-6B in FP16 with a minimum of 3G GPU memory or no GPU at all, with Linux, windows, and Mac support
|
||||
|
||||
If you have other good projects, please refer to the above format to add to README and propose [PR](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).
|
||||
Open source projects using ChatGLM-6B:
|
||||
* [langchain-ChatGLM](https://github.com/imClumsyPanda/langchain-ChatGLM): ChatGLM application based on langchain, realizing Q&A based on extensible knowledge base
|
||||
* [Wenda](https://github.com/l15y/wenda): Large-scale language model call platform, based on ChatGLM-6B to achieve ChatPDF-like functions
|
||||
* [chatgpt_academic](https://github.com/binary-husky/chatgpt_academic): An academic writing and programming toolbox that supports ChatGLM-6B. It has the characteristics of modularization and multi-thread calling LLM, and can call multiple LLMs in parallel.
|
||||
* [glm-bot](https://github.com/initialencounter/glm-bot): Connect ChatGLM to Koishi to call ChatGLM on major chat platforms
|
||||
|
||||
Example projects supporting online training of ChatGLM-6B and related applications:
|
||||
* [ChatGLM-6B deployment and fine-tuning tutorial](https://www.heywhale.com/mw/project/6436d82948f7da1fee2be59e)
|
||||
* [ChatGLM-6B combined with langchain to implement local knowledge base QA Bot](https://www.heywhale.com/mw/project/643977aa446c45f4592a1e59)
|
||||
|
||||
Third-party evaluation:
|
||||
* [Measuring Massive Multitask Chinese Understanding](https://arxiv.org/abs/2304.12986)
|
||||
|
||||
For more open source projects, see [PROJECT.md](PROJECT.md)
|
||||
|
||||
## Getting Started
|
||||
|
||||
|
|
@ -71,10 +78,24 @@ Generate dialogue with the following code
|
|||
|
||||
如果这些方法无法帮助你入睡,你可以考虑咨询医生或睡眠专家,寻求进一步的建议。
|
||||
```
|
||||
The implementation of the model is still in development. If you want to fix the used model implementation to ensure compatibility, you can add the `revision="v0.1.0"` parameter in the `from_pretrained` call. `v0.1.0` is the latest version number. For a complete list of versions, see [Change Log](https://huggingface.co/THUDM/chatglm-6b#change-log).
|
||||
|
||||
The full model implementation is on [HuggingFace Hub](https://huggingface.co/THUDM/chatglm-6b).
|
||||
### Load the model locally
|
||||
The above code will automatically download the model implementation and checkpoints by [transformers](https://github.com/huggingface/transformers). The full model implementation can be found at [Hugging Face Hub](https://huggingface.co/THUDM/chatglm-6b). If your network environment is poor, downloading model parameters may take a long time or even fail. At this point, you can download the model to the local first, and then load it from the local.
|
||||
|
||||
### Demo
|
||||
To download models from Hugging Face Hub, you need to [install Git LFS](https://docs.github.com/zh/repositories/working-with-files/managing-large-files/installing-git-large-file-storage) , then run
|
||||
```Shell
|
||||
git clone https://huggingface.co/THUDM/chatglm-6b
|
||||
```
|
||||
|
||||
After downloading the model locally, replace `THUDM/chatglm-6b` in the above code with the path of your local `chatglm-6b` folder to load the model locally.
|
||||
|
||||
**Optional**: The implementation of the model is still in development. If you want to fix the used model implementation to ensure compatibility, you can execute
|
||||
```Shell
|
||||
git checkout v0.1.0
|
||||
```
|
||||
|
||||
## Demo & API
|
||||
|
||||
We provide a Web demo based on [Gradio](https://gradio.app) and a command line demo in the repo. First clone our repo with:
|
||||
|
||||
|
|
@ -83,9 +104,9 @@ git clone https://github.com/THUDM/ChatGLM-6B
|
|||
cd ChatGLM-6B
|
||||
```
|
||||
|
||||
#### Web Demo
|
||||
### Web Demo
|
||||
|
||||
|
||||
|
||||
|
||||
Install Gradio `pip install gradio`，and run [web_demo.py](web_demo.py):
|
||||
|
||||
|
|
@ -95,6 +116,8 @@ python web_demo.py
|
|||
|
||||
The program runs a web server and outputs the URL. Open the URL in the browser to use the web demo.
|
||||
|
||||
Thanks to [@AdamBear](https://github.com/AdamBear) for implementing a web demo based on Streamlit, see [#117](https://github.com/THUDM/ChatGLM-6B/pull/117 ).
|
||||
|
||||
#### CLI Demo
|
||||
|
||||
|
||||
|
|
@ -136,15 +159,16 @@ By default, the model parameters are loaded with FP16 precision, which require a
|
|||
|
||||
```python
|
||||
# Change according to your hardware. Only support 4/8 bit quantization now.
|
||||
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().quantize(4).cuda()
|
||||
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().quantize(8).cuda()
|
||||
```
|
||||
|
||||
After 2 to 3 rounds of dialogue, the GPU memory usage is about 10GB under 8-bit quantization, and only 6GB under 4-bit quantization. As the number of dialogue rounds increases, the corresponding GPU memory consumption also increases. Due to the use of relative position encoding, ChatGLM-6B theoretically supports an infinitely long context-length, but the performance will gradually decline after the total length exceeds 2048 (training length).
|
||||
|
||||
Model quantization brings a certain performance decline. After testing, ChatGLM-6B can still perform natural and smooth generation under 4-bit quantization. using [GPT-Q](https://arxiv.org/abs/2210.17323) etc. The quantization scheme can further compress the quantization accuracy/improve the model performance under the same quantization accuracy. You are welcome to submit corresponding Pull Requests.
|
||||
|
||||
**[2023/03/19]** The quantization costs about 13GB of CPU memory to load the FP16 model. If your CPU memory is limited, you can directly load the quantized model, which costs only 5.2GB CPU memory:
|
||||
The quantization costs about 13GB of CPU memory to load the FP16 model. If your CPU memory is limited, you can directly load the quantized model, which costs only 5.2GB CPU memory:
|
||||
```python
|
||||
# For INT8-quantized model, change "chatglm-6b-int4" to "chatglm-6b-int8"
|
||||
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).half().cuda()
|
||||
```
|
||||
|
||||
|
|
@ -156,20 +180,18 @@ If your computer is not equipped with GPU, you can also conduct inference on CPU
|
|||
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).float()
|
||||
```
|
||||
|
||||
**[2023/03/19]** If your CPU memory is limited, you can directly load the quantized model:
|
||||
If your CPU memory is limited, you can directly load the quantized model:
|
||||
```python
|
||||
# For INT8-quantized model, change "chatglm-6b-int4" to "chatglm-6b-int8"
|
||||
model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).float()
|
||||
```
|
||||
|
||||
If your encounter the error `Could not find module 'nvcuda.dll'` or `RuntimeError: Unknown platform: darwin`(MacOS), please refer to this [Issue](https://github.com/THUDM/ChatGLM-6B/issues/6#issuecomment-1470060041).
|
||||
If your encounter the error `Could not find module 'nvcuda.dll'` or `RuntimeError: Unknown platform: darwin`(MacOS), please [load the model locally](README_en.md#load-the-model-locally).
|
||||
|
||||
### GPU Inference on Mac
|
||||
For Macs (and MacBooks) with Apple Silicon, it is possible to use the MPS backend to run ChatGLM-6B on the GPU. First, you need to refer to Apple's [official instructions](https://developer.apple.com/metal/pytorch) to install PyTorch-Nightly. Then clone the model repository locally (you need to [install Git LFS](https://docs.github.com/zh/repositories/working-with-files/managing-large-files/installing-git-large-file-storage)）
|
||||
```shell
|
||||
git lfs install
|
||||
git clone https://huggingface.co/THUDM/chatglm-6b
|
||||
```
|
||||
Change the code to load the model from your local path, and use the mps backend:
|
||||
For Macs (and MacBooks) with Apple Silicon, it is possible to use the MPS backend to run ChatGLM-6B on the GPU. First, you need to refer to Apple's [official instructions](https://developer.apple.com/metal/pytorch) to install PyTorch-Nightly.
|
||||
|
||||
Currently you must [load the model locally](README_en.md#load-the-model-locally) on MacOS. Change the code to load the model from your local path, and use the mps backend:
|
||||
```python
|
||||
model = AutoModel.from_pretrained("your local path", trust_remote_code=True).half().to('mps')
|
||||
```
|
||||
|
|
@ -189,6 +211,17 @@ This will deploy the model onto two GPUs for inference. You can change `num_gpus
|
|||
## Parameter-efficient Tuning
|
||||
Parameter-efficient tuning based on [P-tuning v2](https://github.com/THUDM/P-tuning-v2). See [ptuning/README.md](ptuning/README.md) for details on how to use it.
|
||||
|
||||
## Update
|
||||
**[2023/04/16]** Added INT8 quantized model [ChatGLM-6B-INT8](https://huggingface.co/THUDM/chatglm-6b-int8). Added multi-GPU deployment (thanks to [@Cherrysaber](https://github.com/Cherrysaber)).
|
||||
|
||||
**[2023/04/06]** Improve the web demo interface (thanks to [@tuteng0915](https://github.com/tuteng0915)). Remove the image tokens in the embedding layer to reduce the memory usage (need to update the model files `pytorch_model-00001-of-00008.bin` and `pytorch_model-00008-of-00008.bin`, thanks to [@silverriver](https:/ /github.com/silverriver) for proposing the idea). Removed dependency on `icetk` (need to update model file `ice_text.model`).
|
||||
|
||||
**[2023/03/31]** Added a parameter-efficient tuning implementation based on [P-Tuning-v2](https://github.com/THUDM/P-tuning-v2). The minimum INT4 quantization level only needs 7GB GPU memory is enough for model tuning. See [Parameter-efficient tuning method](ptuning/README.md) for details.
|
||||
|
||||
**[2023/03/23]** Add API deployment, thanks to [@LemonQu-GIT](https://github.com/LemonQu-GIT). Add embedding-quantized model [ChatGLM-6B-INT4-QE](https://huggingface.co/THUDM/chatglm-6b-int4-qe). Add support for GPU inference on Mac with Apple Silicon.
|
||||
|
||||
**[2023/03/19]** Add streaming output function `stream_chat`, already applied in web and CLI demo. Fix Chinese punctuations in output. Add quantized model [ChatGLM-6B-INT4](https://huggingface.co/THUDM/chatglm-6b-int4).
|
||||
|
||||
## ChatGLM-6B Examples
|
||||
|
||||
The following are some Chinese examples with `web_demo.py`. Welcome to explore more possibility with ChatGLM-6B.
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
|
||||
下面以 [ADGEN](https://aclanthology.org/D19-1321.pdf) (广告生成) 数据集为例介绍代码的使用方法。
|
||||
|
||||
*Read this in [English](README_en.md).*
|
||||
*Read this in [English](README_en.md).
|
||||
|
||||
## 软件依赖
|
||||
运行微调需要4.27.1版本的`transformers`。除 ChatGLM-6B 的依赖之外，还需要安装以下依赖
|
||||
|
|
@ -26,7 +26,7 @@ ADGEN 数据集任务为根据输入（content）生成一段广告词（summary
|
|||
|
||||
### 训练
|
||||
|
||||
#### P-tuning v2
|
||||
#### P-Tuning v2
|
||||
|
||||
运行以下指令进行训练：
|
||||
```shell
|
||||
|
|
@ -36,7 +36,7 @@ bash train.sh
|
|||
|
||||
在默认配置 `quantization_bit=4`、`per_device_train_batch_size=1`、`gradient_accumulation_steps=16` 下，INT4 的模型参数被冻结，一次训练迭代会以 1 的批处理大小进行 16 次累加的前后向传播，等效为 16 的总批处理大小，此时最低只需 6.7G 显存。若想在同等批处理大小下提升训练效率，可在二者乘积不变的情况下，加大 `per_device_train_batch_size` 的值，但也会带来更多的显存消耗，请根据实际情况酌情调整。
|
||||
|
||||
如果你想要[从本地加载模型](https://github.com/THUDM/ChatGLM-6B#%E4%BB%8E%E6%9C%AC%E5%9C%B0%E5%8A%A0%E8%BD%BD%E6%A8%A1%E5%9E%8B)，可以将 `train.sh` 中的 `THUDM/chatglm-6b` 改为你本地的模型路径。
|
||||
如果你想要[从本地加载模型](../README_en.md#load-the-model-locally)，可以将 `train.sh` 中的 `THUDM/chatglm-6b` 改为你本地的模型路径。
|
||||
|
||||
#### Finetune
|
||||
|
||||
|
|
@ -48,11 +48,7 @@ bash ds_train_finetune.sh
|
|||
|
||||
### 推理
|
||||
|
||||
将 `evaluate.sh` 中的 `CHECKPOINT` 更改为训练时保存的 checkpoint 名称，运行以下指令进行模型推理和评测：
|
||||
```shell
|
||||
bash evaluate.sh
|
||||
```
|
||||
**[2023/04/10更新]** 在 P-tuning v2 训练时模型只保存 PrefixEncoder 部分的参数，所以在推理时需要同时加载原 ChatGLM-6B 模型以及 PrefixEncoder 的权重，因此需要指定参数（已更新 `evaluate.sh`） ：
|
||||
在 P-tuning v2 训练时模型只保存 PrefixEncoder 部分的参数，所以在推理时需要同时加载原 ChatGLM-6B 模型以及 PrefixEncoder 的权重，因此需要指定 `evaluate.sh` 中的参数：
|
||||
|
||||
```shell
|
||||
--model_name_or_path THUDM/chatglm-6b
|
||||
|
|
@ -96,11 +92,11 @@ bash evaluate.sh
|
|||
|
||||
#### 实验设置
|
||||
|
||||
```
|
||||
```
|
||||
max_source_length=64
|
||||
max_target_length=64
|
||||
max_steps=3000
|
||||
```
|
||||
```
|
||||
|
||||
##### P-tuning v2
|
||||
|
||||
|
|
@ -132,14 +128,10 @@ per_device_train_batch_size=16
|
|||
gradient_accumulation_steps=1
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 模型部署
|
||||
首先载入Tokenizer：
|
||||
|
||||
```python
|
||||
import os
|
||||
import torch
|
||||
from transformers import AutoConfig, AutoModel, AutoTokenizer
|
||||
|
||||
# 载入Tokenizer
|
||||
|
|
|
|||
|
|
@ -20,9 +20,12 @@ The task of the ADGEN dataset is to generate an advertisement word (summary) bas
|
|||
}
|
||||
```
|
||||
|
||||
From [Google Drive](https://drive.google.com/file/d/13_vf0xRTQsyneRKdD1bZIr93vBGOczrk/view?usp=sharing) or [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b3f119a008264b1cabd1/?dl=1) Download the processed ADGEN dataset, and put the decompressed `AdvertiseGen` directory into this directory.
|
||||
From [Google Drive](https://drive.google.com/file/d/13_vf0xRTQsyneRKdD1bZIr93vBGOczrk/view?usp=sharing) or [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b3f119a008264b1cabd1/?dl=1) download the processed ADGEN dataset, and put the decompressed `AdvertiseGen` directory into this directory.
|
||||
|
||||
### Training
|
||||
|
||||
#### P-Tuning v2
|
||||
|
||||
Run the following commands for training:
|
||||
```shell
|
||||
bash train.sh
|
||||
|
|
@ -31,11 +34,28 @@ bash train.sh
|
|||
|
||||
Under the default configuration of `per_device_train_batch_size=1`, `gradient_accumulation_steps=16`, the model parameters of INT4 are frozen, and a training iteration will perform 16 cumulative forward and backward propagations with a batch size of 1, which is equivalent to the total batch size of 16, and only 6.7G GPU memory is required at this time with `quantization_bit=4`. If you want to improve the training efficiency under the same batch size, you can increase the value of `per_device_train_batch_size` while keeping the product of the two unchanged, but it will also bring more GPU memory consumption, please adjust it according to the actual situation.
|
||||
|
||||
If you want to [load the model locally](../README_en.md#load-the-model-locally), you can change `THUDM/chatglm-6b` in `train.sh` to your local model path.
|
||||
|
||||
#### Finetune
|
||||
To finetune the full parameters, you need to install [Deepspeed](https://github.com/microsoft/DeepSpeed), and then run the following command:
|
||||
|
||||
```shell
|
||||
bash ds_train_finetune.sh
|
||||
```
|
||||
|
||||
### Inference
|
||||
|
||||
Change `CHECKPOINT` in `evaluate.sh` to the checkpoint name saved during training, and run the following commands for model inference and evaluation:
|
||||
During P-tuning v2 training, the model only saves the parameters of the PrefixEncoder part, so the original ChatGLM-6B model and the weight of the PrefixEncoder need to be loaded at the same time during inference, and the arguments need to be specified in `evaluate.sh`:
|
||||
|
||||
```shell
|
||||
bash evaluate.sh
|
||||
--model_name_or_path THUDM/chatglm-6b
|
||||
--ptuning_checkpoint $CHECKPOINT_PATH
|
||||
```
|
||||
|
||||
It is still compatible with the old version of Checkpoint saved with full parameters, just set `model_name_or_path` as before:
|
||||
|
||||
```shell
|
||||
--model_name_or_path $CHECKPOINT_PATH
|
||||
```
|
||||
|
||||
The evaluation indicators are Chinese Rouge score and BLEU-4. The generated results are saved in
|
||||
|
|
@ -45,34 +65,33 @@ The evaluation indicators are Chinese Rouge score and BLEU-4. The generated resu
|
|||
#### Example 1
|
||||
* Input: 类型#上衣\*材质#牛仔布\*颜色#白色\*风格#简约\*图案#刺绣\*衣样式#外套\*衣款式#破洞
|
||||
* Label: 简约而不简单的牛仔外套,白色的衣身十分百搭。衣身多处有做旧破洞设计,打破单调乏味,增加一丝造型看点。衣身后背处有趣味刺绣装饰,丰富层次感,彰显别样时尚。
|
||||
* Output[微调前]: 这件上衣的材质是牛仔布,颜色是白色,风格是简约,图案是刺绣,衣样式是外套,衣款式是破洞。
|
||||
* Output[微调后]: 这是一款简约的牛仔外套,破洞设计,将牛仔布破洞,带来一种随意与个性。破洞的牛仔外套,展现出时尚气息,带来一种休闲感。同时,刺绣图案,让整件外套更加立体。
|
||||
* Output[before tuning]: 这件上衣的材质是牛仔布,颜色是白色,风格是简约,图案是刺绣,衣样式是外套,衣款式是破洞。
|
||||
* Output[after tuning]: 这是一款简约的牛仔外套,破洞设计,将牛仔布破洞,带来一种随意与个性。破洞的牛仔外套,展现出时尚气息,带来一种休闲感。同时,刺绣图案,让整件外套更加立体。
|
||||
|
||||
#### Example 2
|
||||
|
||||
* Input: 类型#裙\*版型#显瘦\*风格#文艺\*风格#简约\*图案#印花\*图案#撞色\*裙下摆#压褶\*裙长#连衣裙\*裙领型#圆领
|
||||
* Label: 文艺个性的印花连衣裙,藏青色底蕴,低调又大气,撞色太阳花分布整个裙身,绚丽而美好,带来时尚减龄的气质。基础款的舒适圆领,简约不失大方,勾勒精致脸庞。领后是一粒包布扣固定,穿脱十分方便。前片立体的打褶设计,搭配后片压褶的做工,增添层次和空间感,显瘦又有型。
|
||||
* Output[微调前]: 类型#裙*版型#显瘦*风格#文艺*风格#简约*图案#印花*图案#撞色*裙下摆#压褶*裙长#连衣裙*裙领型#圆领 1\. 连衣裙:简约风格,裙长为膝盖以上,裙领型为圆领。2\. 裙下摆:压褶设计,使裙摆呈现出流畅的褶皱效果。3\. 裙领型:裙领型为圆领,使穿上连衣裙后更加有型。4\. 版型:采用显瘦设计,让连衣裙看起来更加苗条。5\. 风格:文艺风格,让连衣裙更加有内涵和品味。6\. 图案:印花设计,在连衣裙上印有独特的图案。7\. 撞色:采用撞色设计,让连衣裙在色彩上更加鲜明、富有层次感。
|
||||
* Output[微调后]: 这是一款文艺范的连衣裙,以印花为元素,采用简约的印花,既能够突出文艺气质,又能够展现简约风。在印花的同时又有领子和裙摆的压褶设计,更加凸显文艺气质。简约而不会过于单调,搭配出街,穿着十分舒适。
|
||||
* Output[before tuning]: 类型#裙*版型#显瘦*风格#文艺*风格#简约*图案#印花*图案#撞色*裙下摆#压褶*裙长#连衣裙*裙领型#圆领 1\. 连衣裙:简约风格,裙长为膝盖以上,裙领型为圆领。2\. 裙下摆:压褶设计,使裙摆呈现出流畅的褶皱效果。3\. 裙领型:裙领型为圆领,使穿上连衣裙后更加有型。4\. 版型:采用显瘦设计,让连衣裙看起来更加苗条。5\. 风格:文艺风格,让连衣裙更加有内涵和品味。6\. 图案:印花设计,在连衣裙上印有独特的图案。7\. 撞色:采用撞色设计,让连衣裙在色彩上更加鲜明、富有层次感。
|
||||
* Output[after tuning]: 这是一款文艺范的连衣裙,以印花为元素,采用简约的印花,既能够突出文艺气质,又能够展现简约风。在印花的同时又有领子和裙摆的压褶设计,更加凸显文艺气质。简约而不会过于单调,搭配出街,穿着十分舒适。
|
||||
|
||||
### evaluation result
|
||||
|
||||
| | P-tuning v2 | LoRA |
|
||||
| ------- | ----------- | ----- |
|
||||
| BLEU-4 | 7.71 | 6.13 |
|
||||
| Rouge-1 | 31.35 | 28.36 |
|
||||
| Rouge-2 | 7.19 | 4.38 |
|
||||
| Rouge-l | 25.17 | 17.54 |
|
||||
| | Finetune | P-tuning v2 | LoRA |
|
||||
| ------------- | ----------- | ----- | ------------- |
|
||||
| BLEU-4 | 8.01 | 8.10 | 7.62 |
|
||||
| Rouge-1 | 31.23 | 31.12 | 30.60 |
|
||||
| Rouge-2 | 7.36 | 7.11 | 6.96 |
|
||||
| Rouge-l | 25.08 | 24.97 | 24.80 |
|
||||
| Training Loss | 3.00 | 3.74 | 3.32 |
|
||||
|
||||
#### Experiment Settings
|
||||
|
||||
```
|
||||
```
|
||||
max_source_length=64
|
||||
max_target_length=64
|
||||
per_device_train_batch_size=1
|
||||
gradient_accumulation_steps=16
|
||||
max_steps=3000
|
||||
```
|
||||
```
|
||||
|
||||
##### P-tuning v2
|
||||
|
||||
|
|
@ -80,29 +99,101 @@ max_steps=3000
|
|||
pre_seq_len=128
|
||||
learning_rate=2e-2
|
||||
quantization_bit=4
|
||||
per_device_train_batch_size=16
|
||||
gradient_accumulation_steps=1
|
||||
```
|
||||
|
||||
##### Finetune
|
||||
|
||||
```
|
||||
learning_rate=1e-4
|
||||
fp16
|
||||
num_gpus=4
|
||||
per_device_train_batch_size=4
|
||||
gradient_accumulation_steps=1
|
||||
```
|
||||
|
||||
##### LoRA
|
||||
|
||||
```
|
||||
learning_rate=5e-4
|
||||
```
|
||||
|
||||
The implementation uses [simple_thu_chatglm6b](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/simple_thu_chatglm6b)
|
||||
|
||||
|
||||
```
|
||||
learning_rate=5e-4
|
||||
per_device_train_batch_size=16
|
||||
gradient_accumulation_steps=1
|
||||
```
|
||||
|
||||
## Model Deployment
|
||||
Replace `THUDM/chatglm-6b` in the corresponding demo or code with the path of the checkpoint after P-Tuning(in the example, `./output/adgen-chatglm-6b-pt-8-1e-2/ checkpoint-3000`). Note that the current fine-tuning does not support multiple rounds of data, so only the responses from the first round of the conversation are fine-tuned.
|
||||
First load the tokenizer:
|
||||
|
||||
```python
|
||||
from transformers import AutoConfig, AutoModel, AutoTokenizer
|
||||
|
||||
# Load Tokenizer
|
||||
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
|
||||
```
|
||||
|
||||
1. If a new Checkpoint needs to be loaded (only contains the PrefixEncoder parameter):
|
||||
|
||||
```python
|
||||
config = AutoConfig.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, pre_seq_len=128)
|
||||
model = AutoModel.from_pretrained("THUDM/chatglm-6b", config=config, trust_remote_code=True)
|
||||
prefix_state_dict = torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin"))
|
||||
new_prefix_state_dict = {}
|
||||
for k, v in prefix_state_dict.items():
|
||||
if k.startswith("transformer.prefix_encoder."):
|
||||
new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v
|
||||
model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)
|
||||
```
|
||||
Note that you may need to change `pre_seq_len` to the actual value of your training. If you [load model from local](../README_en.md#load-the-model-locally), you need to change `THUDM/chatglm-6b` to the local model path (not the checkpoint path).
|
||||
|
||||
2. If you need to load the old checkpoint (including both ChatGLM-6B and PrefixEncoder parameters), or perform full parameter fine-tuning, then directly load the entire checkpoint:
|
||||
|
||||
```python
|
||||
model = AutoModel.from_pretrained(CHECKPOINT_PATH, trust_remote_code=True)
|
||||
```
|
||||
|
||||
Then it can be quantified according to the needs, or it can be used directly:
|
||||
|
||||
```python
|
||||
# Comment out the following line if you don't use quantization
|
||||
model = model. quantize(4)
|
||||
model = model.half().cuda()
|
||||
model.transformer.prefix_encoder.float()
|
||||
model = model.eval()
|
||||
|
||||
response, history = model.chat(tokenizer, "Hello", history=[])
|
||||
```
|
||||
|
||||
**[23/04/19]** You can also directly run [web demo](./web_demo.py) which supports loading P-Tuning v2 checkpoint
|
||||
```shell
|
||||
bash web_demo.sh
|
||||
```
|
||||
It may be necessary to modify the content of [web_demo.sh](./web_demo.sh) to match your actual checkpoint situation.
|
||||
|
||||
## Use your own dataset
|
||||
Modify `train_file`, `validation_file` and `test_file` in `train.sh` and `evaluate.sh` to your own JSON format dataset paths, and change `prompt_column` and `response_column` to the keys in the JSON file corresponding to input text and output text.
|
||||
You may also need to increase `max_source_length` and `max_target_length` to match the maximum input and output lengths in your own dataset.
|
||||
|
||||
## TODO
|
||||
* [ ] Support for chat data
|
||||
* [ ] Support for full finetuning
|
||||
## Dialog Dataset
|
||||
|
||||
## quoting
|
||||
If you need to use multiple rounds of dialogue data to train the model, you can provide chat history. For example, the following is the training data for a three-round dialogue:
|
||||
|
||||
```json lines
|
||||
{"prompt": "长城h3风扇不转。继电器好的。保险丝好的传感器新的风扇也新的这是为什么。就是继电器缺一个信号线", "response": "用电脑能读数据流吗？水温多少", "history": []}
|
||||
{"prompt": "95", "response": "上下水管温差怎么样啊？空气是不是都排干净了呢？", "history": [["长城h3风扇不转。继电器好的。保险丝好的传感器新的风扇也新的这是为什么。就是继电器缺一个信号线", "用电脑能读数据流吗？水温多少"]]}
|
||||
{"prompt": "是的。上下水管都好的", "response": "那就要检查线路了，一般风扇继电器是由电脑控制吸合的，如果电路存在断路，或者电脑坏了的话会出现继电器不吸合的情况！", "history": [["长城h3风扇不转。继电器好的。保险丝好的传感器新的风扇也新的这是为什么。就是继电器缺一个信号线", "用电脑能读数据流吗？水温多少"], ["95", "上下水管温差怎么样啊？空气是不是都排干净了呢？"]]}
|
||||
```
|
||||
|
||||
During training, you need to specify `--history_column` as the key of the chat history in the data (`history` in this example), and the chat history will be stitched automatically. Note that content exceeding the input length `max_source_length` will be truncated.
|
||||
|
||||
You can refer to the following instructions:
|
||||
|
||||
```shell
|
||||
bash train_chat.sh
|
||||
```
|
||||
|
||||
## Citation
|
||||
|
||||
```
|
||||
@inproceedings{liu2022p,
|
||||
|
|
|
|||
Loading…
Reference in New Issue