- Image source: https://openai.com/blog/chatgpt
+Image source: https://openai.com/blog/chatgpt
+
**As Colossal-AI is undergoing some major updates, this project will be actively maintained to stay in line with the Colossal-AI project.**
-
More details can be found in the latest news.
-* [2023/03] [ColossalChat: An Open-Source Solution for Cloning ChatGPT With a Complete RLHF Pipeline](https://medium.com/@yangyou_berkeley/colossalchat-an-open-source-solution-for-cloning-chatgpt-with-a-complete-rlhf-pipeline-5edf08fb538b)
-* [2023/02] [Open Source Solution Replicates ChatGPT Training Process! Ready to go with only 1.6GB GPU Memory](https://www.hpc-ai.tech/blog/colossal-ai-chatgpt)
+
+- [2023/03] [ColossalChat: An Open-Source Solution for Cloning ChatGPT With a Complete RLHF Pipeline](https://medium.com/@yangyou_berkeley/colossalchat-an-open-source-solution-for-cloning-chatgpt-with-a-complete-rlhf-pipeline-5edf08fb538b)
+- [2023/02] [Open Source Solution Replicates ChatGPT Training Process! Ready to go with only 1.6GB GPU Memory](https://www.hpc-ai.tech/blog/colossal-ai-chatgpt)
## Online demo
+
@@ -83,13 +87,13 @@ More details can be found in the latest news.
-> DeepSpeedChat performance comes from its blog on 2023 April 12, ColossalChat performance can be reproduced on an AWS p4d.24xlarge node with 8 A100-40G GPUs with the following command: torchrun --standalone --nproc_per_node 8 benchmark_opt_lora_dummy.py --num_collect_steps 1 --use_kernels --strategy colossalai_zero2 --experience_batch_size 64 --train_batch_size 32
+> DeepSpeedChat performance comes from its blog on 2023 April 12, ColossalChat performance can be reproduced on an AWS p4d.24xlarge node with 8 A100-40G GPUs with the following command: `torchrun --standalone --nproc_per_node 8 benchmark_opt_lora_dummy.py --num_collect_steps 1 --use_kernels --strategy colossalai_zero2 --experience_batch_size 64 --train_batch_size 32`
## Install
### Install the environment
-```shell
+```bash
conda create -n coati
conda activate coati
git clone https://github.com/hpcaitech/ColossalAI.git
@@ -99,7 +103,7 @@ pip install .
### Install the Transformers
-```shell
+```bash
pip install transformers==4.30.2
```
@@ -107,10 +111,11 @@ pip install transformers==4.30.2
### Supervised datasets collection
-we collected 104K bilingual datasets of Chinese and English, and you can find the datasets in this repo
-[InstructionWild](https://github.com/XueFuzhao/InstructionWild)
+We collected 104K bilingual datasets of Chinese and English, and you can find the datasets in this repo
+[InstructionWild](https://github.com/XueFuzhao/InstructionWild) and in this [file](https://github.com/XueFuzhao/InstructionWild/blob/main/data/README.md).
Here is how we collected the data
+
@@ -397,18 +455,22 @@ Thanks so much to all of our amazing contributors!
| Better Cases | 38 ⚔ **41** | **45** ⚔ 33 |
| Win Rate | 48% ⚔ **52%** | **58%** ⚔ 42% |
| Average Score | 7.06 ⚔ **7.13** | **7.31** ⚔ 6.82 |
+
- Our Coati-7B model performs better than Alpaca-7B when using GPT-4 to evaluate model performance. The Coati-7B model we evaluate is an old version we trained a few weeks ago and the new version is around the corner.
## Authors
Coati is developed by ColossalAI Team:
+
- [Fazzie](https://fazzie-key.cool/about/index.html)
- [FrankLeeeee](https://github.com/FrankLeeeee)
- [BlueRum](https://github.com/ht-zhou)
- [ver217](https://github.com/ver217)
- [ofey404](https://github.com/ofey404)
+- [Wenhao Chen](https://github.com/CWHer)
The Phd student from [(HPC-AI) Lab](https://ai.comp.nus.edu.sg/) also contributed a lot to this project.
+
- [Zangwei Zheng](https://github.com/zhengzangw)
- [Xue Fuzhao](https://github.com/XueFuzhao)
diff --git a/applications/Chat/benchmarks/README.md b/applications/Chat/benchmarks/README.md
index bc8ad8ba9..c13f34858 100644
--- a/applications/Chat/benchmarks/README.md
+++ b/applications/Chat/benchmarks/README.md
@@ -27,9 +27,12 @@ We also provide various training strategies:
We only support `torchrun` to launch now. E.g.
-```shell
+```bash
# run OPT-125M with no lora (lora_rank=0) on single-node single-GPU with min batch size
-torchrun --standalone --nproc_per_node 1 benchmark_opt_lora_dummy.py --model 125m --critic_model 125m --strategy ddp --experience_batch_size 1 --train_batch_size 1 --lora_rank 0
+torchrun --standalone --nproc_per_node 1 benchmark_opt_lora_dummy.py \
+ --model 125m --critic_model 125m --strategy ddp \
+ --experience_batch_size 1 --train_batch_size 1 --lora_rank 0
# run Actor (OPT-1.3B) and Critic (OPT-350M) with lora_rank=4 on single-node 4-GPU
-torchrun --standalone --nproc_per_node 4 benchmark_opt_lora_dummy.py --model 1.3b --critic_model 350m --strategy colossalai_zero2 --lora_rank 4
+torchrun --standalone --nproc_per_node 4 benchmark_opt_lora_dummy.py \
+ --model 1.3b --critic_model 350m --strategy colossalai_zero2 --lora_rank 4
```
diff --git a/applications/Chat/coati/ray/README.md b/applications/Chat/coati/ray/README.md
index 228155a68..79b1db347 100644
--- a/applications/Chat/coati/ray/README.md
+++ b/applications/Chat/coati/ray/README.md
@@ -1,3 +1,5 @@
+:warning: **This content may be outdated since the major update of Colossal Chat. We will update this content soon.**
+
# Distributed PPO Training on Stage 3
## Detach Experience Makers and Trainers
@@ -26,124 +28,137 @@ See examples at `ColossalAI/application/Chat/examples/ray`
- define makers' environment variables :
- ```python
- env_info_makers = [{
- 'local_rank': '0',
- 'rank': str(rank),
- 'world_size': str(num_makers),
- 'master_port': maker_port,
- 'master_addr': master_addr
- } for rank in range(num_makers)]
+ ```python
+ env_info_makers = [{
+ 'local_rank': '0',
+ 'rank': str(rank),
+ 'world_size': str(num_makers),
+ 'master_port': maker_port,
+ 'master_addr': master_addr
+ } for rank in range(num_makers)]
+
+ ```
- ```
- define maker models :
- ```python
- def model_fn():
- actor = get_actor_from_args(...)
- critic = get_critic_from_args(...)
- reward_model = get_reward_model_from_args(...)
- initial_model = get_actor_from_args(...)
- return actor, critic, reward_model, initial_model
- ```
+ ```python
+ def model_fn():
+ actor = get_actor_from_args(...)
+ critic = get_critic_from_args(...)
+ reward_model = get_reward_model_from_args(...)
+ initial_model = get_actor_from_args(...)
+ return actor, critic, reward_model, initial_model
+
+ ```
+
- set experience_holder_refs :
- ```python
- experience_holder_refs = [
- ExperienceMakerHolder.options(
- name=f"maker_{i}",
- num_gpus=1,
- max_concurrency=2
- ).remote(
- detached_trainer_name_list=[f"trainer_{x}" for x in target_trainers(...)],
- model_fn=model_fn,
- ...)
- for i, env_info_maker in enumerate(env_info_makers)
- ]
- ```
- The names in the `detached_trainer_name_list` refer to the target trainers that the maker should send experience to.
- We set a trainer's name the same as a maker, by `.options(name="str")`. See below.
+ ```python
+ experience_holder_refs = [
+ ExperienceMakerHolder.options(
+ name=f"maker_{i}",
+ num_gpus=1,
+ max_concurrency=2
+ ).remote(
+ detached_trainer_name_list=[f"trainer_{x}" for x in target_trainers(...)],
+ model_fn=model_fn,
+ ...)
+ for i, env_info_maker in enumerate(env_info_makers)
+ ]
+ ```
+
+ The names in the `detached_trainer_name_list` refer to the target trainers that the maker should send experience to.
+ We set a trainer's name the same as a maker, by `.options(name="str")`. See below.
### Setup Trainers
- define trainers' environment variables :
- ```python
- env_info_trainers = [{
- 'local_rank': '0',
- 'rank': str(rank),
- 'world_size': str(num_trainers),
- 'master_port': trainer_port,
- 'master_addr': master_addr
- } for rank in range(num_trainers)]
- ```
+ ```python
+ env_info_trainers = [{
+ 'local_rank': '0',
+ 'rank': str(rank),
+ 'world_size': str(num_trainers),
+ 'master_port': trainer_port,
+ 'master_addr': master_addr
+ } for rank in range(num_trainers)]
+ ```
- define trainer models :
- ```python
- def trainer_model_fn():
- actor = get_actor_from_args(...)
- critic = get_critic_from_args(...)
- return actor, critic
- ```
+ ```python
+ def trainer_model_fn():
+ actor = get_actor_from_args(...)
+ critic = get_critic_from_args(...)
+ return actor, critic
+ ```
+
- set trainer_refs :
- ```python
- trainer_refs = [
- DetachedPPOTrainer.options(
- name=f"trainer{i}",
- num_gpus=1,
- max_concurrency=2
- ).remote(
- experience_maker_holder_name_list=[f"maker{x}" for x in target_makers(...)],
- model_fn = trainer_model_fn(),
- ...)
- for i, env_info_trainer in enumerate(env_info_trainers)
- ]
- ```
- The names in `experience_maker_holder_name_list` refer to the target makers that the trainer should send updated models to.
- By setting `detached_trainer_name_list` and `experience_maker_holder_name_list`, we can customize the transmission graph.
+ ```python
+ trainer_refs = [
+ DetachedPPOTrainer.options(
+ name=f"trainer{i}",
+ num_gpus=1,
+ max_concurrency=2
+ ).remote(
+ experience_maker_holder_name_list=[f"maker{x}" for x in target_makers(...)],
+ model_fn = trainer_model_fn(),
+ ...)
+ for i, env_info_trainer in enumerate(env_info_trainers)
+ ]
+ ```
+ The names in `experience_maker_holder_name_list` refer to the target makers that the trainer should send updated models to.
+ By setting `detached_trainer_name_list` and `experience_maker_holder_name_list`, we can customize the transmission graph.
### Launch Jobs
+
- define data_loader :
- ```python
- def data_loader_fn():
- return = torch.utils.data.DataLoader(dataset=dataset)
- ```
+ ```python
+ def data_loader_fn():
+ return = torch.utils.data.DataLoader(dataset=dataset)
+
+ ```
+
- launch makers :
- ```python
- wait_tasks = []
- for experience_holder_ref in experience_holder_refs:
- wait_tasks.append(
- experience_holder_ref.workingloop.remote(data_loader_fn(),
- num_steps=experience_steps))
- ```
+ ```python
+ wait_tasks = []
+ for experience_holder_ref in experience_holder_refs:
+ wait_tasks.append(
+ experience_holder_ref.workingloop.remote(data_loader_fn(),
+ num_steps=experience_steps))
+
+ ```
- launch trainers :
- ```python
- for trainer_ref in trainer_refs:
- wait_tasks.append(trainer_ref.fit.remote(total_steps, update_steps, train_epochs))
- ```
+
+ ```python
+ for trainer_ref in trainer_refs:
+ wait_tasks.append(trainer_ref.fit.remote(total_steps, update_steps, train_epochs))
+ ```
- wait for done :
- ```python
- ray.get(wait_tasks)
- ```
+ ```python
+ ray.get(wait_tasks)
+ ```
## Flexible Structure
We can deploy different strategies to makers and trainers. Here are some notions.
### 2 Makers 1 Trainer
+
@@ -162,20 +184,20 @@ Model performance in [Anthropics paper](https://arxiv.org/abs/2204.05862):