github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

[example] Polish README.md (#2658) 

* [tutorial] polish readme.md

* [example] Update README.md
pull/2659/head
Jiatong (Julius) Han 2023-02-09 20:43:55 +08:00 committed by GitHub
parent cd4f02bed8
commit a255a38f7f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 6 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
examples/language/gpt/README.md
View File

@ -54,13 +54,13 @@ However, it requires some efforts to start if facing a new model structure.
bash run_gemini.sh bash run_gemini.sh
``` ```
The `train_gpt_demo.py` provides three distributed plans, you can choose the plan you want in `run_gemini.sh`. The Colossal-AI leverages Tensor Parallel and Gemini + ZeRO DDP. The `train_gpt_demo.py` provides three distributed plans (except ones already provided by PyTorch), you can choose the plan you want in `run_gemini.sh`. The CAI_Gemini leverages Tensor Parallel and Gemini + ZeRO DDP. For their differences, you may check out the answer to issue [here](https://github.com/hpcaitech/ColossalAI/issues/2590#issuecomment-1418766581).
- Colossal-AI - ZeRO1 (CAI_ZeRO1)
- ZeRO1 (Colossal-AI) - ZeRO2 (CAI_ZeRO2)
- ZeRO2 (Colossal-AI) - Gemini + ZeRO DDP (CAI_Gemini)
- Pytorch DDP - Pytorch DDP (Pytorch_DDP)
- Pytorch ZeRO - Pytorch ZeRO (Pytorch_ZeRO)
### Titans (Tensor Parallelism) + ZeRO + Pipeline Parallelism ### Titans (Tensor Parallelism) + ZeRO + Pipeline Parallelism