github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
ColossalAI/examples/language/gpt/experiments/auto_parallel
History
digger-yu b9a8dff7e5
[doc] Fix typo under colossalai and doc(#3618) 
* Fixed several spelling errors under colossalai

* Fix the spelling error in colossalai and docs directory

* Cautious Changed the spelling error under the example folder

* Update runtime_preparation_pass.py

revert autograft to autograd

* Update search_chunk.py

utile to until

* Update check_installation.py

change misteach to mismatch in line 91

* Update 1D_tensor_parallel.md

revert to perceptron

* Update 2D_tensor_parallel.md

revert to perceptron in line 73

* Update 2p5D_tensor_parallel.md

revert to perceptron in line 71

* Update 3D_tensor_parallel.md

revert to perceptron in line 80

* Update README.md

revert to resnet in line 42

* Update reorder_graph.py

revert to indice in line 7

* Update p2p.py

revert to megatron in line 94

* Update initialize.py

revert to torchrun in line 198

* Update routers.py

change to detailed in line 63

* Update routers.py

change to detailed in line 146

* Update README.md

revert  random number in line 402
 		2023-04-26 11:38:43 +08:00
..
saved_solution [autoparallel] integrate device mesh initialization into autoparallelize (#2393) 2023-01-11 14:03:49 +08:00
README.md [doc] Fix typo under colossalai and doc(#3618) 2023-04-26 11:38:43 +08:00
auto_parallel_with_gpt.py [test] refactor tests with spawn (#3452) 2023-04-06 14:51:35 +08:00
gpt_modules.py [example] upload auto parallel gpt2 demo (#2354) 2023-01-06 11:38:38 +08:00
requirements.txt [auto] fix requirements typo for issue #3125 (#3209) 2023-03-23 10:22:08 +08:00

README.md

Auto-Parallelism with GPT2

Requirements

Before you can launch training, you need to install the following requirements.

Install PyTorch

#conda
conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch
#pip
pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113

Install Colossal-AI v0.2.0 From Official Website

pip install colossalai==0.2.0+torch1.12cu11.3 -f https://release.colossalai.org

Install transformers

pip install transformers

Install pulp and coin-or-cbc

pip install pulp
conda install -c conda-forge coin-or-cbc

Dataset

For simplicity, the input data is randomly generated here.

Training

#Run the auto parallel resnet example with 4 GPUs with a dummy dataset.
colossalai run --nproc_per_node 4 auto_parallel_with_gpt.py