mirror of https://github.com/hpcaitech/ColossalAI
b5f9e37c70
* [legacy] remove outdated codes of pipeline (#4692) * [legacy] remove cli of benchmark and update optim (#4690) * [legacy] remove cli of benchmark and update optim * [doc] fix cli doc test * [legacy] fix engine clip grad norm * [legacy] remove outdated colo tensor (#4694) * [legacy] remove outdated colo tensor * [test] fix test import * [legacy] move outdated zero to legacy (#4696) * [legacy] clean up utils (#4700) * [legacy] clean up utils * [example] update examples * [legacy] clean up amp * [legacy] fix amp module * [legacy] clean up gpc (#4742) * [legacy] clean up context * [legacy] clean core, constants and global vars * [legacy] refactor initialize * [example] fix examples ci * [example] fix examples ci * [legacy] fix tests * [example] fix gpt example * [example] fix examples ci * [devops] fix ci installation * [example] fix examples ci |
||
|---|---|---|
| .. | ||
| README.md | ||
| config.py | ||
| requirements.txt | ||
| test_ci.sh | ||
| train.py | ||
README.md
Multi-dimensional Parallelism with Colossal-AI
Table of contents
📚 Overview
This example lets you to quickly try out the hybrid parallelism provided by Colossal-AI.
You can change the parameters below to try out different settings in the config.py.
# parallel setting
TENSOR_PARALLEL_SIZE = 2
TENSOR_PARALLEL_MODE = '1d'
parallel = dict(
pipeline=2,
tensor=dict(mode=TENSOR_PARALLEL_MODE, size=TENSOR_PARALLEL_SIZE),
)
🚀 Quick Start
-
Install PyTorch
-
Install the dependencies.
pip install -r requirements.txt
- Run the training scripts with synthetic data.
colossalai run --nproc_per_node 4 train.py --config config.py
- Modify the config file to play with different types of tensor parallelism, for example, change tensor parallel size to be 4 and mode to be 2d and run on 8 GPUs.