mirror of https://github.com/hpcaitech/ColossalAI
7f8b16635b
* [misc] remove config arg from initialize * [misc] remove old tensor contrusctor * [plugin] add npu support for ddp * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [devops] fix doc test ci * [test] fix test launch * [doc] update launch doc --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| configs | ||
| dataset | ||
| model | ||
| LICENSE | ||
| README.md | ||
| requirements.txt | ||
| run.sh | ||
| test_ci.sh | ||
| train_gpt.py | ||
README.md
Run GPT With Colossal-AI
How to Prepare Webtext Dataset
You can download the preprocessed sample dataset for this demo via our Google Drive sharing link.
You can also avoid dataset preparation by using --use_dummy_dataset during running.
Run this Demo
Use the following commands to install prerequisites.
# assuming using cuda 11.3
pip install -r requirements.txt
Use the following commands to execute training.
#!/usr/bin/env sh
# if you want to use real dataset, then remove --use_dummy_dataset
# export DATA=/path/to/small-gpt-dataset.json'
# run on a single node
colossalai run --nproc_per_node=<num_gpus> train_gpt.py --config configs/<config_file> --from_torch --use_dummy_dataset
# run on multiple nodes
colossalai run --nproc_per_node=<num_gpus> \
--master_addr <hostname> \
--master_port <port-number> \
--hosts <list-of-hostname-separated-by-comma> \
train_gpt.py \
--config configs/<config_file> \
--from_torch \
--use_dummy_dataset
# run on multiple nodes with slurm
srun python \
train_gpt.py \
--config configs/<config_file> \
--host <master_node> \
--use_dummy_dataset
You can set the <config_file> to any file in the configs folder. To simply get it running, you can start with gpt_small_zero3_pp1d.py on a single node first. You can view the explanations in the config file regarding how to change the parallel setting.