github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
ColossalAI/examples/language/gpt
History
Jiarui Fang 60abd86d6a
[example] enhance GPT demo (#1959) 
* [example] enhence GPT demo

* Update README.md

Co-authored-by: binmakeswell <binmakeswell@gmail.com>
 		2022-11-16 11:36:27 +08:00
..
README.md [example] enhance GPT demo (#1959) 2022-11-16 11:36:27 +08:00
requirements.txt [example] simplify the GPT2 huggingface example (#1826) 2022-11-08 16:14:07 +08:00
run.sh [example] enhance GPT demo (#1959) 2022-11-16 11:36:27 +08:00
train_gpt_demo.py [example] enhance GPT demo (#1959) 2022-11-16 11:36:27 +08:00

README.md

Overview

This example shows how to use Colossal-AI to run huggingface GPT training in distributed manners.

GPT

We use the GPT2 model from huggingface transformers. The input data is randonly generated.

Our Modifications

The train_gpt_demo.py provides three distributed plans, i.e. Colossal-AI, PyTorch DDP and ZeRO. The Colossal-AI leverages Tensor Parallel and Gemini.

Quick Start

You can launch training by using the following bash script.

pip install -r requirements.txt
bash run.sh