github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
ColossalAI/examples/language/gpt
History
Jiarui Fang f7e276fa71
[Gemini] add GeminiAdamOptimizer (#1960) 		2022-11-16 14:44:28 +08:00
..
README.md [Gemini] add GeminiAdamOptimizer (#1960) 2022-11-16 14:44:28 +08:00
requirements.txt [example] simplify the GPT2 huggingface example (#1826) 2022-11-08 16:14:07 +08:00
run.sh [example] enhance GPT demo (#1959) 2022-11-16 11:36:27 +08:00
train_gpt_demo.py [Gemini] add GeminiAdamOptimizer (#1960) 2022-11-16 14:44:28 +08:00

README.md

Overview

This example shows how to use Colossal-AI to run huggingface GPT training in distributed manners.

GPT

We use the GPT2 model from huggingface transformers. The input data is randonly generated. The train_gpt_demo.py provides three distributed plans, i.e. ColossalAI, PyTorch DDP and ZeRO. The ColossalAI leverages Tensor Parallel and Gemini.

Quick Start

You can launch training by using the following bash script.

pip install -r requirements.txt
bash run.sh