ColossalAI/examples/language/gpt/README.md

Overview

This example shows how to use Colossal-AI to run huggingface GPT training in distributed manners.

GPT

We use the GPT2 model from huggingface transformers. The input data is randonly generated.

Our Modifications

The train_gpt_demo.py provides three distributed plans, i.e. Colossal-AI, PyTorch DDP and ZeRO. The Colossal-AI leverages Tensor Parallel and Gemini.

Quick Start

You can launch training by using the following bash script.

pip install -r requirements.txt
bash run.sh