mirror of https://github.com/hpcaitech/ColossalAI
360674283d
|
||
|---|---|---|
| .. | ||
| README.md | ||
| benchmark.sh | ||
| requirements.txt | ||
| run_gemini.sh | ||
| test_ci.sh | ||
| train_gemini_opt.py | ||
README.md
OPT
Meta recently released Open Pretrained Transformer (OPT), a 175-Billion parameter AI language model, which stimulates AI programmers to perform various downstream tasks and application deployments.
The following example of Colossal-AI demonstrates fine-tuning Casual Language Modelling at low cost.
We are using the pre-training weights of the OPT model provided by Hugging Face Hub on the raw WikiText-2 (no tokens were replaced before the tokenization). This training script is adapted from the HuggingFace Language Modelling examples.
Our Modifications
We adapt the OPT training code to ColossalAI by leveraging Gemini and ZeRO DDP.
Quick Start
You can launch training by using the following bash script
bash ./run_gemini.sh