mirror of https://github.com/hpcaitech/ColossalAI
b5f9e37c70
* [legacy] remove outdated codes of pipeline (#4692) * [legacy] remove cli of benchmark and update optim (#4690) * [legacy] remove cli of benchmark and update optim * [doc] fix cli doc test * [legacy] fix engine clip grad norm * [legacy] remove outdated colo tensor (#4694) * [legacy] remove outdated colo tensor * [test] fix test import * [legacy] move outdated zero to legacy (#4696) * [legacy] clean up utils (#4700) * [legacy] clean up utils * [example] update examples * [legacy] clean up amp * [legacy] fix amp module * [legacy] clean up gpc (#4742) * [legacy] clean up context * [legacy] clean core, constants and global vars * [legacy] refactor initialize * [example] fix examples ci * [example] fix examples ci * [legacy] fix tests * [example] fix gpt example * [example] fix examples ci * [devops] fix ci installation * [example] fix examples ci |
||
|---|---|---|
| .. | ||
| model | ||
| utils | ||
| README.md | ||
| arguments.py | ||
| bert_dataset_provider.py | ||
| evaluation.py | ||
| hostfile | ||
| loss.py | ||
| nvidia_bert_dataset_provider.py | ||
| pretrain_utils.py | ||
| run_pretrain.sh | ||
| run_pretrain_resume.sh | ||
| run_pretraining.py | ||
README.md
Pretraining
- Pretraining roberta through running the script below. Detailed parameter descriptions can be found in the arguments.py.
data_path_prefixis absolute path specifies output of preprocessing. You have to modify the hostfile according to your cluster.
bash run_pretrain.sh
--hostfile: servers' host name from /etc/hosts--include: servers which will be used--nproc_per_node: number of process(GPU) from each server--data_path_prefix: absolute location of train data, e.g., /h5/0.h5--eval_data_path_prefix: absolute location of eval data--tokenizer_path: tokenizer path contains huggingface tokenizer.json, e.g./tokenizer/tokenizer.json--bert_config: config.json which represent model--mlm: model type of backbone, bert or deberta_v2
- if resume training from earlier checkpoint, run the script below.
bash run_pretrain_resume.sh
--resume_train: whether to resume training--load_pretrain_model: absolute path which contains model checkpoint--load_optimizer_lr: absolute path which contains optimizer checkpoint