ColossalAI/examples/community/roberta/pretraining
binmakeswell f1b3d60cae
[example] reorganize for community examples (#3557)
2023-04-14 16:27:48 +08:00
..
model [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
utils [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
README.md [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
arguments.py [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
bert_dataset_provider.py [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
evaluation.py [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
hostfile [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
loss.py [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
nvidia_bert_dataset_provider.py [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
pretrain_utils.py [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
run_pretrain.sh [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
run_pretrain_resume.sh [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00
run_pretraining.py [example] reorganize for community examples (#3557) 2023-04-14 16:27:48 +08:00

README.md

Pretraining

  1. Pretraining roberta through running the script below. Detailed parameter descriptions can be found in the arguments.py. data_path_prefix is absolute path specifies output of preprocessing. You have to modify the hostfile according to your cluster.
bash run_pretrain.sh
  • --hostfile: servers' host name from /etc/hosts
  • --include: servers which will be used
  • --nproc_per_node: number of process(GPU) from each server
  • --data_path_prefix: absolute location of train data, e.g., /h5/0.h5
  • --eval_data_path_prefix: absolute location of eval data
  • --tokenizer_path: tokenizer path contains huggingface tokenizer.json, e.g./tokenizer/tokenizer.json
  • --bert_config: config.json which represent model
  • --mlm: model type of backbone, bert or deberta_v2
  1. if resume training from earylier checkpoint, run the script below.
bash run_pretrain_resume.sh
  • --resume_train: whether to resume training
  • --load_pretrain_model: absolute path which contains model checkpoint
  • --load_optimizer_lr: absolute path which contains optimizer checkpoint