mirror of https://github.com/hpcaitech/ColossalAI
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
13 lines
363 B
13 lines
363 B
torchrun --standalone --nproc_per_node=4 train_sft.py \
|
|
--pretrain "/path/to/LLaMa-7B/" \
|
|
--model 'llama' \
|
|
--strategy colossalai_zero2 \
|
|
--log_interval 10 \
|
|
--save_path /path/to/Coati-7B \
|
|
--dataset /path/to/data.json \
|
|
--batch_size 4 \
|
|
--accumulation_steps 8 \
|
|
--lr 2e-5 \
|
|
--max_datasets_size 512 \
|
|
--max_epochs 1 \
|