mirror of https://github.com/hpcaitech/ColossalAI
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
14 lines
400 B
14 lines
400 B
8 months ago
|
SAVE_DIR=""
|
||
|
|
|
||
|
|
rm -rf $SAVE_DIR/cache
|
||
|
|
rm -rf $SAVE_DIR/jsonl
|
||
|
|
rm -rf $SAVE_DIR/arrow
|
||
|
|
|
||
|
|
python prepare_dataset.py --type prompt \
|
||
|
|
--data_input_dirs /PATH/TO/PROMPT/DATASET \
|
||
|
|
--conversation_template_config /PATH/TO/CHAT/TEMPLATE/CONFIG.json \
|
||
|
|
--tokenizer_dir "" \
|
||
|
|
--data_cache_dir $SAVE_DIR/cache \
|
||
|
|
--data_jsonl_output_dir $SAVE_DIR/jsonl \
|
||
|
|
--data_arrow_output_dir $SAVE_DIR/arrow
|