diff --git a/README-ja-JP.md b/README-ja-JP.md index 8b0c8b1..233ec99 100644 --- a/README-ja-JP.md +++ b/README-ja-JP.md @@ -177,7 +177,7 @@ InternLMのインストール、データ処理、プレトレーニング、フ InternLM によって学習されたモデルは、コミュニティの様々なオープンソースプロジェクトとシームレスにドッキングするのに便利な Hugging Face Transformers 形式に簡単に変換することができます。`tools/convert2hf.py` の助けを借りて、トレーニング中に保存された weights は 1 つのコマンドで transformers 形式に変換することができます ```bash -python convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer tokenizes/tokenizer.model +python convert2hf.py --src origin_ckpt/ --tgt hf_ckpt/ --tokenizer tokenizes/tokenizer.model --max_pos 4096 ``` 変換後、以下のコードで transformers として読み込むことができます diff --git a/README-zh-Hans.md b/README-zh-Hans.md index 209e908..f5faf2e 100644 --- a/README-zh-Hans.md +++ b/README-zh-Hans.md @@ -270,7 +270,7 @@ torchrun --master_port 12331 --nnodes=1 --node_rank=0 --nproc_per_node=1 -m stre 通过 InternLM 进行训练的模型可以很轻松地转换为 HuggingFace Transformers 格式,方便与社区各种开源项目无缝对接。借助 `tools/transformers/convert2hf.py` 可以将训练保存的权重一键转换为 transformers 格式 ```bash -python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer ./tools/V7_sft.model +python tools/transformers/convert2hf.py --src origin_ckpt/ --tgt hf_ckpt/ --tokenizer ./tools/V7_sft.model --max_pose 4096 ``` 转换之后可以通过以下的代码加载为 transformers diff --git a/README.md b/README.md index bba0f1f..b39f834 100644 --- a/README.md +++ b/README.md @@ -267,7 +267,7 @@ Please refer to [Usage Tutorial](./doc/en/usage.md) to start InternLM installati The model trained by InternLM can be easily converted to HuggingFace Transformers format, which is convenient for seamless docking with various open source projects in the community. With the help of `tools/transformers/convert2hf.py`, the weights saved during training can be converted into transformers format with one command ```bash -python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer ./tools/V7_sft.model +python tools/transformers/convert2hf.py --src origin_ckpt/ --tgt hf_ckpt/ --tokenizer ./tools/V7_sft.model --max_pose 4096 ``` After conversion, it can be loaded as transformers by the following code diff --git a/ci_scripts/model/convert_to_hf.sh b/ci_scripts/model/convert_to_hf.sh index d1af389..c652984 100644 --- a/ci_scripts/model/convert_to_hf.sh +++ b/ci_scripts/model/convert_to_hf.sh @@ -25,7 +25,7 @@ if [[ -d ${CKPTS_OUTPUT} ]]; then fi fi -python ./tools/transformers/convert2hf.py --src_folder ${CKPTS_INPUT} --tgt_folder ${CKPTS_OUTPUT} --tokenizer ./tools/V7_sft.model +python ./tools/transformers/convert2hf.py --src ${CKPTS_INPUT} --tgt ${CKPTS_OUTPUT} --tokenizer ./tools/V7_sft.model [[ $? -ne 0 ]] && { echo "test convert2hf.py failed."; exit_code=$(($exit_code + 1)); } #assert exists model diff --git a/tools/transformers/README-zh-Hans.md b/tools/transformers/README-zh-Hans.md index 8bbdaf5..4c8ee85 100644 --- a/tools/transformers/README-zh-Hans.md +++ b/tools/transformers/README-zh-Hans.md @@ -11,7 +11,7 @@ `convert2hf.py` 可以将训练保存的权重一键转换为 transformers 格式。在仓库根目录运行以下命令: ```bash -python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer ./tools/V7_sft.model +python tools/transformers/convert2hf.py --src origin_ckpt/ --tgt hf_ckpt/ --tokenizer ./tools/V7_sft.model --max_pose 4096 ``` 然后可以使用 `from_pretrained` 接口加载: diff --git a/tools/transformers/README.md b/tools/transformers/README.md index 4fe2a92..40d18bc 100644 --- a/tools/transformers/README.md +++ b/tools/transformers/README.md @@ -10,7 +10,7 @@ This folder contains the `InternLM` model in transformers format. `convert2hf.py` can convert saved training weights into the transformers format with a single command. Execute the command in the root directory of repository: ```bash -python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer ./tools/V7_sft.model +python tools/transformers/convert2hf.py --src origin_ckpt/ --tgt hf_ckpt/ --tokenizer ./tools/V7_sft.model --max_pose 4096 ``` Then, you can load it using the `from_pretrained` interface: