update README & unittest

README-ja-JP.md

@@ -177,7 +177,7 @@ InternLMのインストール、データ処理、プレトレーニング、フ
 InternLM によって学習されたモデルは、コミュニティの様々なオープンソースプロジェクトとシームレスにドッキングするのに便利な Hugging Face Transformers 形式に簡単に変換することができます。`tools/convert2hf.py` の助けを借りて、トレーニング中に保存された weights は 1 つのコマンドで transformers 形式に変換することができます
 
 ```bash
-python convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer tokenizes/tokenizer.model
+python convert2hf.py --src origin_ckpt/ --tgt hf_ckpt/ --tokenizer tokenizes/tokenizer.model --max_pos 4096 
 ```
 
 変換後、以下のコードで transformers として読み込むことができます

README-zh-Hans.md

@@ -270,7 +270,7 @@ torchrun --master_port 12331 --nnodes=1 --node_rank=0 --nproc_per_node=1 -m stre
 通过 InternLM 进行训练的模型可以很轻松地转换为 HuggingFace Transformers 格式，方便与社区各种开源项目无缝对接。借助 `tools/transformers/convert2hf.py` 可以将训练保存的权重一键转换为 transformers 格式
 
 ```bash
-python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer ./tools/V7_sft.model
+python tools/transformers/convert2hf.py --src origin_ckpt/ --tgt hf_ckpt/ --tokenizer ./tools/V7_sft.model --max_pose 4096
 ```
 
 转换之后可以通过以下的代码加载为 transformers

README.md

@@ -267,7 +267,7 @@ Please refer to [Usage Tutorial](./doc/en/usage.md) to start InternLM installati
 The model trained by InternLM can be easily converted to HuggingFace Transformers format, which is convenient for seamless docking with various open source projects in the community. With the help of `tools/transformers/convert2hf.py`, the weights saved during training can be converted into transformers format with one command
 
 ```bash
-python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer ./tools/V7_sft.model
+python tools/transformers/convert2hf.py --src origin_ckpt/ --tgt hf_ckpt/ --tokenizer ./tools/V7_sft.model --max_pose 4096
 ```
 
 After conversion, it can be loaded as transformers by the following code

ci_scripts/model/convert_to_hf.sh

@@ -25,7 +25,7 @@ if [[ -d ${CKPTS_OUTPUT} ]]; then
     fi
 fi
 
-python ./tools/transformers/convert2hf.py --src_folder ${CKPTS_INPUT} --tgt_folder ${CKPTS_OUTPUT} --tokenizer ./tools/V7_sft.model
+python ./tools/transformers/convert2hf.py --src ${CKPTS_INPUT} --tgt ${CKPTS_OUTPUT} --tokenizer ./tools/V7_sft.model
 [[ $? -ne 0 ]] && { echo "test convert2hf.py failed.";  exit_code=$(($exit_code + 1)); }
 
 #assert exists model

tools/transformers/README-zh-Hans.md

@@ -11,7 +11,7 @@
 `convert2hf.py` 可以将训练保存的权重一键转换为 transformers 格式。在仓库根目录运行以下命令：
 
 ```bash
-python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer ./tools/V7_sft.model
+python tools/transformers/convert2hf.py --src origin_ckpt/ --tgt hf_ckpt/ --tokenizer ./tools/V7_sft.model --max_pose 4096
 ```
 
 然后可以使用 `from_pretrained` 接口加载：

tools/transformers/README.md

@@ -10,7 +10,7 @@ This folder contains the `InternLM` model in transformers format.
 `convert2hf.py` can convert saved training weights into the transformers format with a single command. Execute the command in the root directory of repository:
 
 ```bash
-python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer ./tools/V7_sft.model
+python tools/transformers/convert2hf.py --src origin_ckpt/ --tgt hf_ckpt/ --tokenizer ./tools/V7_sft.model --max_pose 4096
 ```
 
 Then, you can load it using the `from_pretrained` interface: