github
/
InternLM
mirror of https://github.com/InternLM/InternLM
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

update README about convert2hf.py

pull/23/head
x54-729 2023-07-10 17:18:56 +08:00
parent cc02611e59
commit ed23624303
4 changed files with 6 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
README-zh-Hans.md
View File

@ -139,10 +139,10 @@ streamlit run web_demo.py
### 转换为 Transformers 格式使用
通过 InternLM 进行训练的模型可以很轻松地转换为 HuggingFace Transformers 格式，方便与社区各种开源项目无缝对接。借助 `tools/convert2hf.py` 可以将训练保存的权重一键转换为 transformers 格式
通过 InternLM 进行训练的模型可以很轻松地转换为 HuggingFace Transformers 格式，方便与社区各种开源项目无缝对接。借助 `tools/transformers/convert2hf.py` 可以将训练保存的权重一键转换为 transformers 格式
```bash
python convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer tokenizes/tokenizer.model
python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer .tools/V7_sft.model
```
转换之后可以通过以下的代码加载为 transformers

4
README.md
View File

@ -142,10 +142,10 @@ Please refer to [Usage Tutorial](./doc/en/usage.md) to start InternLM installati
### Convert to Transformers Format
The model trained by InternLM can be easily converted to HuggingFace Transformers format, which is convenient for seamless docking with various open source projects in the community. With the help of `tools/convert2hf.py`, the weights saved during training can be converted into transformers format with one command
The model trained by InternLM can be easily converted to HuggingFace Transformers format, which is convenient for seamless docking with various open source projects in the community. With the help of `tools/transformers/convert2hf.py`, the weights saved during training can be converted into transformers format with one command
```bash
python convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer tokenizes/tokenizer.model
python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer .tools/V7_sft.model
```
After conversion, it can be loaded as transformers by the following code

2
tools/transformers/README-zh-Hans.md
View File

@ -11,7 +11,7 @@
`convert2hf.py` 可以将训练保存的权重一键转换为 transformers 格式。在仓库根目录运行以下命令：
```bash
python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer ../V7_sft.model
python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer .tools/V7_sft.model
```
然后可以使用 `from_pretrained` 接口加载：

2
tools/transformers/README.md
View File

@ -10,7 +10,7 @@ This folder contains the `InternLM` model in transformers format.
`convert2hf.py` can convert saved training weights into the transformers format with a single command. Execute the command in the root directory of repository:
```bash
python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer ../V7_sft.model
python tools/transformers/convert2hf.py --src_folder origin_ckpt/ --tgt_folder hf_ckpt/ --tokenizer .tools/V7_sft.model
```
Then, you can load it using the `from_pretrained` interface: