github
/
InternLM
mirror of https://github.com/InternLM/InternLM
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

fix import of tools/tokenizer.py

pull/467/head
x54-729 2023-11-02 23:17:30 +08:00
parent e5bdfd1892
commit cccd216977
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
tools/tokenizer.py
View File

@ -8,7 +8,7 @@ import numpy as np
current_dir = os.path.dirname(os.path.abspath(__file__))
model_path = os.path.join(current_dir, "V7_sft.model")
sys.path.append(os.path.join(current_dir, "transformers"))
from tokenization_internlm import InternLMTokenizer
from internlm_model import InternLMTokenizer
tokenizer = InternLMTokenizer(vocab_file=model_path)