5 Commits (acea4554ecb8b74d3619d58c5a1a6fc7dbfc08b9)

Author	SHA1	Message	Date
Miao Zheng	1095263082	[Fix]Fix wrong string cutoff in the script for sft text tokenizing (#106)	2023-07-19 12:12:41 +08:00
x54-729	0c1060435d	Use tempfile for convert2hf.py (#23) ... Fix https://github.com/InternLM/InternLM/issues/50	2023-07-17 21:08:10 +08:00
Yang Gao	555ed60a2c	fix(tokenizer): refactor tokenizer and update usage in readme (#51) ... * update tokenizer example	2023-07-13 17:16:27 +08:00
yhcc	745d2b911a	Fix readme about conversion to transformers (#25) ... * add links for 8k * fix acknowledgement * modified readme for convert_hf	2023-07-07 13:38:06 +08:00
Sun Peng	fa7337b37b	initial commit	2023-07-06 12:55:23 +08:00