55ae973cf0
update moe config file
2023-09-26 12:17:39 +08:00
75774e0b5e
Merge branch 'feature_add_moe' of https://github.com/blankde/InternLM into feature_add_moe
2023-09-26 11:52:52 +08:00
3c8fee01b2
add compatible code for old version
2023-09-26 11:51:34 +08:00
1986116527
add moe module to `__init__.py`
2023-09-25 15:36:35 +08:00
85f4d4af58
fix bugs in save/load moe checkpoint
2023-09-22 17:54:33 +08:00
f6cadcafa2
fix bugs with split group
2023-09-22 16:05:21 +08:00
1fdc7107b4
Merge branch 'feature_add_moe' of https://github.com/blankde/InternLM into feature_add_moe
2023-09-22 15:57:36 +08:00
fd932cfc09
refactor
2023-09-22 15:52:37 +08:00
ccdaf8ec45
fix the moe_loss for ci and val
2023-09-22 15:45:36 +08:00
a5e80bcc56
refactor code for split param group
2023-09-22 15:24:28 +08:00
3df0a51555
fix logger for moe
2023-09-22 14:56:43 +08:00
36d1bd2f41
Merge branch 'develop' of github.com:InternLM/InternLM into feature_add_moe
2023-09-22 14:36:58 +08:00
aa7645a831
Merge pull request #4 from blankde/feature_add_moe_refactor_zl
...
refactor code
2023-09-22 14:22:45 +08:00
9e6e7986b6
refactor code for log
2023-09-22 14:14:58 +08:00
1ed36754df
feat(.github/workflows): update ci e2e tests and add ci unit tests ( #324 )
...
* feat(.github/workflows/e2e_test.yaml): update e2e yaml
* feat(.github/workflows/e2e_test.yaml): update e2e yaml
* test e2e
* test e2e
* test e2e
* test e2e
* test e2e
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): add weekly tests
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-22 14:07:14 +08:00
f5337f6e02
Feat(PythonGC): Do garbage collection manually ( #326 )
...
* feat:add gc control
* feat:add gc control
* feat:add gc control
* feat:add gc
* re-lint
2023-09-22 13:52:25 +08:00
3607548265
refactor code for split group
2023-09-22 13:00:26 +08:00
548d1bd7af
refactor code
2023-09-22 12:30:02 +08:00
80972ff314
refactor code
2023-09-22 11:47:05 +08:00
17bc5f562b
refactor code
2023-09-21 15:00:28 +08:00
3b0eff0c8a
fix(model/embedding.py): ci lint check error ( #345 )
...
* fix(ci): fix ci lint error
* fix(ci): fix ci lint error
2023-09-21 14:46:22 +08:00
9665321745
refactor code
2023-09-21 11:51:17 +08:00
8464425a7b
feat(mdoel): add DynamicNTKScalingRotaryEmbedding ( #339 )
...
* add dynamic ntk rope
* update dynamic ntk rope
* fix lint check
* fix lint check
* add more desc
---------
Co-authored-by: YWMditto <862779238@qq.com>
2023-09-20 23:31:47 +08:00
67eda4cbe1
fix(.github/workflows/e2e_test.yaml): update ci runner name
2023-09-19 18:13:20 +08:00
30b21075e8
merge main
2023-09-19 18:04:47 +08:00
bba9b01c0e
Update 7B_sft.py
2023-09-19 17:45:48 +08:00
b7ddc42dcd
merge Internlm/develop into feature_add_moe
2023-09-19 17:44:12 +08:00
6a5915bf0d
feat(linear): optimize mlp by using jit ( #321 )
...
* fuse silu op
* refactor code
* fix lint
* fix lint
2023-09-19 14:57:43 +08:00
025ca55dfe
test(tests/test_training): add training e2e tests for loss spike and loss accuracy ( #304 )
...
* tests(test_training): add test case for loss accuracy
* tests(test_training): update test cases
* ci(.github/workflows/e2e_test.yaml): remove pull submodule
* ci(.github/workflows/e2e_test.yaml): update ci env and remove useless env var
* test(tests/test_training): add 16 GPUs test cases
* test(tests/test_training): fix training_16GPU_8DP2PP test case error
* test(tests/test_training): add new case for interleaved pp
* test(tests/test_training): remove redundant code
* test(tests/test_training): update ci job timeout minutes to 30m
* feat(initialize/launch.py): check num_chunks and interleaved_overlap
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-19 14:55:40 +08:00
bfefc4ea3c
test(ci_scripts): move ci env ( #317 )
...
* change partition and runner label
* change rm action to mv
* use spot
* use rsync to move test files
* remove *
* remove *
* change into llm_s partition
---------
Co-authored-by: wangmengke <wangmengke@pjlab.org.cn>
2023-09-19 14:52:32 +08:00
b9824fab89
fix(tools): fix yield bug in stream_chat ( #315 )
2023-09-19 14:18:02 +08:00
cd6426a249
feat(tools): support openai api ( #313 )
...
* fix(chat): fix stream_chat to return generator (#123 )
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302 )
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299 )
* support openai api to deploy internlm
* update README for information os openai_api.py
* change example in README_EN.md to English
* delete unnecessary print; fix model card typo; fix chat epoch
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: huangting4201 <1538303371@qq.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
2023-09-19 13:49:48 +08:00
0af5175073
merge internlm/develop into feature_add_moe
2023-09-19 13:27:43 +08:00
4a47872382
refactor code
2023-09-19 12:30:40 +08:00
b2f3611b47
Merge pull request #3 from blankde/fix/gate_inconsistent_issue
...
fix(gate): gate inconsistent issue
2023-09-19 11:36:29 +08:00
edc18bcddd
fix precision inconsistency
2023-09-18 20:54:52 +08:00
2710fa7343
Merge develop to main ( #314 )
...
* feat: add unitest for model (#300 )
* feat: add unitest for model
* feat:add model test
* Merge main to develop (#309 )
* fix(chat): fix stream_chat to return generator (#123 )
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302 )
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299 )
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
* docs(doc/code-docs): add figure for training docs (#307 )
* add training image for docs
* docs(doc/code-docs): add training img for en doc
* docs(doc/code-docs): fix en docs for initialize
* docs(doc/code-docs): update conf file for readthedocs
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): fix typos for reathedocs
* docs(doc/code-docs): minor typo fix for reathedocs
* docs(doc/code-docs): fix readthedocs conf file
* docs(doc/code-docs): update training image
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): update training image
* docs(doc/code-docs): move training image to section initialize
* docs(doc/code-docs): fix lint
* add badge about reathedocs status
* Merge main to develop (#312 )
* fix(chat): fix stream_chat to return generator (#123 )
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302 )
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299 )
* docs(doc/code-docs): update quickstart usage (#301 )
* docs(usage.md): update usage.md
* docs(doc/code-docs): update en usage
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
* docs(doc/code-docs): update en usage
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
Co-authored-by: huangting4201 <huangting3@sensetime.com>
* feat: more tgs (#310 )
* feat:more tgs
* feat:add more tgs
* feat:more tgs
* feat: add optimizer_unitest (#303 )
* feat: add optimizer_unitest
* feat: add optimizer test
* feat: add optimizer test
* feat:add optimizer test
* fianl change
* feat:add optimizer test
* feat:add optimizer test
* feat:add optimizer test
---------
Co-authored-by: jiaxingli <43110891+li126com@users.noreply.github.com>
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
Co-authored-by: Season <caizheng@pjlab.org.cn>
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-15 19:12:38 +08:00
ab513e1ddd
feat: add optimizer_unitest ( #303 )
...
* feat: add optimizer_unitest
* feat: add optimizer test
* feat: add optimizer test
* feat:add optimizer test
* fianl change
* feat:add optimizer test
* feat:add optimizer test
* feat:add optimizer test
2023-09-15 18:56:56 +08:00
794a484666
feat: more tgs ( #310 )
...
* feat:more tgs
* feat:add more tgs
* feat:more tgs
2023-09-15 18:56:11 +08:00
5aa5c96ec8
refactor split_moe_group code
2023-09-15 16:55:16 +08:00
607f691e16
Merge main to develop ( #312 )
...
* fix(chat): fix stream_chat to return generator (#123 )
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302 )
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299 )
* docs(doc/code-docs): update quickstart usage (#301 )
* docs(usage.md): update usage.md
* docs(doc/code-docs): update en usage
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
* docs(doc/code-docs): update en usage
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-15 16:19:26 +08:00
42802a2b31
docs(doc/code-docs): update quickstart usage ( #301 )
...
* docs(usage.md): update usage.md
* docs(doc/code-docs): update en usage
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-15 15:29:58 +08:00
de68cc5007
docs(doc/code-docs): add figure for training docs ( #307 )
...
* add training image for docs
* docs(doc/code-docs): add training img for en doc
* docs(doc/code-docs): fix en docs for initialize
* docs(doc/code-docs): update conf file for readthedocs
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): fix typos for reathedocs
* docs(doc/code-docs): minor typo fix for reathedocs
* docs(doc/code-docs): fix readthedocs conf file
* docs(doc/code-docs): update training image
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): update training image
* docs(doc/code-docs): move training image to section initialize
* docs(doc/code-docs): fix lint
* add badge about reathedocs status
2023-09-15 15:22:22 +08:00
d13f5d3048
Merge branch 'feature_add_moe' of https://github.com/blankde/InternLM into feature_add_moe
2023-09-15 12:26:42 +08:00
462b849942
fix interface for dense pipeline
2023-09-15 12:12:45 +08:00
07fc5f674a
Merge main to develop ( #309 )
...
* fix(chat): fix stream_chat to return generator (#123 )
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302 )
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299 )
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
2023-09-14 16:32:15 +08:00
882a07011c
feat: add unitest for model ( #300 )
...
* feat: add unitest for model
* feat:add model test
2023-09-14 13:18:34 +08:00
717f0c9e64
change float16 to bfloat16
2023-09-13 13:00:49 +08:00
d218a62b79
Merge branch 'develop' of github.com:InternLM/InternLM into feature_add_moe
...
Conflicts:
internlm/core/context/parallel_context.py
internlm/core/context/process_group_initializer.py
internlm/model/modeling_internlm.py
internlm/solver/optimizer/hybrid_zero_optim.py
internlm/train/training_internlm.py
internlm/utils/model_checkpoint.py
train.py
2023-09-12 18:04:48 +08:00
09e71cebf3
fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError ( #299 )
2023-09-11 20:17:11 +08:00
Wenwen Qu