* Fix errant inference_forward.
* Recover use_dynamic_ntk_rope.
* Fix bugs.
* Fit to flash attention 1.0
* Fit to flash attention 1.0
* Fit to flash attention 1.0.5.
* Fit to flash attention 1.0.5.
* feat(fsdp): add training option for fsdp
* fix(fsdp): add mix-precision training
* fix failure in lint-check
* fix format problem
* restore 7B_sft
* fix load ckpt bug
* fix load ckpt bug2
* feat(solver/optimizer): add new file fsdp_optimizer.py
* fix(train.py): fix ci lint error
* fix(fsdp_optimizer.py): wait grad async
* fix bug for loading ckpts when zero1 < dp_size
* fix(context/parallel_context.py): only log warning for fsdp
* change ckpt name
* fix(model/modeling_internlm.py): fix checkpoint=False runtime error
* more wrap
* add support for FSDP with tp
* modify args_sanity_check for fsdp with pipeline and fsdp with moe
* fix(internlm/utils/parallel.py): fix circular import
* fix(internlm/train/training_internlm.py): remove set IS_TENSOR_PARALLEL attr
* fix(internlm/train/training_internlm.py): update wrap class and fix lint error
* fix(internlm/model): reset dropout_selective_checkpoint=True
* feat(configs/7B_sft.py): move fsdp config to parallel zero1
* feat(configs/7B_sft.py): adapt to old version config
---------
Co-authored-by: huangting4201 <1538303371@qq.com>
* support zero for expert local dp
* fix above codes:
*treat optim.zero_world_size and optim.zero_local_rank as list in model_checkpoint.py and test_model_checkpoint.py
*add overlap and zero check for moe in args_sanity_check(.)
* add fused precision support for norm
* refactor code
* refactor code
* change the granularity of hook
* fix bugs if self.model is ModuleList
* add dtype condition for post hook
* refactor code for split group
* refactor code for pre/post hook
* refactor code for split group
* remove fp32 hook for norm
* unit tests for fused precision
* add doc for fused precision
* add doc for En. version
* reformat docs
* Update mixed_precision.rst
* Update mixed_precision.po
* update mixed_precision.po
* add long text generation in doc/usage.md
* add long text generation in doc/usage.md
* add long text generation in doc/usage.md
---------
Co-authored-by: YWMditto <862779238@qq.com>
* feat:add numa
* feat:add bind numa
* feat:add bind numa
* feat:add bind numa
* feat: bind numa
* feat: bind numa
* feat: add numa
* feat:add numa
* feat:add numa
* try_bind_numa should not raise exception
---------
Co-authored-by: 877825076@qq.com <877825076@qq.com>
* fix(storage): fix try_get_storage_backend
* fix typo and print infos only in log rank
* fix typo and print infos only in log rank
---------
Co-authored-by: gaoyang07 <Gary1546308416AL@gmail.com>
* feat(.github/workflows/e2e_test.yaml): update e2e yaml
* feat(.github/workflows/e2e_test.yaml): update e2e yaml
* test e2e
* test e2e
* test e2e
* test e2e
* test e2e
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): add weekly tests
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
* tests(test_training): add test case for loss accuracy
* tests(test_training): update test cases
* ci(.github/workflows/e2e_test.yaml): remove pull submodule
* ci(.github/workflows/e2e_test.yaml): update ci env and remove useless env var
* test(tests/test_training): add 16 GPUs test cases
* test(tests/test_training): fix training_16GPU_8DP2PP test case error
* test(tests/test_training): add new case for interleaved pp
* test(tests/test_training): remove redundant code
* test(tests/test_training): update ci job timeout minutes to 30m
* feat(initialize/launch.py): check num_chunks and interleaved_overlap
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
* fix(chat): fix stream_chat to return generator (#123)
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302)
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299)
* support openai api to deploy internlm
* update README for information os openai_api.py
* change example in README_EN.md to English
* delete unnecessary print; fix model card typo; fix chat epoch
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: huangting4201 <1538303371@qq.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
* add training image for docs
* docs(doc/code-docs): add training img for en doc
* docs(doc/code-docs): fix en docs for initialize
* docs(doc/code-docs): update conf file for readthedocs
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): fix typos for reathedocs
* docs(doc/code-docs): minor typo fix for reathedocs
* docs(doc/code-docs): fix readthedocs conf file
* docs(doc/code-docs): update training image
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): update training image
* docs(doc/code-docs): move training image to section initialize
* docs(doc/code-docs): fix lint
* add badge about reathedocs status
jiaxingli