InternLM

Commit Graph

Author	SHA1	Message	Date
Wenwen Qu	929dd36cf2	avoid moe parameter partition in zero optimizer	2023-08-15 17:52:51 +08:00
Wenwen Qu	c7f9d4f48c	add expert data support and fix bugs	2023-08-10 16:07:35 +08:00
Wenwen Qu	e21ddc364b	modify grad clipping due to moe	2023-08-09 15:04:19 +08:00
Wenwen Qu	9ad7942568	Merge branch 'develop' into feature_add_moe	2023-08-08 16:51:10 +08:00
Wenwen Qu	84476833f3	modified: internlm/core/context/process_group_initializer.py modified: internlm/core/scheduler/no_pipeline_scheduler.py modified: internlm/solver/optimizer/hybrid_zero_optim.py	2023-08-08 15:59:12 +08:00
huangting4201	ff0fa7659f	feat(monitor): support monitor and alert (#175 ) * feat(monitor): support monitor and alert * feat(monitor.py): fix demo error * feat(monitor.py): move cmd monitor args to config file * feat(hybrid_zero_optim.py): if overflow occurs send alert msg * feat(monitor.py): remove alert msg filter * feat(monitor.py): optimize class MonitorTracker * feat(monitor.py): optimize code * feat(monitor.py): optimize code * feat(monitor.py): optimize code * feat(monitor.py): optimize code * feat(train.py): update print to log * style(ci): fix lint error * fix(utils/evaluation.py): remove useless code * fix(model/modeling_internlm.py): fix lint error --------- Co-authored-by: huangting4201 <huangting3@sensetime.com>	2023-08-08 11:18:15 +08:00
Wenwen Qu	c357288a8b	feat(XXX): add moe	2023-08-07 20:17:49 +08:00
ytxiong	853becfb6e	feat(): support fp32 training (#155 ) support float32 training * fix lint * add adaptation in model/utils.py * remove some unnecessary code * fix lint * feat(optim): add support for fp32 zero * Revert "Merge pull request #2 from SolenoidWGT/fp32_zero" This reverts commit `53fc50b0e5`, reversing changes made to `40f24d0a73`. revert commit * merge develop * Update utils.py * support fp32 in zero optimizer * modify the dtype --------- Co-authored-by: wangguoteng.p <wangguoteng925@qq.com>	2023-08-04 16:05:30 +08:00
huangting4201	762ab297ee	feat(core/scheduler): support pipeline parallel (#98 ) * feat(utils/writer.py): support tensorboard writer * feat(utils/writer.py): add class comment * feat(core): support pipeline parallel * fix(core): fix demo running error * feat(solver/optimizer): add pp zero optimizer * fix(solver/optimizer): fix word spelling error * feat(core/scheduler): add new dir scheduler in core/ * fix(core): fix ci lint error * feat(solver/optimizer): merge pp and nopp optimizer * doc(usage.md): update usage doc * feat(core/scheduler): support post func * feat(core/scheduler): add dtype para in pp sche and update func get_tensor_shape * feat(core/scheduler): add _load_micro_batch in base scheduler * feat(core/scheduler): support optimizer overlap communication in pp scheduler * feat(core/scheduler): delete data process func code * feat(core/trainer): schedule pre processing for all schedule --------- Co-authored-by: 黄婷 <huangting3@CN0014010744M.local> Co-authored-by: huangting.p <huangting@sensetime.com>	2023-07-24 20:52:09 +08:00
Sun Peng	fa7337b37b	initial commit	2023-07-06 12:55:23 +08:00

10 Commits (929dd36cf20281fb1faf8a5c0987260369073fa8)