15 Commits (f5caa1c048801ba948dd0344dfb554220951cb1b)

Author	SHA1	Message	Date
Wenwen Qu	1b25bc0325	fix merge bugs	2023-09-21 16:18:26 +08:00
Qu Wenwen	b596255356	merge feature_add_moe into feature_add_moe_data	2023-09-21 15:47:31 +08:00
Qu Wenwen	9665321745	refactor code	2023-09-21 11:51:17 +08:00
Qu Wenwen	4a47872382	refactor code	2023-09-19 12:30:40 +08:00
Wenwen Qu	8a595837fc	merge upstream/develop into feature_add_moe	2023-09-11 16:20:08 +08:00
Wenwen Qu	cd6b28b073	use dummy mode to generate random numbers in model construction	2023-09-08 17:56:42 +08:00
Wenwen Qu	b021995199	fix bugs	2023-08-30 16:14:33 +08:00
Wenwen Qu	629e6a5ad1	add comments for moe	2023-08-25 19:03:31 +08:00
Wenwen Qu	86bcda5ca9	add default setting for expert parallel size	2023-08-24 18:52:12 +08:00
Wenwen Qu	12c614db94	create expert data group and broadcast moe parameter in expert data group	2023-08-21 11:40:39 +08:00
Wenwen Qu	c7f9d4f48c	add expert data support and fix bugs	2023-08-10 16:07:35 +08:00
Wenwen Qu	84476833f3	modified: internlm/core/context/process_group_initializer.py ... modified: internlm/core/scheduler/no_pipeline_scheduler.py modified: internlm/solver/optimizer/hybrid_zero_optim.py	2023-08-08 15:59:12 +08:00
Wenwen Qu	c357288a8b	feat(XXX): add moe	2023-08-07 20:17:49 +08:00
ytxiong	c219065348	feat(*): support sequence_parallel (#180) ... * support sequence_parallel for no pipeline * sequence_parallel does not support no-flash-attn * support sequence parallel for pipeline * add memory profiler * Update 13B.py * add memory profiler * fix evaluation bug * remove some unnecessary code * remove some unnecessary code * Update parallel_context.py * modify the config * remove memory profiler * modify the config * support selective dropout	2023-08-07 16:42:52 +08:00
Sun Peng	fa7337b37b	initial commit	2023-07-06 12:55:23 +08:00