ColossalAI

Commit Graph

Author	SHA1	Message	Date
ver217	a241f61b34	[zero] Update initialize for ZeRO (#458 ) * polish code * shard strategy receive pg in shard() / gather() * update zero engine * polish code	2022-03-18 16:18:31 +08:00
ver217	642846d6f9	update sharded optim and fix zero init ctx (#457 )	2022-03-18 15:44:47 +08:00
Jiarui Fang	e2e9f82588	Revert "[zero] update sharded optim and fix zero init ctx" (#456 ) * Revert "polish code" This reverts commit `8cf7ff08cf`. * Revert "rename variables" This reverts commit `e99af94ab8`. * Revert "remove surplus imports" This reverts commit `46add4a5c5`. * Revert "update sharded optim and fix zero init ctx" This reverts commit `57567ee768`.	2022-03-18 15:22:43 +08:00
ver217	8cf7ff08cf	polish code	2022-03-18 14:25:25 +08:00
ver217	57567ee768	update sharded optim and fix zero init ctx	2022-03-18 14:25:25 +08:00
Jiarui Fang	0fcfb1e00d	[test] make zero engine test really work (#447 )	2022-03-17 17:24:25 +08:00
Jiarui Fang	496cbb0760	[hotfix] fix initialize bug with zero (#442 )	2022-03-17 13:16:22 +08:00
Jiarui Fang	17b8274f8a	[unitest] polish zero config in unittest (#438 )	2022-03-17 10:20:53 +08:00
Frank Lee	526a318032	[unit test] Refactored test cases with component func (#339 ) * refactored test with component func * fixed bug	2022-03-11 15:50:28 +08:00
Jiarui Fang	cb34cd384d	[test] polish zero related unitest (#351 )	2022-03-11 15:50:28 +08:00
ver217	d0ae0f2215	[zero] update sharded optim v2 (#334 )	2022-03-11 15:50:28 +08:00
jiaruifang	7977422aeb	add bert for unitest and sharded model is not able to pass the bert case	2022-03-11 15:50:28 +08:00
ver217	1388671699	[zero] Update sharded model v2 using sharded param v2 (#323 )	2022-03-11 15:50:28 +08:00
ver217	36f9a74ab2	fix sharded param hook and unit test	2022-03-11 15:50:28 +08:00
ver217	7aef75ca42	[zero] add sharded grad and refactor grad hooks for ShardedModel (#287 )	2022-03-11 15:50:28 +08:00
Jiarui Fang	5a560a060a	Feature/zero (#279 ) * add zero1 (#209) * add zero1 * add test zero1 * update zero stage 1 develop (#212) * Implement naive zero3 (#240) * naive zero3 works well * add zero3 param manager * add TODOs in comments * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * fix bugs of hook and add unit tests (#252) * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * polish code and add state dict hook * fix bug * update unit test * refactor reconstructed zero code * clip_grad support zero3 and add unit test * add unit test for Zero3ParameterManager * [WIP] initialize the shard param class * [WIP] Yet another sharded model implementation (#274) * [WIP] initialize the shard param class * [WIP] Yes another implementation of shardModel. Using a better hook method. * torch.concat -> torch.cat * fix test_zero_level_1.py::test_zero_level_1 unitest * remove deepspeed implementation and refactor for the reconstructed zero module * polish zero dp unittests Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com>	2022-03-11 15:50:28 +08:00

16 Commits (84fd7c1d4deaa245d97c78e18ba9559830feb985)