11 Commits (616ed91ecd6b5eee8e298f0f0a897eabca37284d)

Author	SHA1	Message	Date
HELSON	17a3c685b0	[zero] fix unit-tests (#2039)	2022-11-30 10:40:31 +08:00
HELSON	537e181705	[testing] fix testing models (#2036) ... * [testing] fix testing models * roll back	2022-11-29 13:42:06 +08:00
Jiarui Fang	96134e7be3	[hotfix] add bert test for gemini fwd bwd (#2035)	2022-11-29 11:19:52 +08:00
Jiarui Fang	28aa9a4294	[Gemini] more rigorous unit tests for run_fwd_bwd (#2034)	2022-11-29 09:26:06 +08:00
Jiarui Fang	2e9cbfca12	[Gemini] add unitests to check gemini correctness (#2015)	2022-11-24 16:51:45 +08:00
HELSON	c6a1a62636	[hotfix] fix zero's incompatibility with checkpoint in torch-1.12 (#1786) ... * [hotfix] fix zero's incompatibility with checkpoint in torch-1.12 * [zero] add cpu shard init * [zero] add tiny example test * [colo_tensor] fix bugs for torch-1.11	2022-11-02 16:11:34 +08:00
HELSON	f69f9bf223	[zero] add chunk init function for users (#1729) ... * add chunk manager init function * fix unit tests * add comment * add flush=True	2022-10-18 16:31:22 +08:00
HELSON	1468e4bcfc	[zero] add constant placement policy (#1705) ... * fixes memory leak when paramter is in fp16 in ZeroDDP init. * bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release. * adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.	2022-10-14 17:53:16 +08:00
HELSON	b28991dd0a	[feature] A new ZeRO implementation (#1644)	2022-10-09 09:18:51 +08:00
Jiarui Fang	c5d39215f6	Revert "[feature] new zero implementation (#1623)" (#1643) ... This reverts commit 5be118f405.	2022-09-26 10:06:03 +08:00
HELSON	5be118f405	[feature] new zero implementation (#1623)	2022-09-24 19:58:18 +08:00