ColossalAI

Commit Graph

Author	SHA1	Message	Date
HELSON	2458659919	[zero] fix error for BEiT models (#2169 ) * [zero] fix error for BEiT models * [ColoParameter] add unpack operation for tuple arguments * fix bugs * fix chunkv2 unit testing * add assertion for gradient state	2022-12-26 15:03:54 +08:00
Jiarui Fang	e5aa8333e4	[NFC] update chunk manager API (#2119 )	2022-12-12 16:57:22 +08:00
Jiarui Fang	e99edfcb51	[NFC] polish comments for Chunk class (#2116 )	2022-12-12 15:39:31 +08:00
HELSON	63fbba3c19	[zero] add L2 gradient clipping for ZeRO (#2112 ) * [zero] add L2 gradient clipping * [testing] add MlpModel * [zero] add unit test for grad clipping * fix atol	2022-12-09 18:09:17 +08:00
HELSON	c6a1a62636	[hotfix] fix zero's incompatibility with checkpoint in torch-1.12 (#1786 ) * [hotfix] fix zero's incompatibility with checkpoint in torch-1.12 * [zero] add cpu shard init * [zero] add tiny example test * [colo_tensor] fix bugs for torch-1.11	2022-11-02 16:11:34 +08:00
HELSON	1468e4bcfc	[zero] add constant placement policy (#1705 ) * fixes memory leak when paramter is in fp16 in ZeroDDP init. * bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release. * adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.	2022-10-14 17:53:16 +08:00
HELSON	b28991dd0a	[feature] A new ZeRO implementation (#1644 )	2022-10-09 09:18:51 +08:00
Jiarui Fang	c5d39215f6	Revert "[feature] new zero implementation (#1623 )" (#1643 ) This reverts commit `5be118f405`.	2022-09-26 10:06:03 +08:00
HELSON	5be118f405	[feature] new zero implementation (#1623 )	2022-09-24 19:58:18 +08:00

9 Commits (dbd0fd1522c8b779a6d26da3c4cce52442ce494d)