ColossalAI

Commit Graph

Author	SHA1	Message	Date
Boyuan Yao	8e3f66a0d1	[zero] fix wrong import (#2777 )	2023-02-17 10:26:07 +08:00
Nikita Shulga	01066152f1	Don't use `torch._six` (#2775 ) * Don't use `torch._six` This is a private API which is gone after https://github.com/pytorch/pytorch/pull/94709 * Update common.py	2023-02-17 09:22:45 +08:00
HELSON	077a5cdde4	[zero] fix gradient clipping in hybrid parallelism (#2521 ) * [zero] fix gradient clipping in hybrid parallelism * [testing] change model name to avoid pytest warning * [hotfix] fix unit testing	2023-01-29 15:09:57 +08:00
HELSON	a5dc4253c6	[zero] polish low level optimizer (#2473 )	2023-01-13 14:56:17 +08:00
Jiarui Fang	867c8c2d3a	[zero] low level optim supports ProcessGroup (#2464 )	2023-01-13 10:05:58 +08:00
HELSON	a1ce02d740	[zero] test gradient accumulation (#1964 ) * [zero] fix memory leak for zero2 * [zero] test gradient accumulation * [zero] remove grad clip test	2022-11-29 13:00:30 +08:00
Jiarui Fang	4d90a7b513	[refactor] zero directory (#724 )	2022-04-11 23:13:02 +08:00
Jiarui Fang	5a560a060a	Feature/zero (#279 ) * add zero1 (#209) * add zero1 * add test zero1 * update zero stage 1 develop (#212) * Implement naive zero3 (#240) * naive zero3 works well * add zero3 param manager * add TODOs in comments * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * fix bugs of hook and add unit tests (#252) * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * polish code and add state dict hook * fix bug * update unit test * refactor reconstructed zero code * clip_grad support zero3 and add unit test * add unit test for Zero3ParameterManager * [WIP] initialize the shard param class * [WIP] Yet another sharded model implementation (#274) * [WIP] initialize the shard param class * [WIP] Yes another implementation of shardModel. Using a better hook method. * torch.concat -> torch.cat * fix test_zero_level_1.py::test_zero_level_1 unitest * remove deepspeed implementation and refactor for the reconstructed zero module * polish zero dp unittests Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com>	2022-03-11 15:50:28 +08:00

8 Commits (dbd0fd1522c8b779a6d26da3c4cce52442ce494d)