Commit Graph

5 Commits (4e9989344d20e3f8af44767f0eadeaab5fff8c00)

Author SHA1 Message Date
Hakjin Lee 46c009dba4
[format] Run lint on colossalai.engine (#3367) 2023-04-05 23:24:43 +08:00
ver217 7c70bfbefa
[hotfix] fix PipelineSharedModuleGradientHandler (#1314) 2022-07-14 17:31:13 +08:00
Frank Lee 11f54c7b6b
[doc] improved docstring and assertion messages for the engine module (#871) 2022-04-26 10:00:18 +08:00
ver217 8d3250d74b
[zero] ZeRO supports pipeline parallel (#477) 2022-03-21 16:55:37 +08:00
ver217 96780e6ee4
Optimize pipeline schedule (#94)
* add pipeline shared module wrapper and update load batch

* added model parallel process group for amp and clip grad (#86)

* added model parallel process group for amp and clip grad

* update amp and clip with model parallel process group

* remove pipeline_prev/next group (#88)

* micro batch offload

* optimize pipeline gpu memory usage

* pipeline can receive tensor shape (#93)

* optimize pipeline gpu memory usage

* fix grad accumulation step counter

* rename classes and functions

Co-authored-by: Frank Lee <somerlee.9@gmail.com>
2021-12-30 15:56:46 +08:00