github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
2,723 Commits
32 Branches
46 Tags
228 MiB
Commit Graph

19 Commits (a39a5c66fe943741af65df5adf18093692045dab)

Author SHA1 Message Date
Hongxin Liu a39a5c66fe
Merge branch 'main' into feature/shardformer 2023-09-04 23:43:13 +08:00
Hongxin Liu 63ecafb1fb
[checkpointio] optimize zero optim checkpoint io (#4591) 
* [zero] update checkpoint io to save memory

* [checkpointio] add device map to save memory
 2023-09-04 11:26:45 +08:00
LuGY cbac782254
[zero]fix zero ckptIO with offload (#4529) 
* fix zero ckptio with offload

* fix load device

* saved tensors in ckpt should be on CPU

* fix unit test

* fix unit test

* add clear cache

* save memory for CI
 2023-09-01 17:41:19 +08:00
flybird11111 ec18fc7340
[shardformer] support pp+tp+zero1 tests (#4531) 
* [shardformer] fix opt test hanging

* fix

* test

* test

* test

* fix test

* fix test

* remove print

* add fix

* [shardformer] pp+tp+zero1

[shardformer] pp+tp+zero1

[shardformer] pp+tp+zero1

[shardformer] pp+tp+zero1

[shardformer] pp+tp+zero1

[shardformer] pp+tp+zero1

* [shardformer] pp+tp+zero1

* [shardformer] pp+tp+zero1

* [shardformer] pp+tp+zero1

* [shardformer] pp+tp+zero1
 2023-08-30 21:29:18 +08:00
Jianghai 376533a564
[shardformer] zero1+pp and the corresponding tests (#4517) 
* pause

* finish pp+zero1

* Update test_shard_vit.py
 2023-08-28 10:51:16 +08:00
LuGY 839847b7d7
[zero]support zero2 with gradient accumulation (#4511) 
* support gradient accumulation with zero2

* fix type
 2023-08-25 13:44:07 +08:00
LuGY d86ddd9b29
[hotfix] fix unsafe async comm in zero (#4404) 
* improve stablility of zero

* fix wrong index

* add record stream
 2023-08-11 15:09:24 +08:00
LuGY 45b08f08cb [zero] optimize the optimizer step time (#4221) 
* optimize the optimizer step time

* fix corner case

* polish

* replace all-reduce with all-gather

* set comm device to cuda
 2023-07-31 22:13:29 +08:00
LuGY 1a49a5ea00 [zero] support shard optimizer state dict of zero (#4194) 
* support shard optimizer of zero

* polish code

* support sync grad manually
 2023-07-31 22:13:29 +08:00
LuGY dd7cc58299 [zero] add state dict for low level zero (#4179) 
* add state dict for zero

* fix unit test

* polish
 2023-07-31 22:13:29 +08:00
LuGY c668801d36 [zero] allow passing process group to zero12 (#4153) 
* allow passing process group to zero12

* union tp-zero and normal-zero

* polish code
 2023-07-31 22:13:29 +08:00
LuGY 79cf1b5f33 [zero]support no_sync method for zero1 plugin (#4138) 
* support no sync for zero1 plugin

* polish

* polish
 2023-07-31 22:13:29 +08:00
LuGY c6ab96983a [zero] refactor low level zero for shard evenly (#4030) 
* refactor low level zero

* fix zero2 and support cpu offload

* avg gradient and modify unit test

* refactor grad store, support layer drop

* refactor bucket store, support grad accumulation

* fix and update unit test of zero and ddp

* compatible with tp, ga and unit test

* fix memory leak and polish

* add zero layer drop unittest

* polish code

* fix import err in unit test

* support diffenert comm dtype, modify docstring style

* polish code

* test padding and fix

* fix unit test of low level zero

* fix pad recording in bucket store

* support some models

* polish
 2023-07-31 22:13:29 +08:00
digger yu de0d7df33f
[nfc] fix typo colossalai/zero (#3923) 2023-06-08 00:01:29 +08:00
Hongxin Liu ae02d4e4f7
[bf16] add bf16 support (#3882) 
* [bf16] add bf16 support for fused adam (#3844)

* [bf16] fused adam kernel support bf16

* [test] update fused adam kernel test

* [test] update fused adam test

* [bf16] cpu adam and hybrid adam optimizers support bf16 (#3860)

* [bf16] implement mixed precision mixin and add bf16 support for low level zero (#3869)

* [bf16] add mixed precision mixin

* [bf16] low level zero optim support bf16

* [text] update low level zero test

* [text] fix low level zero grad acc test

* [bf16] add bf16 support for gemini (#3872)

* [bf16] gemini support bf16

* [test] update gemini bf16 test

* [doc] update gemini docstring

* [bf16] add bf16 support for plugins (#3877)

* [bf16] add bf16 support for legacy zero (#3879)

* [zero] init context support bf16

* [zero] legacy zero support bf16

* [test] add zero bf16 test

* [doc] add bf16 related docstring for legacy zero
 2023-06-05 15:58:31 +08:00
YH a22407cc02
[zero] Suggests a minor change to confusing variable names in the ZeRO optimizer. (#3173) 
* Fix confusing variable name in zero opt

* Apply lint

* Fix util func

* Fix minor util func

* Fix zero param optimizer name
 2023-04-27 18:43:14 +08:00
Hongxin Liu 4b3240cb59
[booster] add low level zero plugin (#3594) 
* [booster] add low level zero plugin

* [booster] fix gemini plugin test

* [booster] fix precision

* [booster] add low level zero plugin test

* [test] fix booster plugin test oom

* [test] fix booster plugin test oom

* [test] fix googlenet and inception output trans

* [test] fix diffuser clip vision model

* [test] fix torchaudio_wav2vec2_base

* [test] fix low level zero plugin test
 2023-04-26 14:37:25 +08:00
Hongxin Liu 173dad0562
[misc] add verbose arg for zero and op builder (#3552) 
* [misc] add print verbose

* [gemini] add print verbose

* [zero] add print verbose for low level

* [misc] add print verbose for op builder
 2023-04-17 11:25:35 +08:00
ver217 26b7aac0be
[zero] reorganize zero/gemini folder structure (#3424) 
* [zero] refactor low-level zero folder structure

* [zero] fix legacy zero import path

* [zero] fix legacy zero import path

* [zero] remove useless import

* [zero] refactor gemini folder structure

* [zero] refactor gemini folder structure

* [zero] refactor legacy zero import path

* [zero] refactor gemini folder structure

* [zero] refactor gemini folder structure

* [zero] refactor gemini folder structure

* [zero] refactor legacy zero import path

* [zero] fix test import path

* [zero] fix test

* [zero] fix circular import

* [zero] update import
 2023-04-04 13:48:16 +08:00