ver217
0ef8819c67
polish docstring of zero ( #612 )
2022-04-01 14:50:56 +08:00
ver217
9bee119104
[hotfix] fix sharded optim zero grad ( #604 )
...
* fix sharded optim zero grad
* polish comments
2022-04-01 12:41:20 +08:00
Jiarui Fang
e956d93ac2
[refactor] memory utils ( #577 )
2022-04-01 09:22:33 +08:00
ver217
7c6c427db1
[zero] trace states of fp16/32 grad and fp32 param ( #571 )
2022-03-31 16:26:54 +08:00
Jiarui Fang
7675366fce
[polish] rename col_attr -> colo_attr ( #558 )
2022-03-31 12:25:45 +08:00
ver217
014bac0c49
[zero] hijack p.grad in sharded model ( #554 )
...
* hijack p.grad in sharded model
* polish comments
* polish comments
2022-03-30 18:14:50 +08:00
Jiarui Fang
f552b11294
[zero] label state for param fp16 and grad ( #551 )
2022-03-30 15:57:46 +08:00
Jiarui Fang
107b99ddb1
[zero] dump memory stats for sharded model ( #548 )
2022-03-30 09:38:44 +08:00
Jiarui Fang
53b1b6e340
[zero] non model data tracing ( #545 )
2022-03-29 15:45:48 +08:00
ver217
fb841dd5c5
[zero] optimize grad offload ( #539 )
...
* optimize grad offload
* polish code
* polish code
2022-03-29 12:48:00 +08:00
Jiarui Fang
c11ff81b15
[zero] get memory usage of sharded optim v2. ( #542 )
2022-03-29 09:08:18 +08:00
Jiarui Fang
705f56107c
[zero] refactor model data tracing ( #537 )
2022-03-28 16:38:18 +08:00
Jiarui Fang
05e33b2578
[zero] fix grad offload ( #528 )
...
* [zero] fix grad offload
* polish code
2022-03-25 18:23:25 +08:00
Jiarui Fang
4d322b79da
[refactor] remove old zero code ( #517 )
2022-03-25 14:54:39 +08:00
Jiarui Fang
bca0c49a9d
[zero] use colo model data api in optimv2 ( #511 )
2022-03-24 17:19:34 +08:00
Jiarui Fang
0035b7be07
[memory] add model data tensor moving api ( #503 )
2022-03-24 14:29:41 +08:00
ver217
9ec1ce6ab1
[zero] sharded model support the reuse of fp16 shard ( #495 )
...
* sharded model supports reuse fp16 shard
* rename variable
* polish code
* polish code
* polish code
2022-03-23 14:59:59 +08:00
ver217
a9ecb4b244
[zero] polish sharded optimizer v2 ( #490 )
2022-03-22 15:53:48 +08:00
ver217
62b0a8d644
[zero] sharded optim support hybrid cpu adam ( #486 )
...
* sharded optim support hybrid cpu adam
* update unit test
* polish docstring
2022-03-22 14:56:59 +08:00
Jiarui Fang
b334822163
[zero] polish sharded param name ( #484 )
...
* [zero] polish sharded param name
* polish code
* polish
* polish code
* polish
* polsih
* polish
2022-03-22 14:36:16 +08:00
ver217
fc8e6db005
[doc] Update docstring for ZeRO ( #459 )
...
* polish sharded model docstr
* polish sharded optim docstr
* polish zero docstr
* polish shard strategy docstr
2022-03-18 16:48:20 +08:00
ver217
a241f61b34
[zero] Update initialize for ZeRO ( #458 )
...
* polish code
* shard strategy receive pg in shard() / gather()
* update zero engine
* polish code
2022-03-18 16:18:31 +08:00
ver217
642846d6f9
update sharded optim and fix zero init ctx ( #457 )
2022-03-18 15:44:47 +08:00
Jiarui Fang
e2e9f82588
Revert "[zero] update sharded optim and fix zero init ctx" ( #456 )
...
* Revert "polish code"
This reverts commit 8cf7ff08cf
.
* Revert "rename variables"
This reverts commit e99af94ab8
.
* Revert "remove surplus imports"
This reverts commit 46add4a5c5
.
* Revert "update sharded optim and fix zero init ctx"
This reverts commit 57567ee768
.
2022-03-18 15:22:43 +08:00
ver217
e99af94ab8
rename variables
2022-03-18 14:25:25 +08:00
ver217
57567ee768
update sharded optim and fix zero init ctx
2022-03-18 14:25:25 +08:00
Jiarui Fang
0fcfb1e00d
[test] make zero engine test really work ( #447 )
2022-03-17 17:24:25 +08:00
Jiarui Fang
640a6cd304
[refactory] refactory the initialize method for new zero design ( #431 )
2022-03-16 19:29:37 +08:00
Jiarui Fang
23ba3fc450
[zero] refactory ShardedOptimV2 init method ( #416 )
2022-03-15 10:45:55 +08:00
Frank Lee
e79ea44247
[fp16] refactored fp16 optimizer ( #392 )
2022-03-15 10:05:38 +08:00
Jiarui Fang
21dc54e019
[zero] memtracer to record cuda memory usage of model data and overall system ( #395 )
2022-03-14 22:05:30 +08:00
Jiarui Fang
370f567e7d
[zero] new interface for ShardedOptimv2 ( #406 )
2022-03-14 20:48:41 +08:00
Jiarui Fang
3af13a2c3e
[zero] polish ShardedOptimV2 unittest ( #385 )
...
* place params on cpu after zero init context
* polish code
* bucketzed cpu gpu tensor transter
* find a bug in sharded optim unittest
* add offload unittest for ShardedOptimV2.
* polish code and make it more robust
2022-03-11 15:50:28 +08:00
Jiarui Fang
b5f43acee3
[zero] find miss code ( #378 )
2022-03-11 15:50:28 +08:00
jiaruifang
d9217e1960
Revert "[zero] bucketized tensor cpu gpu copy ( #368 )"
...
This reverts commit bef05489b6
.
2022-03-11 15:50:28 +08:00
Jiarui Fang
00670c870e
[zero] bucketized tensor cpu gpu copy ( #368 )
2022-03-11 15:50:28 +08:00
ver217
d0ae0f2215
[zero] update sharded optim v2 ( #334 )
2022-03-11 15:50:28 +08:00
ver217
3092317b80
polish code
2022-03-11 15:50:28 +08:00
ver217
36f9a74ab2
fix sharded param hook and unit test
2022-03-11 15:50:28 +08:00
ver217
001ca624dd
impl shard optim v2 and add unit test
2022-03-11 15:50:28 +08:00
ver217
b105371ace
rename shared adam to sharded optim v2
2022-03-11 15:50:28 +08:00