github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
2,577 Commits
32 Branches
46 Tags
228 MiB
Commit Graph

23 Commits (03654c0ce2de82c86367d16f594f692f0655108f)

Author SHA1 Message Date
LuGY c6ab96983a [zero] refactor low level zero for shard evenly (#4030) 
* refactor low level zero

* fix zero2 and support cpu offload

* avg gradient and modify unit test

* refactor grad store, support layer drop

* refactor bucket store, support grad accumulation

* fix and update unit test of zero and ddp

* compatible with tp, ga and unit test

* fix memory leak and polish

* add zero layer drop unittest

* polish code

* fix import err in unit test

* support diffenert comm dtype, modify docstring style

* polish code

* test padding and fix

* fix unit test of low level zero

* fix pad recording in bucket store

* support some models

* polish
 2023-07-31 22:13:29 +08:00
Frank Lee 58df720570 [shardformer] adapted T5 and LLaMa test to use kit (#4049) 
* [shardformer] adapted T5 and LLaMa test to use kit

* polish code
 2023-07-04 16:05:01 +08:00
Hongxin Liu dbb32692d2
[lazy] refactor lazy init (#3891) 
* [lazy] remove old lazy init

* [lazy] refactor lazy init folder structure

* [lazy] fix lazy tensor deepcopy

* [test] update lazy init test
 2023-06-05 14:20:47 +08:00
wukong1992 6b305a99d6
[booster] torch fsdp fix ckpt (#3788) 2023-05-23 16:58:45 +08:00
Hongxin Liu 5452df63c5
[plugin] torch ddp plugin supports sharded model checkpoint (#3775) 
* [plugin] torch ddp plugin add save sharded model

* [test] fix torch ddp ckpt io test

* [test] fix torch ddp ckpt io test

* [test] fix low level zero plugin test

* [test] fix low level zero plugin test

* [test] add debug info

* [test] add debug info

* [test] add debug info

* [test] add debug info

* [test] add debug info

* [test] fix low level zero plugin test

* [test] fix low level zero plugin test

* [test] remove debug info
 2023-05-18 20:05:59 +08:00
wukong1992 6050f37776
[booster] removed models that don't support fsdp (#3744) 
Co-authored-by: 纪少敏 <jishaomin@jishaomindeMBP.lan>
 2023-05-15 19:35:21 +08:00
Hongxin Liu afb239bbf8
[devops] update torch version of CI (#3725) 
* [test] fix flop tensor test

* [test] fix autochunk test

* [test] fix lazyinit test

* [devops] update torch version of CI

* [devops] enable testmon

* [devops] fix ci

* [devops] fix ci

* [test] fix checkpoint io test

* [test] fix cluster test

* [test] fix timm test

* [devops] fix ci

* [devops] fix ci

* [devops] fix ci

* [devops] fix ci

* [devops] force sync to test ci

* [test] skip fsdp test
 2023-05-15 17:20:56 +08:00
wukong1992 b37797ed3d
[booster] support torch fsdp plugin in booster (#3697) 
Co-authored-by: 纪少敏 <jishaomin@jishaomindeMBP.lan>
 2023-05-15 12:14:38 +08:00
digger-yu 1f73609adb
[CI] fix typo with tests/ etc. (#3727) 
* fix spelling error with examples/comminity/

* fix spelling error with tests/

* fix some spelling error with tests/ colossalai/ etc.

* fix spelling error with tests/ etc. date:2023.5.10
 2023-05-11 16:30:58 +08:00
Hongxin Liu 6552cbf8e1
[booster] fix no_sync method (#3709) 
* [booster] fix no_sync method

* [booster] add test for ddp no_sync

* [booster] fix merge

* [booster] update unit test

* [booster] update unit test

* [booster] update unit test
 2023-05-09 11:10:02 +08:00
Hongxin Liu 3bf09efe74
[booster] update prepare dataloader method for plugin (#3706) 
* [booster] add prepare dataloader method for plug

* [booster] update examples and docstr
 2023-05-08 15:44:03 +08:00
Hongxin Liu d0915f54f4
[booster] refactor all dp fashion plugins (#3684) 
* [booster] add dp plugin base

* [booster] inherit dp plugin base

* [booster] refactor unit tests
 2023-05-05 19:36:10 +08:00
Hongxin Liu 4b3240cb59
[booster] add low level zero plugin (#3594) 
* [booster] add low level zero plugin

* [booster] fix gemini plugin test

* [booster] fix precision

* [booster] add low level zero plugin test

* [test] fix booster plugin test oom

* [test] fix booster plugin test oom

* [test] fix googlenet and inception output trans

* [test] fix diffuser clip vision model

* [test] fix torchaudio_wav2vec2_base

* [test] fix low level zero plugin test
 2023-04-26 14:37:25 +08:00
Hongxin Liu 152239bbfa
[gemini] gemini supports lazy init (#3379) 
* [gemini] fix nvme optimizer init

* [gemini] gemini supports lazy init

* [gemini] add init example

* [gemini] add fool model

* [zero] update gemini ddp

* [zero] update init example

* add chunk method

* add chunk method

* [lazyinit] fix lazy tensor tolist

* [gemini] fix buffer materialization

* [misc] remove useless file

* [booster] update gemini plugin

* [test] update gemini plugin test

* [test] fix gemini plugin test

* [gemini] fix import

* [gemini] fix import

* [lazyinit] use new metatensor

* [lazyinit] use new metatensor

* [lazyinit] fix __set__ method
 2023-04-12 16:03:25 +08:00
Frank Lee 80eba05b0a
[test] refactor tests with spawn (#3452) 
* [test] added spawn decorator

* polish code

* polish code

* polish code

* polish code

* polish code

* polish code
 2023-04-06 14:51:35 +08:00
Frank Lee 1beb85cc25
[checkpoint] refactored the API and added safetensors support (#3427) 
* [checkpoint] refactored the API and added safetensors support

* polish code
 2023-04-04 15:23:01 +08:00
Frank Lee 638a07a7f9
[test] fixed gemini plugin test (#3411) 
* [test] fixed gemini plugin test

* polish code

* polish code
 2023-04-03 17:12:22 +08:00
ver217 5f2e34e6c9
[booster] implement Gemini plugin (#3352) 
* [booster] add gemini plugin

* [booster] update docstr

* [booster] gemini plugin add coloparam convertor

* [booster] fix coloparam convertor

* [booster] fix gemini plugin device

* [booster] add gemini plugin test

* [booster] gemini plugin ignore sync bn

* [booster] skip some model

* [booster] skip some model

* [booster] modify test world size

* [booster] modify test world size

* [booster] skip test
 2023-03-31 16:06:13 +08:00
Frank Lee 73d3e4d309
[booster] implemented the torch ddd + resnet example (#3232) 
* [booster] implemented the torch ddd + resnet example

* polish code
 2023-03-27 10:24:14 +08:00
Frank Lee e7f3bed2d3
[booster] added the plugin base and torch ddp plugin (#3180) 
* [booster] added the plugin base and torch ddp plugin

* polish code

* polish code

* polish code
 2023-03-21 17:39:30 +08:00
Frank Lee a9b8402d93
[booster] added the accelerator implementation (#3159) 2023-03-20 13:59:24 +08:00
Frank Lee 1ad3a636b1
[test] fixed torchrec model test (#3167) 
* [test] fixed torchrec model test

* polish code

* polish code

* polish code

* polish code

* polish code

* polish code
 2023-03-20 11:40:25 +08:00
Frank Lee ed19290560
[booster] implemented mixed precision class (#3151) 
* [booster] implemented mixed precision class

* polish code
 2023-03-17 11:00:15 +08:00