ColossalAI

Commit Graph

Author	SHA1	Message	Date
HELSON	a30e2b4c24	[zero] adapt for no-leaf module in zero (#535 ) only process module's own parameters in Zero context add zero hooks for all modules that contrain parameters gather parameters only belonging to module itself	2022-03-28 17:42:18 +08:00
Jiarui Fang	705f56107c	[zero] refactor model data tracing (#537 )	2022-03-28 16:38:18 +08:00
Jiarui Fang	920c5889a7	[zero] add colo move inline (#521 )	2022-03-25 14:02:55 +08:00
Jiarui Fang	b334822163	[zero] polish sharded param name (#484 ) * [zero] polish sharded param name * polish code * polish * polish code * polish * polsih * polish	2022-03-22 14:36:16 +08:00
ver217	a241f61b34	[zero] Update initialize for ZeRO (#458 ) * polish code * shard strategy receive pg in shard() / gather() * update zero engine * polish code	2022-03-18 16:18:31 +08:00
ver217	9506a8beb2	use double buffer to handle grad	2022-03-16 14:24:09 +08:00
Jiarui Fang	56bb412e72	[polish] use GLOBAL_MODEL_DATA_TRACER (#417 )	2022-03-15 11:29:46 +08:00
Jiarui Fang	21dc54e019	[zero] memtracer to record cuda memory usage of model data and overall system (#395 )	2022-03-14 22:05:30 +08:00
ver217	88804aee49	add bucket tensor shard strategy	2022-03-14 14:48:32 +08:00
yuxuan-lou	3b88eb2259	Flake8 code restyle	2022-03-11 15:50:28 +08:00
Jiarui Fang	44e4891f57	[zero] able to place params on cpu after zero init context (#365 ) * place params on cpu after zero init context * polish code	2022-03-11 15:50:28 +08:00
Jiarui Fang	10e2826426	move async memory to an individual directory (#345 )	2022-03-11 15:50:28 +08:00
Jie Zhu	3213554cc2	[profiler] add adaptive sampling to memory profiler (#330 ) * fix merge conflict modify unit test remove unnessesary log info reformat file * remove unused module * remove unnecessary sync function * change doc string style from Google to Sphinx	2022-03-11 15:50:28 +08:00
ver217	1388671699	[zero] Update sharded model v2 using sharded param v2 (#323 )	2022-03-11 15:50:28 +08:00
Jiarui Fang	11bddb6e55	[zero] update zero context init with the updated test utils (#327 )	2022-03-11 15:50:28 +08:00
ver217	36f9a74ab2	fix sharded param hook and unit test	2022-03-11 15:50:28 +08:00
ver217	001ca624dd	impl shard optim v2 and add unit test	2022-03-11 15:50:28 +08:00
Jie Zhu	d344689274	[profiler] primary memory tracer	2022-03-11 15:50:28 +08:00
ver217	7aef75ca42	[zero] add sharded grad and refactor grad hooks for ShardedModel (#287 )	2022-03-11 15:50:28 +08:00
Jiarui Fang	5a560a060a	Feature/zero (#279 ) * add zero1 (#209) * add zero1 * add test zero1 * update zero stage 1 develop (#212) * Implement naive zero3 (#240) * naive zero3 works well * add zero3 param manager * add TODOs in comments * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * fix bugs of hook and add unit tests (#252) * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * polish code and add state dict hook * fix bug * update unit test * refactor reconstructed zero code * clip_grad support zero3 and add unit test * add unit test for Zero3ParameterManager * [WIP] initialize the shard param class * [WIP] Yet another sharded model implementation (#274) * [WIP] initialize the shard param class * [WIP] Yes another implementation of shardModel. Using a better hook method. * torch.concat -> torch.cat * fix test_zero_level_1.py::test_zero_level_1 unitest * remove deepspeed implementation and refactor for the reconstructed zero module * polish zero dp unittests Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com>	2022-03-11 15:50:28 +08:00
Jiarui Fang	569357fea0	add pytorch hooks (#179 ) * add pytorch hooks fix #175 * remove licenses in src code * add gpu memory tracer * replacing print with logger in ophooks.	2022-01-25 22:20:54 +08:00

21 Commits (fb841dd5c5d23acfbd5543c6d4461c988167477e)