53 Commits (b72b8445c69f56f6522791585897c595dc89df7d)

Author	SHA1	Message	Date
Jiarui Fang	496cbb0760	[hotfix] fix initialize bug with zero (#442)	3 years ago
Jiarui Fang	640a6cd304	[refactory] refactory the initialize method for new zero design (#431)	3 years ago
ver217	fce9432f08	sync before creating empty grad	3 years ago
ver217	ea6905a898	free param.grad	3 years ago
ver217	9506a8beb2	use double buffer to handle grad	3 years ago
Jiarui Fang	adebb3e041	[zero] cuda margin space for OS (#418)	3 years ago
Jiarui Fang	56bb412e72	[polish] use GLOBAL_MODEL_DATA_TRACER (#417)	3 years ago
Jiarui Fang	23ba3fc450	[zero] refactory ShardedOptimV2 init method (#416)	3 years ago
Frank Lee	e79ea44247	[fp16] refactored fp16 optimizer (#392)	3 years ago
Jiarui Fang	21dc54e019	[zero] memtracer to record cuda memory usage of model data and overall system (#395)	3 years ago
Jiarui Fang	370f567e7d	[zero] new interface for ShardedOptimv2 (#406)	3 years ago
ver217	63469c0f91	polish code	3 years ago
ver217	88804aee49	add bucket tensor shard strategy	3 years ago
HELSON	7c079d9c33	[hotfix] fixed bugs in ShardStrategy and PcieProfiler (#394)	3 years ago
Jiarui Fang	3af13a2c3e	[zero] polish ShardedOptimV2 unittest (#385) ... * place params on cpu after zero init context * polish code * bucketzed cpu gpu tensor transter * find a bug in sharded optim unittest * add offload unittest for ShardedOptimV2. * polish code and make it more robust	3 years ago
Jiarui Fang	272ebfb57d	[bug] shard param during initializing the ShardedModelV2 (#381)	3 years ago
Jiarui Fang	b5f43acee3	[zero] find miss code (#378)	3 years ago
Jiarui Fang	6b6002962a	[zero] zero init context collect numel of model (#375)	3 years ago
jiaruifang	d9217e1960	Revert "[zero] bucketized tensor cpu gpu copy (#368)" ... This reverts commit bef05489b6.	3 years ago
Jiarui Fang	00670c870e	[zero] bucketized tensor cpu gpu copy (#368)	3 years ago
Jiarui Fang	44e4891f57	[zero] able to place params on cpu after zero init context (#365) ... * place params on cpu after zero init context * polish code	3 years ago
ver217	253e54d98a	fix grad shape	3 years ago
Jiarui Fang	ea2872073f	[zero] global model data memory tracer (#360)	3 years ago
Jiarui Fang	cb34cd384d	[test] polish zero related unitest (#351)	3 years ago
ver217	d0ae0f2215	[zero] update sharded optim v2 (#334)	3 years ago
jiaruifang	5663616921	polish code	3 years ago
jiaruifang	7977422aeb	add bert for unitest and sharded model is not able to pass the bert case	3 years ago
ver217	1388671699	[zero] Update sharded model v2 using sharded param v2 (#323)	3 years ago
Jiarui Fang	11bddb6e55	[zero] update zero context init with the updated test utils (#327)	3 years ago
Jiarui Fang	de0468c7a8	[zero] zero init context (#321) ... * add zero init context * add more flags for zero init context fix bug of repeated converting param to ShardedParamV2 * polish code	3 years ago
LuGY	a3269de5c9	[zero] cpu adam kernel (#288) ... * Added CPU Adam * finished the cpu adam * updated the license * delete useless parameters, removed resnet * modified the method off cpu adam unittest * deleted some useless codes * removed useless codes Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com> Co-authored-by: jiaruifang <fangjiarui123@gmail.com>	3 years ago
Jiarui Fang	90d3aef62c	[zero] yet an improved sharded param (#311)	3 years ago
Jiarui Fang	c9e7d9582d	[zero] polish shard strategy (#310) ... * init shard param from shape tuple * add more unitest for shard param * add set_payload method for ShardedParam * [zero] add shareded tensor class * polish code * add shard stratgy * move shard and gather logic to shard strategy from shard tensor. * polish code	3 years ago
ver217	3092317b80	polish code	3 years ago
ver217	36f9a74ab2	fix sharded param hook and unit test	3 years ago
ver217	001ca624dd	impl shard optim v2 and add unit test	3 years ago
Jiarui Fang	74f77e314b	[zero] a shard strategy in granularity of tensor (#307)	3 years ago
Jiarui Fang	80364c7686	[zero] sharded tensor (#305) ... * init shard param from shape tuple * add more unitest for shard param * add set_payload method for ShardedParam * [zero] add shareded tensor class * polish code	3 years ago
ver217	b105371ace	rename shared adam to sharded optim v2	3 years ago
ver217	70814dc22f	fix master params dtype	3 years ago
ver217	795210dd99	add fp32 master params in sharded adam	3 years ago
ver217	a109225bc2	add sharded adam	3 years ago
Jiarui Fang	e17e92c54d	Polish sharded parameter (#297) ... * init shard param from shape tuple * add more unitest for shard param * add more unittests to shareded param	3 years ago
ver217	7aef75ca42	[zero] add sharded grad and refactor grad hooks for ShardedModel (#287)	3 years ago
Frank Lee	9afb5c8b2d	fixed typo in ShardParam (#294)	3 years ago
Frank Lee	e17e54e32a	added buffer sync to naive amp model wrapper (#291)	3 years ago
Jiarui Fang	5a560a060a	Feature/zero (#279) ... * add zero1 (#209) * add zero1 * add test zero1 * update zero stage 1 develop (#212) * Implement naive zero3 (#240) * naive zero3 works well * add zero3 param manager * add TODOs in comments * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * fix bugs of hook and add unit tests (#252) * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * polish code and add state dict hook * fix bug * update unit test * refactor reconstructed zero code * clip_grad support zero3 and add unit test * add unit test for Zero3ParameterManager * [WIP] initialize the shard param class * [WIP] Yet another sharded model implementation (#274) * [WIP] initialize the shard param class * [WIP] Yes another implementation of shardModel. Using a better hook method. * torch.concat -> torch.cat * fix test_zero_level_1.py::test_zero_level_1 unitest * remove deepspeed implementation and refactor for the reconstructed zero module * polish zero dp unittests Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com>	3 years ago
HELSON	0f8c7f9804	Fixed docstring in colossalai (#171)	3 years ago
ver217	9ef05ed1fc	try import deepspeed when using zero (#130)	3 years ago
Frank Lee	91c327cb44	fixed zero level 3 dtype bug (#76)	3 years ago