88 Commits (92f4224867581e741d5a673b6b807b644127fbf7)

Author	SHA1	Message	Date
Jiarui Fang	214da761d4	[zero] add stateful tensor (#549)	3 years ago
Jiarui Fang	107b99ddb1	[zero] dump memory stats for sharded model (#548)	3 years ago
HELSON	8c90d4df54	[zero] add zero context manager to change config during initialization (#546)	3 years ago
Jiarui Fang	53b1b6e340	[zero] non model data tracing (#545)	3 years ago
ver217	fb841dd5c5	[zero] optimize grad offload (#539) ... * optimize grad offload * polish code * polish code	3 years ago
ver217	1f90a3b129	[zero] polish ZeroInitContext (#540)	3 years ago
Jiarui Fang	c11ff81b15	[zero] get memory usage of sharded optim v2. (#542)	3 years ago
HELSON	a30e2b4c24	[zero] adapt for no-leaf module in zero (#535) ... only process module's own parameters in Zero context add zero hooks for all modules that contrain parameters gather parameters only belonging to module itself	3 years ago
Jiarui Fang	705f56107c	[zero] refactor model data tracing (#537)	3 years ago
Jiarui Fang	a590ed0ba3	[zero] improve the accuracy of get_memory_usage of sharded param (#538)	3 years ago
Jiarui Fang	37cb70feec	[zero] get memory usage for sharded param (#536)	3 years ago
Jiarui Fang	05e33b2578	[zero] fix grad offload (#528) ... * [zero] fix grad offload * polish code	3 years ago
Jiarui Fang	8d8c5407c0	[zero] refactor model data tracing (#522)	3 years ago
Jiarui Fang	4d322b79da	[refactor] remove old zero code (#517)	3 years ago
Jiarui Fang	920c5889a7	[zero] add colo move inline (#521)	3 years ago
Jiarui Fang	0bebda6ea5	[zero] fix init device bug in zero init context unittest (#516)	3 years ago
Jiarui Fang	7ef3507ace	[zero] show model data cuda memory usage after zero context init. (#515)	3 years ago
ver217	a2e61d61d4	[zero] zero init ctx enable rm_torch_payload_on_the_fly (#512) ... * enable rm_torch_payload_on_the_fly * polish docstr	3 years ago
Jiarui Fang	bca0c49a9d	[zero] use colo model data api in optimv2 (#511)	3 years ago
Jiarui Fang	0035b7be07	[memory] add model data tensor moving api (#503)	3 years ago
ver217	9ec1ce6ab1	[zero] sharded model support the reuse of fp16 shard (#495) ... * sharded model supports reuse fp16 shard * rename variable * polish code * polish code * polish code	3 years ago
ver217	c4c02424f3	[zero] sharded model manages ophooks individually (#492)	3 years ago
ver217	a9ecb4b244	[zero] polish sharded optimizer v2 (#490)	3 years ago
ver217	62b0a8d644	[zero] sharded optim support hybrid cpu adam (#486) ... * sharded optim support hybrid cpu adam * update unit test * polish docstring	3 years ago
Jiarui Fang	b334822163	[zero] polish sharded param name (#484) ... * [zero] polish sharded param name * polish code * polish * polish code * polish * polsih * polish	3 years ago
ver217	8d3250d74b	[zero] ZeRO supports pipeline parallel (#477)	3 years ago
ver217	3cb3fc275e	zero init ctx receives a dp process group (#471)	3 years ago
ver217	fc8e6db005	[doc] Update docstring for ZeRO (#459) ... * polish sharded model docstr * polish sharded optim docstr * polish zero docstr * polish shard strategy docstr	3 years ago
ver217	a241f61b34	[zero] Update initialize for ZeRO (#458) ... * polish code * shard strategy receive pg in shard() / gather() * update zero engine * polish code	3 years ago
ver217	642846d6f9	update sharded optim and fix zero init ctx (#457)	3 years ago
Jiarui Fang	e2e9f82588	Revert "[zero] update sharded optim and fix zero init ctx" (#456) ... * Revert "polish code" This reverts commit 8cf7ff08cf. * Revert "rename variables" This reverts commit e99af94ab8. * Revert "remove surplus imports" This reverts commit 46add4a5c5. * Revert "update sharded optim and fix zero init ctx" This reverts commit 57567ee768.	3 years ago
ver217	e99af94ab8	rename variables	3 years ago
ver217	57567ee768	update sharded optim and fix zero init ctx	3 years ago
Jiarui Fang	0fcfb1e00d	[test] make zero engine test really work (#447)	3 years ago
Jiarui Fang	237d08e7ee	[zero] hybrid cpu adam (#445)	3 years ago
Jiarui Fang	496cbb0760	[hotfix] fix initialize bug with zero (#442)	3 years ago
Jiarui Fang	640a6cd304	[refactory] refactory the initialize method for new zero design (#431)	3 years ago
ver217	fce9432f08	sync before creating empty grad	3 years ago
ver217	ea6905a898	free param.grad	3 years ago
ver217	9506a8beb2	use double buffer to handle grad	3 years ago
Jiarui Fang	adebb3e041	[zero] cuda margin space for OS (#418)	3 years ago
Jiarui Fang	56bb412e72	[polish] use GLOBAL_MODEL_DATA_TRACER (#417)	3 years ago
Jiarui Fang	23ba3fc450	[zero] refactory ShardedOptimV2 init method (#416)	3 years ago
Frank Lee	e79ea44247	[fp16] refactored fp16 optimizer (#392)	3 years ago
Jiarui Fang	21dc54e019	[zero] memtracer to record cuda memory usage of model data and overall system (#395)	3 years ago
Jiarui Fang	370f567e7d	[zero] new interface for ShardedOptimv2 (#406)	3 years ago
ver217	63469c0f91	polish code	3 years ago
ver217	88804aee49	add bucket tensor shard strategy	3 years ago
HELSON	7c079d9c33	[hotfix] fixed bugs in ShardStrategy and PcieProfiler (#394)	3 years ago
Jiarui Fang	3af13a2c3e	[zero] polish ShardedOptimV2 unittest (#385) ... * place params on cpu after zero init context * polish code * bucketzed cpu gpu tensor transter * find a bug in sharded optim unittest * add offload unittest for ShardedOptimV2. * polish code and make it more robust	3 years ago