ColossalAI

Commit Graph

Author	SHA1	Message	Date
Jiarui Fang	ab962b9735	[gemini] a new tensor structure (#818 ) * Revert "[zero] add ZeroTensorShardStrategy (#793)" This reverts commit `88759e289e`. * [gemini] set cpu memory capacity * [log] local throughput collecting * polish * polish * polish * polish code * polish * polish code * add a new tensor structure and override linear for it * polish * polish * polish * polish * polish * polish * polish * polish * polish * polish * polish	3 years ago
Jiarui Fang	e761ad2cd7	Revert "[zero] add ZeroTensorShardStrategy (#793 )" (#806 )	3 years ago
HELSON	88759e289e	[zero] add ZeroTensorShardStrategy (#793 )	3 years ago
Jiarui Fang	681addb512	[refactor] moving grad acc logic to engine (#804 )	3 years ago
Jiarui Fang	4d9332b4c5	[refactor] moving memtracer to gemini (#801 )	3 years ago
HELSON	4c4388c46e	[hotfix] fix memory leak in zero (#781 )	3 years ago
Frank Lee	5a1a095b92	[test] refactored with the new rerun decorator (#763 ) * [test] refactored with the new rerun decorator * polish test case	3 years ago
Jiarui Fang	10ef8afdd2	[gemini] init genimi individual directory (#754 )	3 years ago
ver217	dcca614eee	[hotfix] fix test_stateful_tensor_mgr (#762 )	3 years ago
ver217	a93a7d7364	[hotfix] fix reuse_fp16_shard of sharded model (#756 ) * fix reuse_fp16_shard * disable test stm * polish code	3 years ago
HELSON	84c6700b2a	[zero] refactor memstats_collector (#746 )	3 years ago
ver217	e396bb71f2	[zero] add tensor placement policies (#743 ) * add tensor placement policies * polish comments * polish comments * update moe unit tests	3 years ago
HELSON	22c4b88d56	[zero] refactor ShardedParamV2 for convenience (#742 )	3 years ago
Frank Lee	f4f42d4c3c	[bug] fixed DDP compatibility with torch 1.8 (#739 )	3 years ago
Jiarui Fang	53cb584808	[utils] correct cpu memory used and capacity in the context of multi-process (#726 )	3 years ago
HELSON	b9b469ea50	[moe] add checkpoint for moe zero test (#729 )	3 years ago
FrankLeeeee	e88a498c9c	[test] removed trivial outdated test	3 years ago
FrankLeeeee	62b4ce7326	[test] added missing decorators to model checkpointing tests	3 years ago
Jiarui Fang	4d90a7b513	[refactor] zero directory (#724 )	3 years ago
Frank Lee	20ab1f5520	[bug] fixed broken test_found_inf (#725 )	3 years ago
Jiarui Fang	193dc8dacb	[refactor] refactor the memory utils (#715 )	3 years ago
HELSON	dbd96fe90a	[zero] check whether gradients have inf and nan in gpu (#712 )	3 years ago
HELSON	a9b8300d54	[zero] improve adaptability for not-shard parameters (#708 ) * adapt post grad hooks for not-shard parameters * adapt optimizer for not-shard parameters * offload gradients for not-replicated parameters	3 years ago
ver217	ab8c6b4a0e	[zero] refactor memstats collector (#706 ) * refactor memstats collector * fix disposable * polish code	3 years ago
HELSON	ee112fe1da	[zero] adapt zero hooks for unsharded module (#699 )	3 years ago
ver217	3c9cd5bb5e	[zero] stateful tensor manager (#687 ) * [WIP] stateful tensor manager * add eviction strategy * polish code * polish code * polish comment * add unit test * fix sampler bug * polish code * fix max sampling cnt resetting bug * fix sampler bug * polish code * fix bug * fix unit test Co-authored-by: jiaruifang <fangjiarui123@gmail.com>	3 years ago
HELSON	d7ecaf362b	[zero] fix init bugs in zero context (#686 ) * adapt model weight initialization for methods in Pytorch nn.init	3 years ago
Jiarui Fang	0aab52301e	[hotfix] fix a bug in model data stats tracing (#655 )	3 years ago
YuliangLiu0306	ade05a5d83	[refactor] pipeline, put runtime schedule into engine. (#627 )	3 years ago
HELSON	e5d615aeee	[hotfix] fix bugs in testing (#659 ) * remove hybrid adam in test_moe_zero_optim * fix activation checkpointing and its unitest	3 years ago
HELSON	b31daed4cf	fix bugs in CPU adam (#633 ) * add cpu adam counter for all cpu adam * fixed updating error in adam kernel	3 years ago
HELSON	055fbf5be6	[zero] adapt zero for unsharded paramters (Optimizer part) (#601 )	3 years ago
アマデウス	354b7954d1	[model checkpoint] added unit tests for checkpoint save/load (#599 )	3 years ago
FredHuang99	93f14d2a33	[zero] test zero tensor utils (#609 )	3 years ago
Jiarui Fang	e956d93ac2	[refactor] memory utils (#577 )	3 years ago
HELSON	e6d50ec107	[zero] adapt zero for unsharded parameters (#561 ) * support existing sharded and unsharded parameters in zero * add unitest for moe-zero model init * polish moe gradient handler	3 years ago
ver217	7c6c427db1	[zero] trace states of fp16/32 grad and fp32 param (#571 )	3 years ago
Jiarui Fang	7675366fce	[polish] rename col_attr -> colo_attr (#558 )	3 years ago
ver217	014bac0c49	[zero] hijack p.grad in sharded model (#554 ) * hijack p.grad in sharded model * polish comments * polish comments	3 years ago
Jiarui Fang	f552b11294	[zero] label state for param fp16 and grad (#551 )	3 years ago
Jiarui Fang	214da761d4	[zero] add stateful tensor (#549 )	3 years ago
HELSON	8c90d4df54	[zero] add zero context manager to change config during initialization (#546 )	3 years ago
Liang Bowen	ec5086c49c	Refactored docstring to google style	3 years ago
Jiarui Fang	53b1b6e340	[zero] non model data tracing (#545 )	3 years ago
ver217	1f90a3b129	[zero] polish ZeroInitContext (#540 )	3 years ago
Jiarui Fang	c11ff81b15	[zero] get memory usage of sharded optim v2. (#542 )	3 years ago
HELSON	a30e2b4c24	[zero] adapt for no-leaf module in zero (#535 ) only process module's own parameters in Zero context add zero hooks for all modules that contrain parameters gather parameters only belonging to module itself	3 years ago
Jiarui Fang	705f56107c	[zero] refactor model data tracing (#537 )	3 years ago
Jiarui Fang	a590ed0ba3	[zero] improve the accuracy of get_memory_usage of sharded param (#538 )	3 years ago
Jiarui Fang	37cb70feec	[zero] get memory usage for sharded param (#536 )	3 years ago

... 15 16 17 18 19 ...

955 Commits (39f2582e987871c198f2f2526cd4435cbd569741)