170 Commits (4898ff8af45a013f13e5fdadf9b240b2d240b3ca)

Author	SHA1	Message	Date
YuliangLiu0306	2053e138a2	[context]use meta tensor to init model lazily. (#1187) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [context]use meta tensor to init model lazily. * polish * make module with device kwargs bypass the normal init. * change unit test to adapt updated context.	2 years ago
YuliangLiu0306	e27645376d	[hotfix]different overflow status lead to communication stuck. (#1175) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [hotfix]fix some bugs caused by refactored schedule. * [hotfix]different overflow statu llead to communication stuck.	2 years ago
Jiarui Fang	4b9bba8116	[ColoTensor] rename APIs and add output_replicate to ComputeSpec (#1168)	2 years ago
Frank Lee	f8eec98ff5	[tensor] fixed non-serializable colo parameter during model checkpointing (#1153)	2 years ago
Frank Lee	73ad05fc8c	[zero] added error message to handle on-the-fly import of torch Module class (#1135) ... * [zero] added error message to handle on-the-fly import of torch Module class * polish code	2 years ago
Frank Lee	2b2dc1c86b	[pipeline] refactor the pipeline module (#1087) ... * [pipeline] refactor the pipeline module * polish code	2 years ago
Frank Lee	bad5d4c0a1	[context] support lazy init of module (#1088) ... * [context] support lazy init of module * polish code	2 years ago
Frank Lee	bfdc5ccb7b	[context] maintain the context object in with statement (#1073)	3 years ago
Jiarui Fang	49832b2344	[refactory] add nn.parallel module (#1068)	3 years ago
Jiarui Fang	a00644079e	reorgnize colotensor directory (#1062) ... * reorgnize colotensor directory * polish code	3 years ago
Ziyue Jiang	df9dcbbff6	[Tensor] add hybrid device demo and fix bugs (#1059)	3 years ago
Ziyue Jiang	7c530b9de2	[Tensor] add Parameter inheritance for ColoParameter (#1041) ... * add Parameter inheritance for ColoParameter * remove tricks * remove tricks * polish * polish	3 years ago
Ziyue Jiang	6c5996a56e	[Tensor] add module check and bert test (#1031) ... * add Embedding * Add bert test * polish * add check module test * polish * polish * polish * polish	3 years ago
Ziyue Jiang	32291dd73f	[Tensor] add module handler for linear (#1021) ... * add module spec for linear * polish * polish * polish	3 years ago
ver217	007ca0df92	fix colo init context (#1026)	3 years ago
ver217	ad536e308e	[tensor] refactor colo-tensor (#992) ... * refactor colo-tensor and update linear op * polish code * polish code * update ops and unit tests * update unit tests * polish code * rename dist_spec module * polish code * polish code * remove unneeded import * fix pipelinable	3 years ago
Ziyue Jiang	d73c2b1d79	[Tensor] fix init context (#931) ... * change torch.Parameter to ColoParameter * fix post assignment for init context * polish * polish	3 years ago
Ziyue Jiang	dfc88b85ea	[Tensor] simplify named param (#928) ... * simplify ColoModulize * simplify ColoModulize * polish * polish	3 years ago
YuliangLiu0306	32a45cd7ef	[pipelinable]use pipelinable to support GPT model. (#903) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [pipelinable]use pipelinable to support GPT model. * fix a bug caused by ShardedModel * polish * fix front func list	3 years ago
Ziyue Jiang	c195d2814c	[Tensor] add from_pretrained support and bert pretrained test (#921) ... * add from_pretrained support and test * polish * polish * polish * polish	3 years ago
Jiarui Fang	ab95ec9aea	[Tensor] init ColoParameter (#914)	3 years ago
Jiarui Fang	d16671da75	[Tensor] initialize the ColoOptimizer (#898) ... * [Tensor] activation is an attr of ColoTensor * [Tensor] add optimizer * only detach parameters in context * polish code	3 years ago
Jiarui Fang	676f191532	[Tensor] activation is an attr of ColoTensor (#897)	3 years ago
Jiarui Fang	26c49639d8	[Tensor] overriding paramters() for Module using ColoTensor (#889)	3 years ago
ver217	4df6471f5d	fix import error (#880)	3 years ago
Jiarui Fang	d01d3b8cb0	colo init context add device attr. (#866)	3 years ago
YuliangLiu0306	c6930d8ddf	[pipelinable]use ColoTensor to replace dummy tensor. (#853)	3 years ago
ver217	232142f402	[utils] refactor profiler (#837) ... * add model data profiler * add a subclass of torch.profiler.profile * refactor folder structure * remove redundant codes * polish code * use GeminiMemoryManager * fix import path * fix stm profiler ext * polish comments * remove useless file	3 years ago
Jiarui Fang	62f059251b	[Tensor] init a tp network training unittest (#849)	3 years ago
ver217	0dea140760	[hotfix] add deconstructor for stateful tensor (#848) ... * add deconstructor for stateful tensor * fix colo init context	3 years ago
YuliangLiu0306	35ea6e1023	[pipelinable]use pipelinable context to initialize non-pipeline model (#816) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [pipeline]add module lazy init feature to support large model initization. * [pipeline]add to_layer_list and partition method to support arbitrary non-pp model * refactor the module structure * polish * [pipelinable]add unit test for pipelinable * polish * polish * Fix CodeFactor issues.	3 years ago
Jiarui Fang	8789850eea	Init Conext supports lazy allocate model memory (#842)	3 years ago
Jiarui Fang	eb1b89908c	[refactor] moving InsertPostInitMethodToModuleSubClasses to utils. (#824)	3 years ago
Jiarui Fang	227d1cd4b3	[gemini] APIs to set cpu memory capacity (#809)	3 years ago
Jiarui Fang	681addb512	[refactor] moving grad acc logic to engine (#804)	3 years ago
Jiarui Fang	4d9332b4c5	[refactor] moving memtracer to gemini (#801)	3 years ago
HELSON	84c6700b2a	[zero] refactor memstats_collector (#746)	3 years ago
HELSON	340e59f968	[utils] add synchronized cuda memory monitor (#740)	3 years ago
Jiarui Fang	53cb584808	[utils] correct cpu memory used and capacity in the context of multi-process (#726)	3 years ago
Frank Lee	2412429d54	[util] fixed activation checkpointing on torch 1.9 (#719)	3 years ago
Jiarui Fang	193dc8dacb	[refactor] refactor the memory utils (#715)	3 years ago
LuGY	140263a394	[hotfix]fixed bugs of assigning grad states to non leaf nodes (#711) ... * fixed bugs of assigning grad states to non leaf nodes * use detach()	3 years ago
ver217	ab8c6b4a0e	[zero] refactor memstats collector (#706) ... * refactor memstats collector * fix disposable * polish code	3 years ago
ver217	3c9cd5bb5e	[zero] stateful tensor manager (#687) ... * [WIP] stateful tensor manager * add eviction strategy * polish code * polish code * polish comment * add unit test * fix sampler bug * polish code * fix max sampling cnt resetting bug * fix sampler bug * polish code * fix bug * fix unit test Co-authored-by: jiaruifang <fangjiarui123@gmail.com>	3 years ago
Jiarui Fang	59bf2dc590	[zero] initialize a stateful tensor manager (#614)	3 years ago
Jiarui Fang	0aab52301e	[hotfix] fix a bug in model data stats tracing (#655)	3 years ago
HELSON	e5d615aeee	[hotfix] fix bugs in testing (#659) ... * remove hybrid adam in test_moe_zero_optim * fix activation checkpointing and its unitest	3 years ago
LuGY	1e2557e801	[zero] fixed the activation offload (#647) ... * fixed the activation offload * polish	3 years ago
ver217	f5d3a9c2b0	polish checkpoint docstring (#637)	3 years ago
HELSON	055fbf5be6	[zero] adapt zero for unsharded paramters (Optimizer part) (#601)	3 years ago
アマデウス	acae68eb04	[model checkpoint] updated checkpoint save/load utils (#592)	3 years ago
ver217	369a288bf3	polish utils docstring (#620)	3 years ago
LuGY	02b187c14f	[zero] add sampling time for memstats collector (#610)	3 years ago
アマデウス	54e688b623	moved ensure_path_exists to utils.common (#591)	3 years ago
Jiarui Fang	e956d93ac2	[refactor] memory utils (#577)	3 years ago
HELSON	e6d50ec107	[zero] adapt zero for unsharded parameters (#561) ... * support existing sharded and unsharded parameters in zero * add unitest for moe-zero model init * polish moe gradient handler	3 years ago
ver217	7c6c427db1	[zero] trace states of fp16/32 grad and fp32 param (#571)	3 years ago
Jiarui Fang	7675366fce	[polish] rename col_attr -> colo_attr (#558)	3 years ago
Liang Bowen	2c45efc398	html refactor (#555)	3 years ago
Jiarui Fang	d1211148a7	[utils] update colo tensor moving APIs (#553)	3 years ago
Jiarui Fang	107b99ddb1	[zero] dump memory stats for sharded model (#548)	3 years ago
Liang Bowen	ec5086c49c	Refactored docstring to google style	3 years ago
Jiarui Fang	53b1b6e340	[zero] non model data tracing (#545)	3 years ago
Jie Zhu	73d36618a6	[profiler] add MemProfiler (#356) ... * add memory trainer hook * fix bug * add memory trainer hook * fix import bug * fix import bug * add trainer hook * fix #370 git log bug * modify `to_tensorboard` function to support better output * remove useless output * change the name of `MemProfiler` * complete memory profiler * replace error with warning * finish trainer hook * modify interface of MemProfiler * modify `__init__.py` in profiler * remove unnecessary pass statement * add usage to doc string * add usage to trainer hook * new location to store temp data file	3 years ago
Jiarui Fang	c11ff81b15	[zero] get memory usage of sharded optim v2. (#542)	3 years ago
Jiarui Fang	705f56107c	[zero] refactor model data tracing (#537)	3 years ago
Jiarui Fang	05e33b2578	[zero] fix grad offload (#528) ... * [zero] fix grad offload * polish code	3 years ago
Jiarui Fang	8d8c5407c0	[zero] refactor model data tracing (#522)	3 years ago
Jiarui Fang	920c5889a7	[zero] add colo move inline (#521)	3 years ago
Jiarui Fang	0bebda6ea5	[zero] fix init device bug in zero init context unittest (#516)	3 years ago
Jiarui Fang	7ef3507ace	[zero] show model data cuda memory usage after zero context init. (#515)	3 years ago
Jiarui Fang	9330be0f3c	[memory] set cuda mem frac (#506)	3 years ago
Jiarui Fang	0035b7be07	[memory] add model data tensor moving api (#503)	3 years ago
Jiarui Fang	a445e118cf	[polish] polish singleton and global context (#500)	3 years ago
HELSON	f24b5ed201	[MOE] remove old MoE legacy (#493)	3 years ago
Jiarui Fang	b334822163	[zero] polish sharded param name (#484) ... * [zero] polish sharded param name * polish code * polish * polish code * polish * polsih * polish	3 years ago
Jiarui Fang	65c0f380c2	[format] polish name format for MOE (#481)	3 years ago
HELSON	7544347145	[MOE] add unitest for MOE experts layout, gradient handler and kernel (#469)	3 years ago
HELSON	aff9d354f7	[MOE] polish moe_env (#467)	3 years ago
HELSON	84fd7c1d4d	add moe context, moe utilities and refactor gradient handler (#455)	3 years ago
Frank Lee	b72b8445c6	optimized context test time consumption (#446)	3 years ago
Jiarui Fang	496cbb0760	[hotfix] fix initialize bug with zero (#442)	3 years ago
Frank Lee	b03b3ae99c	fixed mem monitor device (#433) ... fixed mem monitor device	3 years ago
Jiarui Fang	adebb3e041	[zero] cuda margin space for OS (#418)	3 years ago
Jiarui Fang	56bb412e72	[polish] use GLOBAL_MODEL_DATA_TRACER (#417)	3 years ago
Jiarui Fang	21dc54e019	[zero] memtracer to record cuda memory usage of model data and overall system (#395)	3 years ago
LuGY	a9c27be42e	Added tensor detector (#393) ... * Added tensor detector * Added the - states * Allowed change include_cpu when detect()	3 years ago
1SAA	907ac4a2dc	fixed error when no collective communication in CommProfiler	3 years ago
HELSON	dfd0363f68	polished output format for communication profiler and pcie profiler (#404) ... fixed typing error	3 years ago
HELSON	7c079d9c33	[hotfix] fixed bugs in ShardStrategy and PcieProfiler (#394)	3 years ago
LuGY	de46450461	Added activation offload (#331) ... * Added activation offload * Fixed the import bug, used the pytest	3 years ago
HELSON	8c18eb0998	[profiler] Fixed bugs in CommProfiler and PcieProfiler (#377)	3 years ago
Jiarui Fang	b5f43acee3	[zero] find miss code (#378)	3 years ago
HELSON	1ed7c24c02	Added PCIE profiler to dectect data transmission (#373)	3 years ago
jiaruifang	d9217e1960	Revert "[zero] bucketized tensor cpu gpu copy (#368)" ... This reverts commit bef05489b6.	3 years ago
Jiarui Fang	00670c870e	[zero] bucketized tensor cpu gpu copy (#368)	3 years ago
Jiarui Fang	ea2872073f	[zero] global model data memory tracer (#360)	3 years ago
HELSON	534e0bb118	Fixed import bug for no-tensorboard environment (#354)	3 years ago
HELSON	c57e089824	[profile] added example for ProfilerContext (#349)	3 years ago
Jiarui Fang	10e2826426	move async memory to an individual directory (#345)	3 years ago