195 Commits (8d56c9c3894b882c94810a575bb3eb13fe7e2a98)

Author	SHA1	Message	Date
ver217	821c6172e2	[utils] Impl clip_grad_norm for ColoTensor and ZeroOptimizer (#1442)	2 years ago
HELSON	527758b2ae	[hotfix] fix a running error in test_colo_checkpoint.py (#1387)	2 years ago
HELSON	b6fd165f66	[checkpoint] add kwargs for load_state_dict (#1374)	2 years ago
Frank Lee	0c1a16ea5b	[util] standard checkpoint function naming (#1377)	2 years ago
Super Daniel	be229217ce	[fx] add torchaudio test (#1369) ... * [fx]add torchaudio test * [fx]add torchaudio test * [fx] add torchaudio test * [fx] add torchaudio test * [fx] add torchaudio test * [fx] add torchaudio test * [fx] add torchaudio test * [fx] add torchaudio test and test patches * Delete ~ * [fx] add patches and patches test * [fx] add patches and patches test * [fx] fix patches * [fx] fix rnn patches * [fx] fix rnn patches * [fx] fix rnn patches * [fx] fix rnn patches * [fx] merge upstream * [fx] fix import errors	2 years ago
HELSON	8463290642	[checkpoint] use args, kwargs in save_checkpoint, load_checkpoint (#1368)	2 years ago
HELSON	87775a0682	[colotensor] use cpu memory to store state_dict (#1367)	2 years ago
HELSON	943a96323e	[hotfix] fix no optimizer in save/load (#1363)	2 years ago
HELSON	7a8702c06d	[colotensor] add Tensor.view op and its unit test (#1343) ... [colotensor] add megatron initialization for gpt2	2 years ago
Frank Lee	2cc1175c76	[fx] tested the complete workflow for auto-parallel (#1336) ... * [fx] tested the complete workflow for auto-parallel * polish code * polish code * polish code	2 years ago
HELSON	f92c100ddd	[checkpoint] use gather_tensor in checkpoint and update its unit test (#1339)	2 years ago
Frank Lee	250be4d31e	[utils] integrated colotensor with lazy init context (#1324) ... * [utils] integrated colotensor with lazy init context * polish code * polish code * polish code	2 years ago
Jiarui Fang	9e4c6449b0	[checkpoint] add ColoOptimizer checkpointing (#1316)	2 years ago
Jiarui Fang	3ef3791a3b	[checkpoint] add test for bert and hotfix save bugs (#1297)	2 years ago
Jiarui Fang	4165eabb1e	[hotfix] remove potiential circle import (#1307) ... * make it faster * [hotfix] remove circle import	2 years ago
Jiarui Fang	c92f84fcdb	[tensor] distributed checkpointing for parameters (#1240)	2 years ago
Jiarui Fang	9bcd2fd4af	[tensor] a shorter shard and replicate spec (#1245)	2 years ago
Jiarui Fang	20da6e48c8	[checkpoint] save sharded optimizer states (#1237)	2 years ago
Jiarui Fang	3b500984b1	[tensor] fix some unittests (#1234)	2 years ago
ver217	a45ddf2d5f	[hotfix] fix sharded optim step and clip_grad_norm (#1226)	2 years ago
Yi Zhao	04537bf83e	[checkpoint]support generalized scheduler (#1222)	2 years ago
Jiarui Fang	52736205d9	[checkpoint] make unitest faster (#1217)	2 years ago
Jiarui Fang	f38006ea83	[checkpoint] checkpoint for ColoTensor Model (#1196)	2 years ago
Jiarui Fang	ae7d3f4927	[refactor] move process group from _DistSpec to ColoTensor. (#1203)	2 years ago
YuliangLiu0306	63d2a93878	[context]support arbitary module materialization. (#1193) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [context]support arbitary module materialization. * [test]add numerical check for lazy init context.	2 years ago
YuliangLiu0306	2053e138a2	[context]use meta tensor to init model lazily. (#1187) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [context]use meta tensor to init model lazily. * polish * make module with device kwargs bypass the normal init. * change unit test to adapt updated context.	2 years ago
YuliangLiu0306	e27645376d	[hotfix]different overflow status lead to communication stuck. (#1175) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [hotfix]fix some bugs caused by refactored schedule. * [hotfix]different overflow statu llead to communication stuck.	2 years ago
Jiarui Fang	4b9bba8116	[ColoTensor] rename APIs and add output_replicate to ComputeSpec (#1168)	2 years ago
Frank Lee	f8eec98ff5	[tensor] fixed non-serializable colo parameter during model checkpointing (#1153)	2 years ago
Frank Lee	73ad05fc8c	[zero] added error message to handle on-the-fly import of torch Module class (#1135) ... * [zero] added error message to handle on-the-fly import of torch Module class * polish code	2 years ago
Frank Lee	2b2dc1c86b	[pipeline] refactor the pipeline module (#1087) ... * [pipeline] refactor the pipeline module * polish code	3 years ago
Frank Lee	bad5d4c0a1	[context] support lazy init of module (#1088) ... * [context] support lazy init of module * polish code	3 years ago
Frank Lee	bfdc5ccb7b	[context] maintain the context object in with statement (#1073)	3 years ago
Jiarui Fang	49832b2344	[refactory] add nn.parallel module (#1068)	3 years ago
Jiarui Fang	a00644079e	reorgnize colotensor directory (#1062) ... * reorgnize colotensor directory * polish code	3 years ago
Ziyue Jiang	df9dcbbff6	[Tensor] add hybrid device demo and fix bugs (#1059)	3 years ago
Ziyue Jiang	7c530b9de2	[Tensor] add Parameter inheritance for ColoParameter (#1041) ... * add Parameter inheritance for ColoParameter * remove tricks * remove tricks * polish * polish	3 years ago
Ziyue Jiang	6c5996a56e	[Tensor] add module check and bert test (#1031) ... * add Embedding * Add bert test * polish * add check module test * polish * polish * polish * polish	3 years ago
Ziyue Jiang	32291dd73f	[Tensor] add module handler for linear (#1021) ... * add module spec for linear * polish * polish * polish	3 years ago
ver217	007ca0df92	fix colo init context (#1026)	3 years ago
ver217	ad536e308e	[tensor] refactor colo-tensor (#992) ... * refactor colo-tensor and update linear op * polish code * polish code * update ops and unit tests * update unit tests * polish code * rename dist_spec module * polish code * polish code * remove unneeded import * fix pipelinable	3 years ago
Ziyue Jiang	d73c2b1d79	[Tensor] fix init context (#931) ... * change torch.Parameter to ColoParameter * fix post assignment for init context * polish * polish	3 years ago
Ziyue Jiang	dfc88b85ea	[Tensor] simplify named param (#928) ... * simplify ColoModulize * simplify ColoModulize * polish * polish	3 years ago
YuliangLiu0306	32a45cd7ef	[pipelinable]use pipelinable to support GPT model. (#903) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [pipelinable]use pipelinable to support GPT model. * fix a bug caused by ShardedModel * polish * fix front func list	3 years ago
Ziyue Jiang	c195d2814c	[Tensor] add from_pretrained support and bert pretrained test (#921) ... * add from_pretrained support and test * polish * polish * polish * polish	3 years ago
Jiarui Fang	ab95ec9aea	[Tensor] init ColoParameter (#914)	3 years ago
Jiarui Fang	d16671da75	[Tensor] initialize the ColoOptimizer (#898) ... * [Tensor] activation is an attr of ColoTensor * [Tensor] add optimizer * only detach parameters in context * polish code	3 years ago
Jiarui Fang	676f191532	[Tensor] activation is an attr of ColoTensor (#897)	3 years ago
Jiarui Fang	26c49639d8	[Tensor] overriding paramters() for Module using ColoTensor (#889)	3 years ago
ver217	4df6471f5d	fix import error (#880)	3 years ago
Jiarui Fang	d01d3b8cb0	colo init context add device attr. (#866)	3 years ago
YuliangLiu0306	c6930d8ddf	[pipelinable]use ColoTensor to replace dummy tensor. (#853)	3 years ago
ver217	232142f402	[utils] refactor profiler (#837) ... * add model data profiler * add a subclass of torch.profiler.profile * refactor folder structure * remove redundant codes * polish code * use GeminiMemoryManager * fix import path * fix stm profiler ext * polish comments * remove useless file	3 years ago
Jiarui Fang	62f059251b	[Tensor] init a tp network training unittest (#849)	3 years ago
ver217	0dea140760	[hotfix] add deconstructor for stateful tensor (#848) ... * add deconstructor for stateful tensor * fix colo init context	3 years ago
YuliangLiu0306	35ea6e1023	[pipelinable]use pipelinable context to initialize non-pipeline model (#816) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [pipeline]add module lazy init feature to support large model initization. * [pipeline]add to_layer_list and partition method to support arbitrary non-pp model * refactor the module structure * polish * [pipelinable]add unit test for pipelinable * polish * polish * Fix CodeFactor issues.	3 years ago
Jiarui Fang	8789850eea	Init Conext supports lazy allocate model memory (#842)	3 years ago
Jiarui Fang	eb1b89908c	[refactor] moving InsertPostInitMethodToModuleSubClasses to utils. (#824)	3 years ago
Jiarui Fang	227d1cd4b3	[gemini] APIs to set cpu memory capacity (#809)	3 years ago
Jiarui Fang	681addb512	[refactor] moving grad acc logic to engine (#804)	3 years ago
Jiarui Fang	4d9332b4c5	[refactor] moving memtracer to gemini (#801)	3 years ago
HELSON	84c6700b2a	[zero] refactor memstats_collector (#746)	3 years ago
HELSON	340e59f968	[utils] add synchronized cuda memory monitor (#740)	3 years ago
Jiarui Fang	53cb584808	[utils] correct cpu memory used and capacity in the context of multi-process (#726)	3 years ago
Frank Lee	2412429d54	[util] fixed activation checkpointing on torch 1.9 (#719)	3 years ago
Jiarui Fang	193dc8dacb	[refactor] refactor the memory utils (#715)	3 years ago
LuGY	140263a394	[hotfix]fixed bugs of assigning grad states to non leaf nodes (#711) ... * fixed bugs of assigning grad states to non leaf nodes * use detach()	3 years ago
ver217	ab8c6b4a0e	[zero] refactor memstats collector (#706) ... * refactor memstats collector * fix disposable * polish code	3 years ago
ver217	3c9cd5bb5e	[zero] stateful tensor manager (#687) ... * [WIP] stateful tensor manager * add eviction strategy * polish code * polish code * polish comment * add unit test * fix sampler bug * polish code * fix max sampling cnt resetting bug * fix sampler bug * polish code * fix bug * fix unit test Co-authored-by: jiaruifang <fangjiarui123@gmail.com>	3 years ago
Jiarui Fang	59bf2dc590	[zero] initialize a stateful tensor manager (#614)	3 years ago
Jiarui Fang	0aab52301e	[hotfix] fix a bug in model data stats tracing (#655)	3 years ago
HELSON	e5d615aeee	[hotfix] fix bugs in testing (#659) ... * remove hybrid adam in test_moe_zero_optim * fix activation checkpointing and its unitest	3 years ago
LuGY	1e2557e801	[zero] fixed the activation offload (#647) ... * fixed the activation offload * polish	3 years ago
ver217	f5d3a9c2b0	polish checkpoint docstring (#637)	3 years ago
HELSON	055fbf5be6	[zero] adapt zero for unsharded paramters (Optimizer part) (#601)	3 years ago
アマデウス	acae68eb04	[model checkpoint] updated checkpoint save/load utils (#592)	3 years ago
ver217	369a288bf3	polish utils docstring (#620)	3 years ago
LuGY	02b187c14f	[zero] add sampling time for memstats collector (#610)	3 years ago
アマデウス	54e688b623	moved ensure_path_exists to utils.common (#591)	3 years ago
Jiarui Fang	e956d93ac2	[refactor] memory utils (#577)	3 years ago
HELSON	e6d50ec107	[zero] adapt zero for unsharded parameters (#561) ... * support existing sharded and unsharded parameters in zero * add unitest for moe-zero model init * polish moe gradient handler	3 years ago
ver217	7c6c427db1	[zero] trace states of fp16/32 grad and fp32 param (#571)	3 years ago
Jiarui Fang	7675366fce	[polish] rename col_attr -> colo_attr (#558)	3 years ago
Liang Bowen	2c45efc398	html refactor (#555)	3 years ago
Jiarui Fang	d1211148a7	[utils] update colo tensor moving APIs (#553)	3 years ago
Jiarui Fang	107b99ddb1	[zero] dump memory stats for sharded model (#548)	3 years ago
Liang Bowen	ec5086c49c	Refactored docstring to google style	3 years ago
Jiarui Fang	53b1b6e340	[zero] non model data tracing (#545)	3 years ago
Jie Zhu	73d36618a6	[profiler] add MemProfiler (#356) ... * add memory trainer hook * fix bug * add memory trainer hook * fix import bug * fix import bug * add trainer hook * fix #370 git log bug * modify `to_tensorboard` function to support better output * remove useless output * change the name of `MemProfiler` * complete memory profiler * replace error with warning * finish trainer hook * modify interface of MemProfiler * modify `__init__.py` in profiler * remove unnecessary pass statement * add usage to doc string * add usage to trainer hook * new location to store temp data file	3 years ago
Jiarui Fang	c11ff81b15	[zero] get memory usage of sharded optim v2. (#542)	3 years ago
Jiarui Fang	705f56107c	[zero] refactor model data tracing (#537)	3 years ago
Jiarui Fang	05e33b2578	[zero] fix grad offload (#528) ... * [zero] fix grad offload * polish code	3 years ago
Jiarui Fang	8d8c5407c0	[zero] refactor model data tracing (#522)	3 years ago
Jiarui Fang	920c5889a7	[zero] add colo move inline (#521)	3 years ago
Jiarui Fang	0bebda6ea5	[zero] fix init device bug in zero init context unittest (#516)	3 years ago
Jiarui Fang	7ef3507ace	[zero] show model data cuda memory usage after zero context init. (#515)	3 years ago
Jiarui Fang	9330be0f3c	[memory] set cuda mem frac (#506)	3 years ago
Jiarui Fang	0035b7be07	[memory] add model data tensor moving api (#503)	3 years ago
Jiarui Fang	a445e118cf	[polish] polish singleton and global context (#500)	3 years ago
HELSON	f24b5ed201	[MOE] remove old MoE legacy (#493)	3 years ago