57 Commits (31c78f2be3272a9a4062fe78eca34b3847a0c900)

Author	SHA1	Message	Date
ver217	f8289d4221	[lazyinit] combine lazy tensor with dtensor (#3204) ... * [lazyinit] lazy tensor add distribute * [lazyinit] refactor distribute * [lazyinit] add test dist lazy init * [lazyinit] add verbose info for dist lazy init * [lazyinit] fix rnn flatten weight op * [lazyinit] polish test * [lazyinit] polish test * [lazyinit] fix lazy tensor data setter * [lazyinit] polish test * [lazyinit] fix clean * [lazyinit] make materialize inplace * [lazyinit] refactor materialize * [lazyinit] refactor test distribute * [lazyinit] fix requires_grad * [lazyinit] fix tolist after materialization * [lazyinit] refactor distribute module * [lazyinit] polish docstr * [lazyinit] polish lazy init context * [lazyinit] temporarily skip test * [lazyinit] polish test * [lazyinit] add docstr	2 years ago
ver217	6ae8ed0407	[lazyinit] add correctness verification (#3147) ... * [lazyinit] fix shared module * [tests] add lazy init test utils * [tests] add torchvision for lazy init * [lazyinit] fix pre op fn * [lazyinit] handle legacy constructor * [tests] refactor lazy init test models * [tests] refactor lazy init test utils * [lazyinit] fix ops don't support meta * [tests] lazy init test timm models * [lazyinit] fix set data * [lazyinit] handle apex layers * [tests] lazy init test transformers models * [tests] lazy init test torchaudio models * [lazyinit] fix import path * [tests] lazy init test torchrec models * [tests] update torch version in CI * [tests] revert torch version in CI * [tests] skip lazy init test	2 years ago
ver217	ed8f60b93b	[lazyinit] refactor lazy tensor and lazy init ctx (#3131) ... * [lazyinit] refactor lazy tensor and lazy init ctx * [lazyinit] polish docstr * [lazyinit] polish docstr	2 years ago
ver217	823f3b9cf4	[doc] add deepspeed citation and copyright (#2996) ... * [doc] add deepspeed citation and copyright * [doc] add deepspeed citation and copyright * [doc] add deepspeed citation and copyright	2 years ago
ver217	f0aa191f51	[gemini] fix colo_init_context (#2683)	2 years ago
HELSON	552183bb74	[polish] polish ColoTensor and its submodules (#2537)	2 years ago
Super Daniel	35c0c0006e	[utils] lazy init. (#2148) ... * [utils] lazy init. * [utils] remove description. * [utils] complete. * [utils] finalize. * [utils] fix names.	2 years ago
BlueRum	b3f73ce1c8	[Gemini] Update coloinit_ctx to support meta_tensor (#2147)	2 years ago
Jiarui Fang	8e14344ec9	[hotfix] fix a type in ColoInitContext (#2106)	2 years ago
Jiarui Fang	05545bfee9	[ColoTensor] throw error when ColoInitContext meets meta parameter. (#2105)	2 years ago
HELSON	f6178728a0	[gemini] fix init bugs for modules (#2047) ... * [gemini] fix init bugs for modules * fix bugs	2 years ago
Jiarui Fang	31c644027b	[hotfix] hotfix Gemini for no leaf modules bug (#2043)	2 years ago
Jiarui Fang	52c6ad26e0	[ColoTensor] reconfig ColoInitContext, decouple default_pg and default_dist_spec. (#1953)	2 years ago
Jiarui Fang	9f4fb3f28a	[ColoTensor] ColoInitContext initialize parameters in shard mode. (#1937)	2 years ago
Frank Lee	e6ec99d389	[utils] fixed lazy init context (#1867)	2 years ago
Jiarui Fang	3ce4463fe6	[utils] remove lazy_memory_allocate from ColoInitContext (#1844)	2 years ago
HELSON	1468e4bcfc	[zero] add constant placement policy (#1705) ... * fixes memory leak when paramter is in fp16 in ZeroDDP init. * bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release. * adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.	2 years ago
ver217	a203b709d5	[hotfix] fix init context (#1543) ... * fix init context * fix lazy init ctx	2 years ago
Frank Lee	2cc1175c76	[fx] tested the complete workflow for auto-parallel (#1336) ... * [fx] tested the complete workflow for auto-parallel * polish code * polish code * polish code	2 years ago
Frank Lee	250be4d31e	[utils] integrated colotensor with lazy init context (#1324) ... * [utils] integrated colotensor with lazy init context * polish code * polish code * polish code	2 years ago
Jiarui Fang	c92f84fcdb	[tensor] distributed checkpointing for parameters (#1240)	2 years ago
Jiarui Fang	9bcd2fd4af	[tensor] a shorter shard and replicate spec (#1245)	2 years ago
Jiarui Fang	3b500984b1	[tensor] fix some unittests (#1234)	2 years ago
Jiarui Fang	f38006ea83	[checkpoint] checkpoint for ColoTensor Model (#1196)	2 years ago
Jiarui Fang	ae7d3f4927	[refactor] move process group from _DistSpec to ColoTensor. (#1203)	2 years ago
YuliangLiu0306	63d2a93878	[context]support arbitary module materialization. (#1193) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [context]support arbitary module materialization. * [test]add numerical check for lazy init context.	2 years ago
YuliangLiu0306	2053e138a2	[context]use meta tensor to init model lazily. (#1187) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [context]use meta tensor to init model lazily. * polish * make module with device kwargs bypass the normal init. * change unit test to adapt updated context.	2 years ago
Jiarui Fang	4b9bba8116	[ColoTensor] rename APIs and add output_replicate to ComputeSpec (#1168)	2 years ago
Frank Lee	f8eec98ff5	[tensor] fixed non-serializable colo parameter during model checkpointing (#1153)	2 years ago
Frank Lee	73ad05fc8c	[zero] added error message to handle on-the-fly import of torch Module class (#1135) ... * [zero] added error message to handle on-the-fly import of torch Module class * polish code	2 years ago
Frank Lee	2b2dc1c86b	[pipeline] refactor the pipeline module (#1087) ... * [pipeline] refactor the pipeline module * polish code	3 years ago
Frank Lee	bad5d4c0a1	[context] support lazy init of module (#1088) ... * [context] support lazy init of module * polish code	3 years ago
Frank Lee	bfdc5ccb7b	[context] maintain the context object in with statement (#1073)	3 years ago
Jiarui Fang	49832b2344	[refactory] add nn.parallel module (#1068)	3 years ago
Jiarui Fang	a00644079e	reorgnize colotensor directory (#1062) ... * reorgnize colotensor directory * polish code	3 years ago
Ziyue Jiang	df9dcbbff6	[Tensor] add hybrid device demo and fix bugs (#1059)	3 years ago
Ziyue Jiang	7c530b9de2	[Tensor] add Parameter inheritance for ColoParameter (#1041) ... * add Parameter inheritance for ColoParameter * remove tricks * remove tricks * polish * polish	3 years ago
Ziyue Jiang	6c5996a56e	[Tensor] add module check and bert test (#1031) ... * add Embedding * Add bert test * polish * add check module test * polish * polish * polish * polish	3 years ago
Ziyue Jiang	32291dd73f	[Tensor] add module handler for linear (#1021) ... * add module spec for linear * polish * polish * polish	3 years ago
ver217	007ca0df92	fix colo init context (#1026)	3 years ago
ver217	ad536e308e	[tensor] refactor colo-tensor (#992) ... * refactor colo-tensor and update linear op * polish code * polish code * update ops and unit tests * update unit tests * polish code * rename dist_spec module * polish code * polish code * remove unneeded import * fix pipelinable	3 years ago
Ziyue Jiang	d73c2b1d79	[Tensor] fix init context (#931) ... * change torch.Parameter to ColoParameter * fix post assignment for init context * polish * polish	3 years ago
Ziyue Jiang	dfc88b85ea	[Tensor] simplify named param (#928) ... * simplify ColoModulize * simplify ColoModulize * polish * polish	3 years ago
YuliangLiu0306	32a45cd7ef	[pipelinable]use pipelinable to support GPT model. (#903) ... * [CLI] add CLI launcher * Revert "[CLI] add CLI launcher" This reverts commit df7e6506d4. * [pipelinable]use pipelinable to support GPT model. * fix a bug caused by ShardedModel * polish * fix front func list	3 years ago
Ziyue Jiang	c195d2814c	[Tensor] add from_pretrained support and bert pretrained test (#921) ... * add from_pretrained support and test * polish * polish * polish * polish	3 years ago
Jiarui Fang	ab95ec9aea	[Tensor] init ColoParameter (#914)	3 years ago
Jiarui Fang	d16671da75	[Tensor] initialize the ColoOptimizer (#898) ... * [Tensor] activation is an attr of ColoTensor * [Tensor] add optimizer * only detach parameters in context * polish code	3 years ago
Jiarui Fang	676f191532	[Tensor] activation is an attr of ColoTensor (#897)	3 years ago
Jiarui Fang	26c49639d8	[Tensor] overriding paramters() for Module using ColoTensor (#889)	3 years ago
ver217	4df6471f5d	fix import error (#880)	3 years ago