681 Commits (6474e31556ae011410f29d8d0a2d80de67ed6956)

Author	SHA1	Message	Date
Jiarui Fang	00670c870e	[zero] bucketized tensor cpu gpu copy (#368)	3 years ago
Jiarui Fang	44e4891f57	[zero] able to place params on cpu after zero init context (#365) ... * place params on cpu after zero init context * polish code	3 years ago
ver217	253e54d98a	fix grad shape	3 years ago
Jiarui Fang	ea2872073f	[zero] global model data memory tracer (#360)	3 years ago
Jiarui Fang	cb34cd384d	[test] polish zero related unitest (#351)	3 years ago
HELSON	534e0bb118	Fixed import bug for no-tensorboard environment (#354)	3 years ago
HELSON	c57e089824	[profile] added example for ProfilerContext (#349)	3 years ago
Jiarui Fang	10e2826426	move async memory to an individual directory (#345)	3 years ago
HELSON	425bb0df3f	Added Profiler Context to manage all profilers (#340)	3 years ago
ver217	d0ae0f2215	[zero] update sharded optim v2 (#334)	3 years ago
jiaruifang	5663616921	polish code	3 years ago
jiaruifang	7977422aeb	add bert for unitest and sharded model is not able to pass the bert case	3 years ago
Frank Lee	3d5d64bd10	refactored grad scaler (#338)	3 years ago
Frank Lee	6a3188167c	set criterion as optional in colossalai initialize (#336)	3 years ago
Jie Zhu	3213554cc2	[profiler] add adaptive sampling to memory profiler (#330) ... * fix merge conflict modify unit test remove unnessesary log info reformat file * remove unused module * remove unnecessary sync function * change doc string style from Google to Sphinx	3 years ago
ver217	1388671699	[zero] Update sharded model v2 using sharded param v2 (#323)	3 years ago
Jiarui Fang	11bddb6e55	[zero] update zero context init with the updated test utils (#327)	3 years ago
HELSON	4f26fabe4f	fixed strings in profiler outputs (#325)	3 years ago
Jiarui Fang	de0468c7a8	[zero] zero init context (#321) ... * add zero init context * add more flags for zero init context fix bug of repeated converting param to ShardedParamV2 * polish code	3 years ago
1SAA	73bff11288	Added profiler communication operations ... Fixed bug for learning rate scheduler	3 years ago
LuGY	a3269de5c9	[zero] cpu adam kernel (#288) ... * Added CPU Adam * finished the cpu adam * updated the license * delete useless parameters, removed resnet * modified the method off cpu adam unittest * deleted some useless codes * removed useless codes Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com> Co-authored-by: jiaruifang <fangjiarui123@gmail.com>	3 years ago
Jiarui Fang	90d3aef62c	[zero] yet an improved sharded param (#311)	3 years ago
Jiarui Fang	c9e7d9582d	[zero] polish shard strategy (#310) ... * init shard param from shape tuple * add more unitest for shard param * add set_payload method for ShardedParam * [zero] add shareded tensor class * polish code * add shard stratgy * move shard and gather logic to shard strategy from shard tensor. * polish code	3 years ago
ver217	3092317b80	polish code	3 years ago
ver217	36f9a74ab2	fix sharded param hook and unit test	3 years ago
ver217	001ca624dd	impl shard optim v2 and add unit test	3 years ago
Jiarui Fang	74f77e314b	[zero] a shard strategy in granularity of tensor (#307)	3 years ago
Jiarui Fang	80364c7686	[zero] sharded tensor (#305) ... * init shard param from shape tuple * add more unitest for shard param * add set_payload method for ShardedParam * [zero] add shareded tensor class * polish code	3 years ago
Jie Zhu	d344689274	[profiler] primary memory tracer	3 years ago
ver217	b105371ace	rename shared adam to sharded optim v2	3 years ago
ver217	70814dc22f	fix master params dtype	3 years ago
ver217	795210dd99	add fp32 master params in sharded adam	3 years ago
ver217	a109225bc2	add sharded adam	3 years ago
Jiarui Fang	e17e92c54d	Polish sharded parameter (#297) ... * init shard param from shape tuple * add more unitest for shard param * add more unittests to shareded param	3 years ago
ver217	7aef75ca42	[zero] add sharded grad and refactor grad hooks for ShardedModel (#287)	3 years ago
Frank Lee	9afb5c8b2d	fixed typo in ShardParam (#294)	3 years ago
Frank Lee	e17e54e32a	added buffer sync to naive amp model wrapper (#291)	3 years ago
Jiarui Fang	8d653af408	add a common util for hooks registered on parameter. (#292)	3 years ago
Jie Zhu	f867365aba	bug fix: pass hook_list to engine (#273) ... * bug fix: pass hook_list to engine * change parameter name	3 years ago
Jiarui Fang	5a560a060a	Feature/zero (#279) ... * add zero1 (#209) * add zero1 * add test zero1 * update zero stage 1 develop (#212) * Implement naive zero3 (#240) * naive zero3 works well * add zero3 param manager * add TODOs in comments * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * fix bugs of hook and add unit tests (#252) * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * polish code and add state dict hook * fix bug * update unit test * refactor reconstructed zero code * clip_grad support zero3 and add unit test * add unit test for Zero3ParameterManager * [WIP] initialize the shard param class * [WIP] Yet another sharded model implementation (#274) * [WIP] initialize the shard param class * [WIP] Yes another implementation of shardModel. Using a better hook method. * torch.concat -> torch.cat * fix test_zero_level_1.py::test_zero_level_1 unitest * remove deepspeed implementation and refactor for the reconstructed zero module * polish zero dp unittests Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com>	3 years ago
1SAA	82023779bb	Added TPExpert for special situation	3 years ago
HELSON	36b8477228	Fixed parameter initialization in FFNExpert (#251)	3 years ago
アマデウス	e13293bb4c	fixed CI dataset directory; fixed import error of 2.5d accuracy (#255)	3 years ago
1SAA	219df6e685	Optimized MoE layer and fixed some bugs; ... Decreased moe tests; Added FFNExperts and ViTMoE model	3 years ago
zbian	3dba070580	fixed padding index issue for vocab parallel embedding layers; updated 3D linear to be compatible with examples in the tutorial	3 years ago
Frank Lee	f5ca88ec97	fixed apex import (#227)	3 years ago
Frank Lee	3a1a9820b0	fixed mkdir conflict and align yapf config with flake (#220)	3 years ago
アマデウス	9ee197d0e9	moved env variables to global variables; (#215) ... added branch context; added vocab parallel layers; moved split_batch from load_batch to tensor parallel embedding layers; updated gpt model; updated unit test cases; fixed few collective communicator bugs	3 years ago
Frank Lee	812357d63c	fixed utils docstring and add example to readme (#200)	3 years ago
Frank Lee	765db512b5	fixed ddp bug on torch 1.8 (#194)	3 years ago
Jiarui Fang	569357fea0	add pytorch hooks (#179) ... * add pytorch hooks fix #175 * remove licenses in src code * add gpu memory tracer * replacing print with logger in ophooks.	3 years ago
ver217	708404d5f8	fix pipeline forward return tensors (#176)	3 years ago
HELSON	0f8c7f9804	Fixed docstring in colossalai (#171)	3 years ago
Frank Lee	e2089c5c15	adapted for sequence parallel (#163)	3 years ago
puck_WCR	9473a1b9c8	AMP docstring/markdown update (#160)	3 years ago
Frank Lee	f3802d6b06	fixed jit default setting (#154)	3 years ago
ver217	7bf1e98b97	pipeline last stage supports multi output (#151)	3 years ago
ver217	f68eddfb3d	refactor kernel (#142)	3 years ago
BoxiangW	4a3d3446b0	Update layer integration documentations (#108) ... Update the documentations of layer integration Update _log_hook.py Update _operation.py	3 years ago
ver217	9ef05ed1fc	try import deepspeed when using zero (#130)	3 years ago
HELSON	dceae85195	Added MoE parallel (#127)	3 years ago
ver217	293fb40c42	add scatter/gather optim for pipeline (#123)	3 years ago
Jiarui Fang	2c0c85d3d3	fix a bug in timer (#114)	3 years ago
ver217	7904baf6e1	fix layers/schedule for hybrid parallelization (#111) (#112)	3 years ago
ver217	a951bc6089	update default logger (#100) (#101)	3 years ago
ver217	96780e6ee4	Optimize pipeline schedule (#94) ... * add pipeline shared module wrapper and update load batch * added model parallel process group for amp and clip grad (#86) * added model parallel process group for amp and clip grad * update amp and clip with model parallel process group * remove pipeline_prev/next group (#88) * micro batch offload * optimize pipeline gpu memory usage * pipeline can receive tensor shape (#93) * optimize pipeline gpu memory usage * fix grad accumulation step counter * rename classes and functions Co-authored-by: Frank Lee <somerlee.9@gmail.com>	3 years ago
アマデウス	01a80cd86d	Hotfix/Colossalai layers (#92) ... * optimized 1d layer apis; reorganized nn.layer modules; fixed tests * fixed 2.5d runtime issue * reworked split batch, now called in trainer.schedule.load_batch Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>	3 years ago
アマデウス	0fedef4f3c	Layer integration (#83) ... * integrated parallel layers for ease of building models * integrated 2.5d layers * cleaned codes and unit tests * added log metric by step hook; updated imagenet benchmark; fixed some bugs * reworked initialization; cleaned codes Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>	3 years ago
shenggan	5c3843dc98	add colossalai kernel module (#55)	3 years ago
ver217	8f02a88db2	add interleaved pipeline, fix naive amp and update pipeline model initializer (#80)	3 years ago
Frank Lee	91c327cb44	fixed zero level 3 dtype bug (#76)	3 years ago
HELSON	632e622de8	overlap computation and communication in 2d operations (#75)	3 years ago
Frank Lee	cd9c28e055	added CI for unit testing (#69)	3 years ago
Frank Lee	35813ed3c4	update examples and sphnix docs for the new api (#63)	3 years ago
ver217	7d3711058f	fix zero3 fp16 and add zero3 model context (#62)	3 years ago
Frank Lee	9a0466534c	update markdown docs (english) (#60)	3 years ago
Frank Lee	da01c234e1	Develop/experiments (#59) ... * Add gradient accumulation, fix lr scheduler * fix FP16 optimizer and adapted torch amp with tensor parallel (#18) * fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes * fixed trainer * Revert "fixed trainer" This reverts commit 2e0b0b7699. * improved consistency between trainer, engine and schedule (#23) Co-authored-by: 1SAA <c2h214748@gmail.com> * Split conv2d, class token, positional embedding in 2d, Fix random number in ddp Fix convergence in cifar10, Imagenet1000 * Integrate 1d tensor parallel in Colossal-AI (#39) * fixed 1D and 2D convergence (#38) * optimized 2D operations * fixed 1D ViT convergence problem * Feature/ddp (#49) * remove redundancy func in setup (#19) (#20) * use env to control the language of doc (#24) (#25) * Support TP-compatible Torch AMP and Update trainer API (#27) * Add gradient accumulation, fix lr scheduler * fix FP16 optimizer and adapted torch amp with tensor parallel (#18) * fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes * fixed trainer * Revert "fixed trainer" This reverts commit 2e0b0b7699. * improved consistency between trainer, engine and schedule (#23) Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: ver217 <lhx0217@gmail.com> * add an example of ViT-B/16 and remove w_norm clipping in LAMB (#29) * add explanation for ViT example (#35) (#36) * support torch ddp * fix loss accumulation * add log for ddp * change seed * modify timing hook Co-authored-by: Frank Lee <somerlee.9@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: binmakeswell <binmakeswell@gmail.com> * Feature/pipeline (#40) * remove redundancy func in setup (#19) (#20) * use env to control the language of doc (#24) (#25) * Support TP-compatible Torch AMP and Update trainer API (#27) * Add gradient accumulation, fix lr scheduler * fix FP16 optimizer and adapted torch amp with tensor parallel (#18) * fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes * fixed trainer * Revert "fixed trainer" This reverts commit 2e0b0b7699. * improved consistency between trainer, engine and schedule (#23) Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: ver217 <lhx0217@gmail.com> * add an example of ViT-B/16 and remove w_norm clipping in LAMB (#29) * add explanation for ViT example (#35) (#36) * optimize communication of pipeline parallel * fix grad clip for pipeline Co-authored-by: Frank Lee <somerlee.9@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: binmakeswell <binmakeswell@gmail.com> * optimized 3d layer to fix slow computation ; tested imagenet performance with 3d; reworked lr_scheduler config definition; fixed launch args; fixed some printing issues; simplified apis of 3d layers (#51) * Update 2.5d layer code to get a similar accuracy on imagenet-1k dataset * update api for better usability (#58) update api for better usability Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: puck_WCR <46049915+WANG-CR@users.noreply.github.com> Co-authored-by: binmakeswell <binmakeswell@gmail.com> Co-authored-by: アマデウス <kurisusnowdeng@users.noreply.github.com> Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>	3 years ago
ver217	dbe62c67b8	add an example of ViT-B/16 and remove w_norm clipping in LAMB (#29)	3 years ago
Frank Lee	3defa32aee	Support TP-compatible Torch AMP and Update trainer API (#27) ... * Add gradient accumulation, fix lr scheduler * fix FP16 optimizer and adapted torch amp with tensor parallel (#18) * fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes * fixed trainer * Revert "fixed trainer" This reverts commit 2e0b0b7699. * improved consistency between trainer, engine and schedule (#23) Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: ver217 <lhx0217@gmail.com>	3 years ago
ver217	3c7604ba30	update documentation	3 years ago
zbian	404ecbdcc6	Migrated project	3 years ago