1335 Commits (1bed38ef37c6edd4ddc9c935218382305c5f9438)

Author	SHA1	Message	Date
ver217	642846d6f9	update sharded optim and fix zero init ctx (#457)	3 years ago
Jiarui Fang	e2e9f82588	Revert "[zero] update sharded optim and fix zero init ctx" (#456) ... * Revert "polish code" This reverts commit 8cf7ff08cf. * Revert "rename variables" This reverts commit e99af94ab8. * Revert "remove surplus imports" This reverts commit 46add4a5c5. * Revert "update sharded optim and fix zero init ctx" This reverts commit 57567ee768.	3 years ago
ver217	e99af94ab8	rename variables	3 years ago
ver217	57567ee768	update sharded optim and fix zero init ctx	3 years ago
Jiarui Fang	0fcfb1e00d	[test] make zero engine test really work (#447)	3 years ago
Jiarui Fang	237d08e7ee	[zero] hybrid cpu adam (#445)	3 years ago
Frank Lee	b72b8445c6	optimized context test time consumption (#446)	3 years ago
Jiarui Fang	496cbb0760	[hotfix] fix initialize bug with zero (#442)	3 years ago
Jiarui Fang	640a6cd304	[refactory] refactory the initialize method for new zero design (#431)	3 years ago
Frank Lee	bffd85bf34	added testing module (#435)	3 years ago
HELSON	dbdc9a7783	added Multiply Jitter and capacity factor eval for MOE (#434)	3 years ago
Frank Lee	b03b3ae99c	fixed mem monitor device (#433) ... fixed mem monitor device	3 years ago
Frank Lee	14a7094243	fixed fp16 optimizer none grad bug (#432)	3 years ago
ver217	fce9432f08	sync before creating empty grad	3 years ago
ver217	ea6905a898	free param.grad	3 years ago
ver217	9506a8beb2	use double buffer to handle grad	3 years ago
Jiarui Fang	54229cd33e	[log] better logging display with rich (#426) ... * better logger using rich * remove deepspeed in zero requirements	3 years ago
HELSON	3f70a2b12f	removed noisy function during evaluation of MoE router (#419)	3 years ago
Jiarui Fang	adebb3e041	[zero] cuda margin space for OS (#418)	3 years ago
Jiarui Fang	56bb412e72	[polish] use GLOBAL_MODEL_DATA_TRACER (#417)	3 years ago
Jiarui Fang	23ba3fc450	[zero] refactory ShardedOptimV2 init method (#416)	3 years ago
Frank Lee	e79ea44247	[fp16] refactored fp16 optimizer (#392)	3 years ago
Jiarui Fang	21dc54e019	[zero] memtracer to record cuda memory usage of model data and overall system (#395)	3 years ago
Jiarui Fang	370f567e7d	[zero] new interface for ShardedOptimv2 (#406)	3 years ago
LuGY	a9c27be42e	Added tensor detector (#393) ... * Added tensor detector * Added the - states * Allowed change include_cpu when detect()	3 years ago
1SAA	907ac4a2dc	fixed error when no collective communication in CommProfiler	3 years ago
Frank Lee	2fe68b359a	Merge pull request #403 from ver217/feature/shard-strategy ... [zero] Add bucket tensor shard strategy	3 years ago
HELSON	dfd0363f68	polished output format for communication profiler and pcie profiler (#404) ... fixed typing error	3 years ago
ver217	63469c0f91	polish code	3 years ago
ver217	88804aee49	add bucket tensor shard strategy	3 years ago
HELSON	7c079d9c33	[hotfix] fixed bugs in ShardStrategy and PcieProfiler (#394)	3 years ago
Frank Lee	1e4bf85cdb	fixed bug in activation checkpointing test (#387)	3 years ago
Jiarui Fang	3af13a2c3e	[zero] polish ShardedOptimV2 unittest (#385) ... * place params on cpu after zero init context * polish code * bucketzed cpu gpu tensor transter * find a bug in sharded optim unittest * add offload unittest for ShardedOptimV2. * polish code and make it more robust	3 years ago
Jiang Zhuo	5a4a3b77d9	fix format (#376)	3 years ago
LuGY	de46450461	Added activation offload (#331) ... * Added activation offload * Fixed the import bug, used the pytest	3 years ago
Jiarui Fang	272ebfb57d	[bug] shard param during initializing the ShardedModelV2 (#381)	3 years ago
HELSON	8c18eb0998	[profiler] Fixed bugs in CommProfiler and PcieProfiler (#377)	3 years ago
Jiarui Fang	b5f43acee3	[zero] find miss code (#378)	3 years ago
Jiarui Fang	6b6002962a	[zero] zero init context collect numel of model (#375)	3 years ago
HELSON	1ed7c24c02	Added PCIE profiler to dectect data transmission (#373)	3 years ago
jiaruifang	d9217e1960	Revert "[zero] bucketized tensor cpu gpu copy (#368)" ... This reverts commit bef05489b6.	3 years ago
RichardoLuo	8539898ec6	flake8 style change (#363)	3 years ago
Kai Wang (Victor Kai)	53bb3bcc0a	fix format (#362)	3 years ago
ziyu huang	a77d73f22b	fix format parallel_context.py (#359) ... Co-authored-by: huangziyu <202476410arsmart@gmail.com>	3 years ago
Zangwei	c695369af0	fix format constants.py (#358)	3 years ago
Yuer867	4a0f8c2c50	fix format parallel_2p5d (#357)	3 years ago
Liang Bowen	7eb87f516d	flake8 style (#352)	3 years ago
Xu Kai	54ee8d1254	Fix/format colossalai/engine/paramhooks/(#350)	3 years ago
Maruyama_Aya	e83970e3dc	fix format ColossalAI\colossalai\context\process_group_initializer	3 years ago
yuxuan-lou	3b88eb2259	Flake8 code restyle	3 years ago
xuqifan897	148207048e	Qifan formated file ColossalAI\colossalai\nn\layer\parallel_1d\layers.py (#342)	3 years ago
Cautiousss	3a51d909af	fix format (#332) ... Co-authored-by: 何晓昕 <cautious@r-205-106-25-172.comp.nus.edu.sg>	3 years ago
DouJS	cbb6436ff0	fix format for dir-[parallel_3d] (#333)	3 years ago
ExtremeViscent	eaac03ae1d	[formart] format fixed for kernel\cuda_native codes (#335)	3 years ago
Jiarui Fang	00670c870e	[zero] bucketized tensor cpu gpu copy (#368)	3 years ago
Jiarui Fang	44e4891f57	[zero] able to place params on cpu after zero init context (#365) ... * place params on cpu after zero init context * polish code	3 years ago
ver217	253e54d98a	fix grad shape	3 years ago
Jiarui Fang	ea2872073f	[zero] global model data memory tracer (#360)	3 years ago
Jiarui Fang	cb34cd384d	[test] polish zero related unitest (#351)	3 years ago
HELSON	534e0bb118	Fixed import bug for no-tensorboard environment (#354)	3 years ago
HELSON	c57e089824	[profile] added example for ProfilerContext (#349)	3 years ago
Jiarui Fang	10e2826426	move async memory to an individual directory (#345)	3 years ago
HELSON	425bb0df3f	Added Profiler Context to manage all profilers (#340)	3 years ago
ver217	d0ae0f2215	[zero] update sharded optim v2 (#334)	3 years ago
jiaruifang	5663616921	polish code	3 years ago
jiaruifang	7977422aeb	add bert for unitest and sharded model is not able to pass the bert case	3 years ago
Frank Lee	3d5d64bd10	refactored grad scaler (#338)	3 years ago
Frank Lee	6a3188167c	set criterion as optional in colossalai initialize (#336)	3 years ago
Jie Zhu	3213554cc2	[profiler] add adaptive sampling to memory profiler (#330) ... * fix merge conflict modify unit test remove unnessesary log info reformat file * remove unused module * remove unnecessary sync function * change doc string style from Google to Sphinx	3 years ago
ver217	1388671699	[zero] Update sharded model v2 using sharded param v2 (#323)	3 years ago
Jiarui Fang	11bddb6e55	[zero] update zero context init with the updated test utils (#327)	3 years ago
HELSON	4f26fabe4f	fixed strings in profiler outputs (#325)	3 years ago
Jiarui Fang	de0468c7a8	[zero] zero init context (#321) ... * add zero init context * add more flags for zero init context fix bug of repeated converting param to ShardedParamV2 * polish code	3 years ago
1SAA	73bff11288	Added profiler communication operations ... Fixed bug for learning rate scheduler	3 years ago
LuGY	a3269de5c9	[zero] cpu adam kernel (#288) ... * Added CPU Adam * finished the cpu adam * updated the license * delete useless parameters, removed resnet * modified the method off cpu adam unittest * deleted some useless codes * removed useless codes Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com> Co-authored-by: jiaruifang <fangjiarui123@gmail.com>	3 years ago
Jiarui Fang	90d3aef62c	[zero] yet an improved sharded param (#311)	3 years ago
Jiarui Fang	c9e7d9582d	[zero] polish shard strategy (#310) ... * init shard param from shape tuple * add more unitest for shard param * add set_payload method for ShardedParam * [zero] add shareded tensor class * polish code * add shard stratgy * move shard and gather logic to shard strategy from shard tensor. * polish code	3 years ago
ver217	3092317b80	polish code	3 years ago
ver217	36f9a74ab2	fix sharded param hook and unit test	3 years ago
ver217	001ca624dd	impl shard optim v2 and add unit test	3 years ago
Jiarui Fang	74f77e314b	[zero] a shard strategy in granularity of tensor (#307)	3 years ago
Jiarui Fang	80364c7686	[zero] sharded tensor (#305) ... * init shard param from shape tuple * add more unitest for shard param * add set_payload method for ShardedParam * [zero] add shareded tensor class * polish code	3 years ago
Jie Zhu	d344689274	[profiler] primary memory tracer	3 years ago
ver217	b105371ace	rename shared adam to sharded optim v2	3 years ago
ver217	70814dc22f	fix master params dtype	3 years ago
ver217	795210dd99	add fp32 master params in sharded adam	3 years ago
ver217	a109225bc2	add sharded adam	3 years ago
Jiarui Fang	e17e92c54d	Polish sharded parameter (#297) ... * init shard param from shape tuple * add more unitest for shard param * add more unittests to shareded param	3 years ago
ver217	7aef75ca42	[zero] add sharded grad and refactor grad hooks for ShardedModel (#287)	3 years ago
Frank Lee	9afb5c8b2d	fixed typo in ShardParam (#294)	3 years ago
Frank Lee	e17e54e32a	added buffer sync to naive amp model wrapper (#291)	3 years ago
Jiarui Fang	8d653af408	add a common util for hooks registered on parameter. (#292)	3 years ago
Jie Zhu	f867365aba	bug fix: pass hook_list to engine (#273) ... * bug fix: pass hook_list to engine * change parameter name	3 years ago
Jiarui Fang	5a560a060a	Feature/zero (#279) ... * add zero1 (#209) * add zero1 * add test zero1 * update zero stage 1 develop (#212) * Implement naive zero3 (#240) * naive zero3 works well * add zero3 param manager * add TODOs in comments * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * fix bugs of hook and add unit tests (#252) * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * polish code and add state dict hook * fix bug * update unit test * refactor reconstructed zero code * clip_grad support zero3 and add unit test * add unit test for Zero3ParameterManager * [WIP] initialize the shard param class * [WIP] Yet another sharded model implementation (#274) * [WIP] initialize the shard param class * [WIP] Yes another implementation of shardModel. Using a better hook method. * torch.concat -> torch.cat * fix test_zero_level_1.py::test_zero_level_1 unitest * remove deepspeed implementation and refactor for the reconstructed zero module * polish zero dp unittests Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com>	3 years ago
1SAA	82023779bb	Added TPExpert for special situation	3 years ago
HELSON	36b8477228	Fixed parameter initialization in FFNExpert (#251)	3 years ago
アマデウス	e13293bb4c	fixed CI dataset directory; fixed import error of 2.5d accuracy (#255)	3 years ago
1SAA	219df6e685	Optimized MoE layer and fixed some bugs; ... Decreased moe tests; Added FFNExperts and ViTMoE model	3 years ago
zbian	3dba070580	fixed padding index issue for vocab parallel embedding layers; updated 3D linear to be compatible with examples in the tutorial	3 years ago
Frank Lee	f5ca88ec97	fixed apex import (#227)	3 years ago
Frank Lee	3a1a9820b0	fixed mkdir conflict and align yapf config with flake (#220)	3 years ago
アマデウス	9ee197d0e9	moved env variables to global variables; (#215) ... added branch context; added vocab parallel layers; moved split_batch from load_batch to tensor parallel embedding layers; updated gpt model; updated unit test cases; fixed few collective communicator bugs	3 years ago
Frank Lee	812357d63c	fixed utils docstring and add example to readme (#200)	3 years ago
Frank Lee	765db512b5	fixed ddp bug on torch 1.8 (#194)	3 years ago
Jiarui Fang	569357fea0	add pytorch hooks (#179) ... * add pytorch hooks fix #175 * remove licenses in src code * add gpu memory tracer * replacing print with logger in ophooks.	3 years ago
ver217	708404d5f8	fix pipeline forward return tensors (#176)	3 years ago
HELSON	0f8c7f9804	Fixed docstring in colossalai (#171)	3 years ago
Frank Lee	e2089c5c15	adapted for sequence parallel (#163)	3 years ago
puck_WCR	9473a1b9c8	AMP docstring/markdown update (#160)	3 years ago
Frank Lee	f3802d6b06	fixed jit default setting (#154)	3 years ago
ver217	7bf1e98b97	pipeline last stage supports multi output (#151)	3 years ago
ver217	f68eddfb3d	refactor kernel (#142)	3 years ago
BoxiangW	4a3d3446b0	Update layer integration documentations (#108) ... Update the documentations of layer integration Update _log_hook.py Update _operation.py	3 years ago
ver217	9ef05ed1fc	try import deepspeed when using zero (#130)	3 years ago
HELSON	dceae85195	Added MoE parallel (#127)	3 years ago
ver217	293fb40c42	add scatter/gather optim for pipeline (#123)	3 years ago
Jiarui Fang	2c0c85d3d3	fix a bug in timer (#114)	3 years ago
ver217	7904baf6e1	fix layers/schedule for hybrid parallelization (#111) (#112)	3 years ago
ver217	a951bc6089	update default logger (#100) (#101)	3 years ago
ver217	96780e6ee4	Optimize pipeline schedule (#94) ... * add pipeline shared module wrapper and update load batch * added model parallel process group for amp and clip grad (#86) * added model parallel process group for amp and clip grad * update amp and clip with model parallel process group * remove pipeline_prev/next group (#88) * micro batch offload * optimize pipeline gpu memory usage * pipeline can receive tensor shape (#93) * optimize pipeline gpu memory usage * fix grad accumulation step counter * rename classes and functions Co-authored-by: Frank Lee <somerlee.9@gmail.com>	3 years ago
アマデウス	01a80cd86d	Hotfix/Colossalai layers (#92) ... * optimized 1d layer apis; reorganized nn.layer modules; fixed tests * fixed 2.5d runtime issue * reworked split batch, now called in trainer.schedule.load_batch Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>	3 years ago
アマデウス	0fedef4f3c	Layer integration (#83) ... * integrated parallel layers for ease of building models * integrated 2.5d layers * cleaned codes and unit tests * added log metric by step hook; updated imagenet benchmark; fixed some bugs * reworked initialization; cleaned codes Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>	3 years ago
shenggan	5c3843dc98	add colossalai kernel module (#55)	3 years ago
ver217	8f02a88db2	add interleaved pipeline, fix naive amp and update pipeline model initializer (#80)	3 years ago
Frank Lee	91c327cb44	fixed zero level 3 dtype bug (#76)	3 years ago
HELSON	632e622de8	overlap computation and communication in 2d operations (#75)	3 years ago
Frank Lee	cd9c28e055	added CI for unit testing (#69)	3 years ago
Frank Lee	35813ed3c4	update examples and sphnix docs for the new api (#63)	3 years ago
ver217	7d3711058f	fix zero3 fp16 and add zero3 model context (#62)	3 years ago
Frank Lee	9a0466534c	update markdown docs (english) (#60)	3 years ago
Frank Lee	da01c234e1	Develop/experiments (#59) ... * Add gradient accumulation, fix lr scheduler * fix FP16 optimizer and adapted torch amp with tensor parallel (#18) * fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes * fixed trainer * Revert "fixed trainer" This reverts commit 2e0b0b7699. * improved consistency between trainer, engine and schedule (#23) Co-authored-by: 1SAA <c2h214748@gmail.com> * Split conv2d, class token, positional embedding in 2d, Fix random number in ddp Fix convergence in cifar10, Imagenet1000 * Integrate 1d tensor parallel in Colossal-AI (#39) * fixed 1D and 2D convergence (#38) * optimized 2D operations * fixed 1D ViT convergence problem * Feature/ddp (#49) * remove redundancy func in setup (#19) (#20) * use env to control the language of doc (#24) (#25) * Support TP-compatible Torch AMP and Update trainer API (#27) * Add gradient accumulation, fix lr scheduler * fix FP16 optimizer and adapted torch amp with tensor parallel (#18) * fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes * fixed trainer * Revert "fixed trainer" This reverts commit 2e0b0b7699. * improved consistency between trainer, engine and schedule (#23) Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: ver217 <lhx0217@gmail.com> * add an example of ViT-B/16 and remove w_norm clipping in LAMB (#29) * add explanation for ViT example (#35) (#36) * support torch ddp * fix loss accumulation * add log for ddp * change seed * modify timing hook Co-authored-by: Frank Lee <somerlee.9@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: binmakeswell <binmakeswell@gmail.com> * Feature/pipeline (#40) * remove redundancy func in setup (#19) (#20) * use env to control the language of doc (#24) (#25) * Support TP-compatible Torch AMP and Update trainer API (#27) * Add gradient accumulation, fix lr scheduler * fix FP16 optimizer and adapted torch amp with tensor parallel (#18) * fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes * fixed trainer * Revert "fixed trainer" This reverts commit 2e0b0b7699. * improved consistency between trainer, engine and schedule (#23) Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: ver217 <lhx0217@gmail.com> * add an example of ViT-B/16 and remove w_norm clipping in LAMB (#29) * add explanation for ViT example (#35) (#36) * optimize communication of pipeline parallel * fix grad clip for pipeline Co-authored-by: Frank Lee <somerlee.9@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: binmakeswell <binmakeswell@gmail.com> * optimized 3d layer to fix slow computation ; tested imagenet performance with 3d; reworked lr_scheduler config definition; fixed launch args; fixed some printing issues; simplified apis of 3d layers (#51) * Update 2.5d layer code to get a similar accuracy on imagenet-1k dataset * update api for better usability (#58) update api for better usability Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: puck_WCR <46049915+WANG-CR@users.noreply.github.com> Co-authored-by: binmakeswell <binmakeswell@gmail.com> Co-authored-by: アマデウス <kurisusnowdeng@users.noreply.github.com> Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>	3 years ago
ver217	dbe62c67b8	add an example of ViT-B/16 and remove w_norm clipping in LAMB (#29)	3 years ago
Frank Lee	3defa32aee	Support TP-compatible Torch AMP and Update trainer API (#27) ... * Add gradient accumulation, fix lr scheduler * fix FP16 optimizer and adapted torch amp with tensor parallel (#18) * fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes * fixed trainer * Revert "fixed trainer" This reverts commit 2e0b0b7699. * improved consistency between trainer, engine and schedule (#23) Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: ver217 <lhx0217@gmail.com>	3 years ago
ver217	3c7604ba30	update documentation	3 years ago
zbian	404ecbdcc6	Migrated project	3 years ago