21 Commits (5774fe02706427532597acb94147d1134a40c2f8)

Author	SHA1	Message	Date
ver217	7c70bfbefa	[hotfix] fix PipelineSharedModuleGradientHandler (#1314)	2 years ago
Frank Lee	11f54c7b6b	[doc] improved docstring and assertion messages for the engine module (#871)	3 years ago
doubleHU	f2da21a827	fix format (#586)	3 years ago
fanjinfucool	ffad81e1d1	fix format (#585) ... Co-authored-by: fanjifu <FAN>	3 years ago
Jiarui Fang	e956d93ac2	[refactor] memory utils (#577)	3 years ago
HELSON	e6d50ec107	[zero] adapt zero for unsharded parameters (#561) ... * support existing sharded and unsharded parameters in zero * add unitest for moe-zero model init * polish moe gradient handler	3 years ago
Liang Bowen	ec5086c49c	Refactored docstring to google style	3 years ago
Jiarui Fang	a445e118cf	[polish] polish singleton and global context (#500)	3 years ago
Jiarui Fang	65c0f380c2	[format] polish name format for MOE (#481)	3 years ago
ver217	8d3250d74b	[zero] ZeRO supports pipeline parallel (#477)	3 years ago
HELSON	aff9d354f7	[MOE] polish moe_env (#467)	3 years ago
HELSON	84fd7c1d4d	add moe context, moe utilities and refactor gradient handler (#455)	3 years ago
yuxuan-lou	3b88eb2259	Flake8 code restyle	3 years ago
Jiarui Fang	5a560a060a	Feature/zero (#279) ... * add zero1 (#209) * add zero1 * add test zero1 * update zero stage 1 develop (#212) * Implement naive zero3 (#240) * naive zero3 works well * add zero3 param manager * add TODOs in comments * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * fix bugs of hook and add unit tests (#252) * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * polish code and add state dict hook * fix bug * update unit test * refactor reconstructed zero code * clip_grad support zero3 and add unit test * add unit test for Zero3ParameterManager * [WIP] initialize the shard param class * [WIP] Yet another sharded model implementation (#274) * [WIP] initialize the shard param class * [WIP] Yes another implementation of shardModel. Using a better hook method. * torch.concat -> torch.cat * fix test_zero_level_1.py::test_zero_level_1 unitest * remove deepspeed implementation and refactor for the reconstructed zero module * polish zero dp unittests Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com>	3 years ago
アマデウス	9ee197d0e9	moved env variables to global variables; (#215) ... added branch context; added vocab parallel layers; moved split_batch from load_batch to tensor parallel embedding layers; updated gpt model; updated unit test cases; fixed few collective communicator bugs	3 years ago
HELSON	0f8c7f9804	Fixed docstring in colossalai (#171)	3 years ago
Frank Lee	e2089c5c15	adapted for sequence parallel (#163)	3 years ago
HELSON	dceae85195	Added MoE parallel (#127)	3 years ago
ver217	96780e6ee4	Optimize pipeline schedule (#94) ... * add pipeline shared module wrapper and update load batch * added model parallel process group for amp and clip grad (#86) * added model parallel process group for amp and clip grad * update amp and clip with model parallel process group * remove pipeline_prev/next group (#88) * micro batch offload * optimize pipeline gpu memory usage * pipeline can receive tensor shape (#93) * optimize pipeline gpu memory usage * fix grad accumulation step counter * rename classes and functions Co-authored-by: Frank Lee <somerlee.9@gmail.com>	3 years ago
ver217	8f02a88db2	add interleaved pipeline, fix naive amp and update pipeline model initializer (#80)	3 years ago
zbian	404ecbdcc6	Migrated project	3 years ago