ColossalAI

Commit Graph

Author	SHA1	Message	Date
Jiarui Fang	11bddb6e55	[zero] update zero context init with the updated test utils (#327 )	3 years ago
HELSON	4f26fabe4f	fixed strings in profiler outputs (#325 )	3 years ago
Jiarui Fang	de0468c7a8	[zero] zero init context (#321 ) * add zero init context * add more flags for zero init context fix bug of repeated converting param to ShardedParamV2 * polish code	3 years ago
1SAA	73bff11288	Added profiler communication operations Fixed bug for learning rate scheduler	3 years ago
LuGY	a3269de5c9	[zero] cpu adam kernel (#288 ) * Added CPU Adam * finished the cpu adam * updated the license * delete useless parameters, removed resnet * modified the method off cpu adam unittest * deleted some useless codes * removed useless codes Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com> Co-authored-by: jiaruifang <fangjiarui123@gmail.com>	3 years ago
Jiarui Fang	90d3aef62c	[zero] yet an improved sharded param (#311 )	3 years ago
Jiarui Fang	c9e7d9582d	[zero] polish shard strategy (#310 ) * init shard param from shape tuple * add more unitest for shard param * add set_payload method for ShardedParam * [zero] add shareded tensor class * polish code * add shard stratgy * move shard and gather logic to shard strategy from shard tensor. * polish code	3 years ago
ver217	3092317b80	polish code	3 years ago
ver217	36f9a74ab2	fix sharded param hook and unit test	3 years ago
ver217	001ca624dd	impl shard optim v2 and add unit test	3 years ago
Jiarui Fang	74f77e314b	[zero] a shard strategy in granularity of tensor (#307 )	3 years ago
Jiarui Fang	80364c7686	[zero] sharded tensor (#305 ) * init shard param from shape tuple * add more unitest for shard param * add set_payload method for ShardedParam * [zero] add shareded tensor class * polish code	3 years ago
Jie Zhu	d344689274	[profiler] primary memory tracer	3 years ago
ver217	b105371ace	rename shared adam to sharded optim v2	3 years ago
ver217	70814dc22f	fix master params dtype	3 years ago
ver217	795210dd99	add fp32 master params in sharded adam	3 years ago
ver217	a109225bc2	add sharded adam	3 years ago
Jiarui Fang	e17e92c54d	Polish sharded parameter (#297 ) * init shard param from shape tuple * add more unitest for shard param * add more unittests to shareded param	3 years ago
ver217	7aef75ca42	[zero] add sharded grad and refactor grad hooks for ShardedModel (#287 )	3 years ago
Frank Lee	9afb5c8b2d	fixed typo in ShardParam (#294 )	3 years ago
Frank Lee	e17e54e32a	added buffer sync to naive amp model wrapper (#291 )	3 years ago
Jiarui Fang	8d653af408	add a common util for hooks registered on parameter. (#292 )	3 years ago
Jie Zhu	f867365aba	bug fix: pass hook_list to engine (#273 ) * bug fix: pass hook_list to engine * change parameter name	3 years ago
Jiarui Fang	5a560a060a	Feature/zero (#279 ) * add zero1 (#209) * add zero1 * add test zero1 * update zero stage 1 develop (#212) * Implement naive zero3 (#240) * naive zero3 works well * add zero3 param manager * add TODOs in comments * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * fix bugs of hook and add unit tests (#252) * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * polish code and add state dict hook * fix bug * update unit test * refactor reconstructed zero code * clip_grad support zero3 and add unit test * add unit test for Zero3ParameterManager * [WIP] initialize the shard param class * [WIP] Yet another sharded model implementation (#274) * [WIP] initialize the shard param class * [WIP] Yes another implementation of shardModel. Using a better hook method. * torch.concat -> torch.cat * fix test_zero_level_1.py::test_zero_level_1 unitest * remove deepspeed implementation and refactor for the reconstructed zero module * polish zero dp unittests Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com>	3 years ago
1SAA	82023779bb	Added TPExpert for special situation	3 years ago
HELSON	36b8477228	Fixed parameter initialization in FFNExpert (#251 )	3 years ago
アマデウス	e13293bb4c	fixed CI dataset directory; fixed import error of 2.5d accuracy (#255 )	3 years ago
1SAA	219df6e685	Optimized MoE layer and fixed some bugs; Decreased moe tests; Added FFNExperts and ViTMoE model	3 years ago
zbian	3dba070580	fixed padding index issue for vocab parallel embedding layers; updated 3D linear to be compatible with examples in the tutorial	3 years ago
Frank Lee	f5ca88ec97	fixed apex import (#227 )	3 years ago
Frank Lee	3a1a9820b0	fixed mkdir conflict and align yapf config with flake (#220 )	3 years ago
アマデウス	9ee197d0e9	moved env variables to global variables; (#215 ) added branch context; added vocab parallel layers; moved split_batch from load_batch to tensor parallel embedding layers; updated gpt model; updated unit test cases; fixed few collective communicator bugs	3 years ago
Frank Lee	812357d63c	fixed utils docstring and add example to readme (#200 )	3 years ago
Frank Lee	765db512b5	fixed ddp bug on torch 1.8 (#194 )	3 years ago
Jiarui Fang	569357fea0	add pytorch hooks (#179 ) * add pytorch hooks fix #175 * remove licenses in src code * add gpu memory tracer * replacing print with logger in ophooks.	3 years ago
ver217	708404d5f8	fix pipeline forward return tensors (#176 )	3 years ago
HELSON	0f8c7f9804	Fixed docstring in colossalai (#171 )	3 years ago
Frank Lee	e2089c5c15	adapted for sequence parallel (#163 )	3 years ago
puck_WCR	9473a1b9c8	AMP docstring/markdown update (#160 )	3 years ago
Frank Lee	f3802d6b06	fixed jit default setting (#154 )	3 years ago
ver217	7bf1e98b97	pipeline last stage supports multi output (#151 )	3 years ago
ver217	f68eddfb3d	refactor kernel (#142 )	3 years ago
BoxiangW	4a3d3446b0	Update layer integration documentations (#108 ) Update the documentations of layer integration Update _log_hook.py Update _operation.py	3 years ago
ver217	9ef05ed1fc	try import deepspeed when using zero (#130 )	3 years ago
HELSON	dceae85195	Added MoE parallel (#127 )	3 years ago
ver217	293fb40c42	add scatter/gather optim for pipeline (#123 )	3 years ago
Jiarui Fang	2c0c85d3d3	fix a bug in timer (#114 )	3 years ago
ver217	7904baf6e1	fix layers/schedule for hybrid parallelization (#111 ) (#112 )	3 years ago
ver217	a951bc6089	update default logger (#100 ) (#101 )	3 years ago
ver217	96780e6ee4	Optimize pipeline schedule (#94 ) * add pipeline shared module wrapper and update load batch * added model parallel process group for amp and clip grad (#86) * added model parallel process group for amp and clip grad * update amp and clip with model parallel process group * remove pipeline_prev/next group (#88) * micro batch offload * optimize pipeline gpu memory usage * pipeline can receive tensor shape (#93) * optimize pipeline gpu memory usage * fix grad accumulation step counter * rename classes and functions Co-authored-by: Frank Lee <somerlee.9@gmail.com>	3 years ago

1 2

65 Commits (dec24561cf4048219dac98401b70e9fc35e985ad)