264 Commits (19d1510ea26d10484a804eb62f6d03dbcc7c80a8)

Author	SHA1	Message	Date
Jianghai	1094e0f0d3	[pipeline] Bert pipeline for shardformer and its tests (#4197) ... * add pipeline forward * complete pipeline forward check * fix bert forward without pipeline * fix comments * discard useless line * add todo * clean prints * fix distribute layers	1 year ago
Hongxin Liu	890774b2fb	[shardformer] support lazy init (#4202) ... * [shardformer] support lazy init * [shardformer] linear support lazy init * [shardformer] embedding support lazy init * [shardformer] norm support lazy init * [shardformer] fused linear support lazy init * [test] update shardformer test layer * [test] shardformer with lazy init fit ddp * [lazy] hotfix deepcopy of param * [shardformer] fix bert policy and update test * [shardformer] fix bloom policy and update test * [shardformer] fix opt policy and update test * [shardformer] fix t5 policy and update test * [shardformer] fix gpt2 policy and update test * [shardformer] fix llama policy and update test	1 year ago
Jianghai	f3bcc292c8	[pipeline] move bert related pipeline components to shardformer (#4187) ... * move bert related pipeline components to shardformer * fix bugs * revision * fix bert model tests * fix bert_lm_head model tests * fix tests * fix tests * done checks * skip bloom	1 year ago
ver217	d35bd7d0e6	[shardformer] fix type hint	1 year ago
ver217	1ed3f8a24f	[shardformer] rename policy file name	1 year ago
ver217	b0b8ad2823	[pipeline] update shardformer docstring	1 year ago
ver217	59f6f573f1	[pipeline] update shardformer policy	1 year ago
Jianghai	9a4842c571	revise shardformer readme (#4246)	1 year ago
digger yu	2ac24040eb	fix some typo colossalai/shardformer (#4160)	1 year ago
github-actions[bot]	c77b3b19be	[format] applied code formatting on changed files in pull request 4152 (#4157) ... Co-authored-by: github-actions <github-actions@github.com>	1 year ago
Frank Lee	89f45eda5a	[shardformer] added development protocol for standardization (#4149)	1 year ago
Frank Lee	1fb0d95df0	[shardformer] made tensor parallelism configurable (#4144) ... * [shardformer] made tensor parallelism configurable * polish code	1 year ago
Frank Lee	74257cb446	[shardformer] refactored some doc and api (#4137) ... * [shardformer] refactored some doc and api * polish code	1 year ago
jiangmingyan	7f9b30335b	[shardformer] write an shardformer example with bert finetuning (#4126) ... * [shardformer] add benchmark of shardformer * [shardformer] add benchmark of shardformer	1 year ago
Frank Lee	ae035d305d	[shardformer] added embedding gradient check (#4124)	1 year ago
Frank Lee	44a190e6ac	[shardformer] import huggingface implicitly (#4101)	1 year ago
Frank Lee	6a88bae4ec	[shardformer] integrate with data parallelism (#4103)	1 year ago
Frank Lee	f3b6aaa6b7	[shardformer] supported fused normalization (#4112)	1 year ago
Frank Lee	b1c2901530	[shardformer] supported bloom model (#4098)	1 year ago
Kun Lin	8af29ee47a	[shardformer] support vision transformer (#4096) ... * first v of vit shardformer * keep vit * update * vit shard add vitattention vitlayer * update num head shard para * finish test for vit * add new_model_class & postprocess * add vit readme * delete old files & fix the conflict * fix sth	1 year ago
jiangmingyan	ac80937138	[shardformer] shardformer support opt models (#4091) ... * [shardformer] shardformer support opt models * [shardformer] shardformer support opt models, fix * [shardformer] shardformer support opt models, fix * [shardformer] shardformer support opt models, fix	1 year ago
Frank Lee	d33a44e8c3	[shardformer] refactored layernorm (#4086)	1 year ago
FoolPlayer	92f6791095	[shardformer] Add layernorm (#4072) ... * add layernorm to bert * add layernorm test * add layernorm test with load state dict * add use_mixedfusedLN in shard config * refactor policy to support fused_layernorm	1 year ago
Frank Lee	70c58cfd4f	[shardformer] supported fused qkv checkpoint (#4073)	1 year ago
FoolPlayer	0803a61412	[shardformer] add linearconv1d test (#4067) ... * add linearconv1d test * add linearconv1d test	1 year ago
Frank Lee	8eb09a4c69	[shardformer] support module saving and loading (#4062) ... * [shardformer] support module saving and loading * polish code	1 year ago
FoolPlayer	7740c55c55	support kit use for bert/gpt test (#4055) ... * support kit use for bert test * support kit test for gpt2	1 year ago
Frank Lee	f22ddacef0	[shardformer] refactored the shardformer layer structure (#4053)	1 year ago
Frank Lee	58df720570	[shardformer] adapted T5 and LLaMa test to use kit (#4049) ... * [shardformer] adapted T5 and LLaMa test to use kit * polish code	1 year ago
FoolPlayer	4021b9a8a2	[shardformer] add gpt2 test and layer class refactor (#4041) ... * add gpt2 test and layer class refactor * add dropout in gpt2 policy	1 year ago
Frank Lee	d857f3dbba	[shardformer] supported T5 and its variants (#4045)	1 year ago
Frank Lee	c1d5453e9f	[shardformer] adapted llama to the new API (#4036)	1 year ago
FoolPlayer	74d176c8d8	[shardformer] fix bert and gpt downstream with new api (#4024) ... * fix bert downstream with new api * remove comment line	1 year ago
Frank Lee	e253a07007	[shardformer] updated doc (#4016)	1 year ago
FoolPlayer	df018fc305	support bert with new api	1 year ago
FoolPlayer	507c0ad368	add vocabembedding layer	1 year ago
Frank Lee	45d9384346	[shardformer] removed inplace tensor sharding (#4018)	1 year ago
Frank Lee	3893fa1a8d	[shardformer] refactored embedding and dropout to parallel module (#4013) ... * [shardformer] refactored embedding and dropout to parallel module * polish code	1 year ago
FoolPlayer	dfca9678fa	integrate with dist layer (#4011)	1 year ago
Frank Lee	015af592f8	[shardformer] integrated linear 1D with dtensor (#3996) ... * [shardformer] integrated linear 1D with dtensor * polish code	1 year ago
FoolPlayer	d3bc530849	[shardformer] Refactor shardformer api (#4001) ... * fix an error in readme * simplify code * refactor shardformer * add todo * remove slicer * resolve code review	1 year ago
FoolPlayer	f7774ec0f3	[Shardformer] Downstream bert (#3979) ... * add dist dropout in model * update docstring and bert policy with dropout * refactor basepolicy and sharded, update bert * update format * update gpt2 policy * update bert policy * remove unused code * update readme for new policy usage * add downstream model of bert * remove unused code	1 year ago
wukong1992	c1c672d0f0	[shardformer] shardformer support t5 model (#3994) ... test t5	1 year ago
wukong1992	6b30dfb7ce	[shardformer] support llama model using shardformer (#3969) ... adjust layer attr	1 year ago
FoolPlayer	45927d5527	[shardformer] Add dropout layer in shard model and refactor policy api (#3949) ... * add dist dropout in model * update docstring and bert policy with dropout * refactor basepolicy and sharded, update bert * update format * update gpt2 policy * update bert policy * remove unused code * update readme for new policy usage	1 year ago
FoolPlayer	a73130482d	[shardformer] Unit test (#3928) ... * fix bug in slicer, add slicer unit test * add dropout test * use pid as dropout seed * updata dropout test with local pattern * ad todo	1 year ago
FoolPlayer	f1cb5ac6bf	[shardformer] Align bert value (#3907) ... * add bert align test, fix dist loss bug * forward and backward align * add ignore index * add shardformer CI * add gather_output optional for user in shardconfig * update readme with optional gather_ouput * add dist crossentropy loss test, remove unused files * remove unused file * remove unused file * rename the file * polish code	1 year ago
FoolPlayer	79f8d5d54b	[shardformer] add gpt2 policy and modify shard and slicer to support (#3883) ... * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code	1 year ago
FoolPlayer	70173e3123	update README (#3909)	1 year ago
FoolPlayer	ab8a47f830	[shardformer] add Dropout layer support different dropout pattern (#3856) ... * add dropout layer, add dropout test * modify seed manager as context manager * add a copy of col_nn.layer * add dist_crossentropy loss; separate module test * polish the code * fix dist crossentropy loss	1 year ago
FoolPlayer	c594dc2f1c	[shardformer] update readme with modules implement doc (#3834) ... * update readme with modules content * remove img	1 year ago
Frank Lee	4972e1f40e	[shardformer] refactored the user api (#3828) ... * [shardformer] refactored the user api * polish code	1 year ago
Frank Lee	235792f170	[shardformer] updated readme (#3827)	1 year ago
FoolPlayer	8cc11235c0	[shardformer]: Feature/shardformer, add some docstring and readme (#3816) ... * init shardformer code structure * add implement of sharder (inject and replace) * add implement of replace layer to colossal layer * separate different layer policy, add some notion * implement 1d and 2d slicer, can tell col or row * fix bug when slicing and inject model * fix some bug; add inference test example * add share weight and train example * add train * add docstring and readme * add docstring for other files * pre-commit	1 year ago
FoolPlayer	8d68de767d	[shardformer] init shardformer code structure (#3731) ... * init shardformer code structure * add implement of sharder (inject and replace) * add implement of replace layer to colossal layer * separate different layer policy, add some notion * implement 1d and 2d slicer, can tell col or row * fix bug when slicing and inject model * fix some bug; add inference test example	1 year ago
Frank Lee	ddcf58cacf	Revert "[sync] sync feature/shardformer with develop"	1 year ago
FoolPlayer	ef1537759c	[shardformer] add gpt2 policy and modify shard and slicer to support (#3883) ... * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code	1 year ago
FoolPlayer	6370a935f6	update README (#3909)	1 year ago
FoolPlayer	21a3915c98	[shardformer] add Dropout layer support different dropout pattern (#3856) ... * add dropout layer, add dropout test * modify seed manager as context manager * add a copy of col_nn.layer * add dist_crossentropy loss; separate module test * polish the code * fix dist crossentropy loss	1 year ago
FoolPlayer	997544c1f9	[shardformer] update readme with modules implement doc (#3834) ... * update readme with modules content * remove img	1 year ago
Frank Lee	537a52b7a2	[shardformer] refactored the user api (#3828) ... * [shardformer] refactored the user api * polish code	1 year ago
Frank Lee	bc19024bf9	[shardformer] updated readme (#3827)	1 year ago
FoolPlayer	58f6432416	[shardformer]: Feature/shardformer, add some docstring and readme (#3816) ... * init shardformer code structure * add implement of sharder (inject and replace) * add implement of replace layer to colossal layer * separate different layer policy, add some notion * implement 1d and 2d slicer, can tell col or row * fix bug when slicing and inject model * fix some bug; add inference test example * add share weight and train example * add train * add docstring and readme * add docstring for other files * pre-commit	1 year ago
FoolPlayer	6a69b44dfc	[shardformer] init shardformer code structure (#3731) ... * init shardformer code structure * add implement of sharder (inject and replace) * add implement of replace layer to colossal layer * separate different layer policy, add some notion * implement 1d and 2d slicer, can tell col or row * fix bug when slicing and inject model * fix some bug; add inference test example	1 year ago