3705 Commits (5ecc27e1509575e605c47279f243491e20968558)

 

All Branches   

Search

Author	SHA1	Message	Date
wangbluo	5ecc27e150	fix	2 months ago
wangbluo	f98384aef6	fix	2 months ago
wangbluo	b635dd0669	fix	2 months ago
wangbluo	3532f77b90	fix	2 months ago
wangbluo	3fab92166e	fix	2 months ago
wangbluo	6705dad41b	fix	2 months ago
wangbluo	91ed32c256	fix	2 months ago
wangbluo	6fb1322db1	fix	2 months ago
wangbluo	65c8297710	fix the attn	2 months ago
wangbluo	cfd9eda628	fix the ring attn	2 months ago
wangbluo	10e4f7da72	fix	2 months ago
Wang Binluo	37e35230ff	Merge pull request #6061 from wangbluo/sp_fix ... [sp] : fix the attention kernel for sp	2 months ago
wangbluo	827ef3ee9a	fix	2 months ago
Guangyao Zhang	bdb125f83f	[doc] FP8 training and communication document (#6050) ... * Add FP8 training and communication document * add fp8 docstring for plugins * fix typo * fix typo	2 months ago
Guangyao Zhang	f20b066c59	[fp8] Disable all_gather intranode. Disable Redundant all_gather fp8 (#6059) ... * all_gather only internode, fix pytest * fix cuda arch <89 compile pytest error * fix pytest failure * disable all_gather_into_tensor_flat_fp8 * fix fp8 format * fix pytest * fix conversations * fix chunk tuple to list	2 months ago
wangbluo	b582319273	fix	2 months ago
wangbluo	0ad3129cb9	fix	2 months ago
wangbluo	0b14a5512e	fix	2 months ago
botbw	696fced0d7	[fp8] fix missing fp8_comm flag in mixtral (#6057)	2 months ago
wangbluo	dc032172c3	fix	2 months ago
wangbluo	f393867cff	fix	2 months ago
wangbluo	6eb8832366	fix	2 months ago
wangbluo	683179cefd	fix	2 months ago
wangbluo	0a01e2a453	fix the attn	2 months ago
pre-commit-ci[bot]	216d54e374	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2 months ago
wangbluo	fdd84b9087	fix the sp	2 months ago
flybird11111	a35a078f08	[doc] update sp doc (#6055) ... * update sp doc * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * fix * fix --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2 months ago
Hongxin Liu	13946c4448	[fp8] hotfix backward hook (#6053) ... * [fp8] hotfix backward hook * [fp8] hotfix pipeline loss accumulation	2 months ago
botbw	c54c4fcd15	[hotfix] moe hybrid parallelism benchmark & follow-up fix (#6048) ... * [example] pass use_fp8_comm flag to all plugins * [example] add mixtral benchmark * [moe] refine assertion and check * [moe] fix mixtral & add more tests * [moe] consider checking dp * sp group and moe_dp_group * [mixtral] remove gate tp & add more tests * [deepseek] fix tp & sp for deepseek * [mixtral] minor fix * [deepseek] add deepseek benchmark	3 months ago
Wenxuan Tan	8fd25d6e09	[Feature] Split cross-entropy computation in SP (#5959) ... * halfway * fix cross-PP-stage position id length diff bug * fix typo * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * unified cross entropy func for all shardformer models * remove redundant lines * add basic ring attn; debug cross entropy * fwd bwd logic complete * fwd bwd logic complete; add experimental triton rescale * precision tests passed * precision tests passed * fix typos and remove misc files * update softmax_lse shape by new interface * change tester name * remove buffer clone; support packed seq layout * add varlen tests * fix typo * all tests passed * add dkv_group; fix mask * remove debug statements * adapt chatglm, command-R, qwen * debug * halfway * fix cross-PP-stage position id length diff bug * fix typo * fix typo * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * unified cross entropy func for all shardformer models * remove redundant lines * add basic ring attn; debug cross entropy * fwd bwd logic complete * fwd bwd logic complete; add experimental triton rescale * precision tests passed * precision tests passed * fix typos and remove misc files * add sp_mode to benchmark; fix varlen interface * update softmax_lse shape by new interface * add varlen tests * fix typo * all tests passed * add dkv_group; fix mask * remove debug statements * add comments * q1 index only once * remove events to simplify stream sync * simplify forward/backward logic * 2d ring forward passed * 2d ring backward passed * fixes * fix ring attn loss * 2D ring backward + llama passed * merge * update logger * fix typo * rebase * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix typo * remove typos * fixes * support GPT --------- Co-authored-by: Edenzzzz <wtan45@wisc.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	3 months ago
Hongxin Liu	b3db1058ec	[release] update version (#6041) ... * [release] update version * [devops] update comp test * [devops] update comp test debug * [devops] debug comp test * [devops] debug comp test * [devops] debug comp test * [devops] debug comp test * [devops] debug comp test	3 months ago
Hanks	5ce6dd75bf	[fp8] disable all_to_all_fp8 in intranode (#6045) ... * enhance all_to_all_fp8 with internode comm control * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * disable some fp8 ops due to performance issue * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	3 months ago
Hongxin Liu	26e553937b	[fp8] fix linear hook (#6046)	3 months ago
Hongxin Liu	c3b5caff0e	[fp8] optimize all-gather (#6043) ... * [fp8] optimize all-gather * [fp8] fix all gather fp8 ring * [fp8] enable compile * [fp8] fix all gather fp8 ring	3 months ago
Tong Li	c650a906db	[Hotfix] Remove deprecated install (#6042) ... * remove deprecated install * remove unused folder	3 months ago
Gao, Ruiyuan	e9032fb0b2	[colossalai/checkpoint_io/...] fix bug in load_state_dict_into_model; format error msg (#6020) ... * fix bug in load_state_dict_into_model; format error msg * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py to support checking missing_keys * Update general_checkpoint_io.py fix bug in missing_keys error message * retrigger tests --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	3 months ago
Guangyao Zhang	e96a0761ea	[FP8] unsqueeze scale to make it compatible with torch.compile (#6040)	3 months ago
Tong Li	0d3a85d04f	add fused norm (#6038)	3 months ago
Tong Li	4a68efb7da	[Colossal-LLaMA] Refactor latest APIs (#6030) ... * refactor latest code * update api * add dummy dataset * update Readme * add setup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update files * add PP support * update arguments * update argument * reorg folder * update version * remove IB infor * update utils * update readme * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update save for zero * update save * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add apex * update --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	3 months ago
Hongxin Liu	cc1b0efc17	[plugin] hotfix zero plugin (#6036) ... * [plugin] hotfix zero plugin * [plugin] hotfix zero plugin	3 months ago
Wenxuan Tan	d383449fc4	[CI] Remove triton version for compatibility bug; update req torch >=2.2 (#6018) ... * remove triton version * remove torch 2.2 * remove torch 2.1 * debug * remove 2.1 build tests * require torch >=2.2 --------- Co-authored-by: Edenzzzz <wtan45@wisc.edu>	3 months ago
Hongxin Liu	17904cb5bf	Merge pull request #6012 from hpcaitech/feature/fp8_comm ... [fp8] support fp8 communication and fp8 training for Colossalai	3 months ago
Wang Binluo	4a6f31eb0c	Merge pull request #6033 from wangbluo/fix ... [fp8] fix the merge	3 months ago
pre-commit-ci[bot]	80d24ae519	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	3 months ago
wangbluo	dae39999d7	fix	3 months ago
Wenxuan Tan	7cf9df07bc	[Hotfix] Fix llama fwd replacement bug (#6031) ... Co-authored-by: Edenzzzz <wtan45@wisc.edu>	3 months ago
Wang Binluo	0bf46c54af	Merge pull request #6029 from hpcaitech/flybird11111-patch-1 ... Update train_dpo.py	3 months ago
flybird11111	9e767643dd	Update low_level_zero_plugin.py	3 months ago
pre-commit-ci[bot]	3b0df30362	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	3 months ago
flybird11111	0bc9a870c0	Update train_dpo.py	3 months ago