ColossalAI

Commit Graph

Author	SHA1	Message	Date
Hongxin Liu	5ddad486ca	[fp8] add fallback and make compile option configurable (#6092 )	1 month ago
Guangyao Zhang	f20b066c59	[fp8] Disable all_gather intranode. Disable Redundant all_gather fp8 (#6059 ) * all_gather only internode, fix pytest * fix cuda arch <89 compile pytest error * fix pytest failure * disable all_gather_into_tensor_flat_fp8 * fix fp8 format * fix pytest * fix conversations * fix chunk tuple to list	2 months ago
Hanks	5ce6dd75bf	[fp8] disable all_to_all_fp8 in intranode (#6045 ) * enhance all_to_all_fp8 with internode comm control * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * disable some fp8 ops due to performance issue * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2 months ago
Hongxin Liu	c3b5caff0e	[fp8] optimize all-gather (#6043 ) * [fp8] optimize all-gather * [fp8] fix all gather fp8 ring * [fp8] enable compile * [fp8] fix all gather fp8 ring	3 months ago
Guangyao Zhang	e96a0761ea	[FP8] unsqueeze scale to make it compatible with torch.compile (#6040 )	3 months ago
botbw	1a2e90dcc1	[fp8] linear perf enhancement	3 months ago
botbw	88fa096d78	[fp8] update torch.compile for linear_fp8 to >= 2.4.0 (#6004 )	3 months ago
flybird11111	597b206001	[fp8] support asynchronous FP8 communication (#5997 ) * fix * fix * fix * support async all2all * support async op for all gather * fix * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	3 months ago
Hongxin Liu	0978080a69	[fp8] refactor fp8 linear with compile (#5993 ) * [fp8] refactor fp8 linear with compile * [fp8] fix linear test * [fp8] fix linear test	3 months ago
flybird11111	f1a3a326c4	[fp8]Moe support fp8 communication (#5977 ) * fix * support moe fp8 * fix * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * fix * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * fix * fix fix fi * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	4 months ago
botbw	e4aadeee20	[fp8] use torch compile (torch >= 2.3.0) (#5979 ) * [fp8] use torch compile (torch >= 2.4.0) * [fp8] set use_fast_accum in linear * [chore] formal version check * [chore] fix sig	4 months ago
Hongxin Liu	8241c0c054	[fp8] support gemini plugin (#5978 ) * [fp8] refactor hook * [fp8] support gemini plugin * [example] add fp8 option for llama benchmark	4 months ago
Hanks	b480eec738	[Feature]: support FP8 communication in DDP, FSDP, Gemini (#5928 ) * support fp8_communication in the Torch DDP grad comm, FSDP grad comm, and FSDP params comm * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * implement communication hook for FSDP params all-gather * added unit test for fp8 operators * support fp8 communication in GeminiPlugin * update training scripts to support fsdp and fp8 communication * fixed some minor bugs observed in unit test * add all_gather_into_tensor_flat_fp8 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add skip the test if torch < 2.2.0 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add skip the test if torch < 2.2.0 * add skip the test if torch < 2.2.0 * add fp8_comm flag * rebase latest fp8 operators * rebase latest fp8 operators * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	4 months ago
flybird11111	7739629b9d	fix (#5976 )	4 months ago
Hongxin Liu	ccabcf6485	[fp8] support fp8 amp for hybrid parallel plugin (#5975 ) * [fp8] support fp8 amp for hybrid parallel plugin * [test] add fp8 hook test * [fp8] fix fp8 linear compatibility	4 months ago
Hongxin Liu	76ea16466f	[fp8] add fp8 linear (#5967 ) * [fp8] add fp8 linear * [test] fix fp8 linear test condition * [test] fix fp8 linear test condition * [test] fix fp8 linear test condition	4 months ago
flybird11111	afb26de873	[fp8]support all2all fp8 (#5953 ) * support all2all fp8 * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * fix * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	4 months ago
Guangyao Zhang	53cb9606bd	[Feature] llama shardformer fp8 support (#5938 ) * add llama shardformer fp8 * Llama Shardformer Parity * fix typo * fix all reduce * fix pytest failure * fix reduce op and move function to fp8.py * fix typo	4 months ago
Hongxin Liu	5fd0592767	[fp8] support all-gather flat tensor (#5932 )	4 months ago
GuangyaoZhang	6a20f07b80	remove all to all	4 months ago
GuangyaoZhang	5a310b9ee1	fix rebase	4 months ago
GuangyaoZhang	457a0de79f	shardformer fp8	4 months ago
pre-commit-ci[bot]	51f916b11d	[pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci	4 months ago
BurkeHulk	e88190184a	support fp8 communication in pipeline parallelism	4 months ago
BurkeHulk	1e1959467e	fix scaling algorithm in FP8 casting	4 months ago
GuangyaoZhang	dbfa7d39fc	fix typo	5 months ago
pre-commit-ci[bot]	e17f835df7	[pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci	5 months ago
HangXu	f5a52e1600	fp8 operators for compressed communication cast_to_fp8, cast_from_fp8, all_reduce_fp8	5 months ago

26 Commits (pre-commit-ci-update-config)