ColossalAI

Commit Graph

Author	SHA1	Message	Date
Hongxin Liu	8241c0c054	[fp8] support gemini plugin (#5978 ) * [fp8] refactor hook * [fp8] support gemini plugin * [example] add fp8 option for llama benchmark	2024-08-09 14:09:48 +08:00
Hanks	b480eec738	[Feature]: support FP8 communication in DDP, FSDP, Gemini (#5928 ) * support fp8_communication in the Torch DDP grad comm, FSDP grad comm, and FSDP params comm * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * implement communication hook for FSDP params all-gather * added unit test for fp8 operators * support fp8 communication in GeminiPlugin * update training scripts to support fsdp and fp8 communication * fixed some minor bugs observed in unit test * add all_gather_into_tensor_flat_fp8 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add skip the test if torch < 2.2.0 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add skip the test if torch < 2.2.0 * add skip the test if torch < 2.2.0 * add fp8_comm flag * rebase latest fp8 operators * rebase latest fp8 operators * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2024-08-08 15:55:01 +08:00
flybird11111	7739629b9d	fix (#5976 )	2024-08-07 18:58:39 +08:00
Hongxin Liu	ccabcf6485	[fp8] support fp8 amp for hybrid parallel plugin (#5975 ) * [fp8] support fp8 amp for hybrid parallel plugin * [test] add fp8 hook test * [fp8] fix fp8 linear compatibility	2024-08-07 18:21:08 +08:00
Hongxin Liu	76ea16466f	[fp8] add fp8 linear (#5967 ) * [fp8] add fp8 linear * [test] fix fp8 linear test condition * [test] fix fp8 linear test condition * [test] fix fp8 linear test condition	2024-08-07 15:41:49 +08:00
flybird11111	afb26de873	[fp8]support all2all fp8 (#5953 ) * support all2all fp8 * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * fix * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2024-08-06 16:58:23 +08:00
Guangyao Zhang	53cb9606bd	[Feature] llama shardformer fp8 support (#5938 ) * add llama shardformer fp8 * Llama Shardformer Parity * fix typo * fix all reduce * fix pytest failure * fix reduce op and move function to fp8.py * fix typo	2024-08-05 10:05:47 +08:00
Hongxin Liu	5fd0592767	[fp8] support all-gather flat tensor (#5932 )	2024-07-24 16:55:20 +08:00
GuangyaoZhang	6a20f07b80	remove all to all	2024-07-17 07:14:55 +00:00
GuangyaoZhang	5a310b9ee1	fix rebase	2024-07-17 03:43:23 +00:00
GuangyaoZhang	457a0de79f	shardformer fp8	2024-07-16 06:56:51 +00:00
pre-commit-ci[bot]	51f916b11d	[pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci	2024-07-12 07:33:45 +00:00
BurkeHulk	1f1b856354	Merge remote-tracking branch 'origin/feature/fp8_comm' into feature/fp8_comm # Conflicts: # colossalai/quantization/fp8.py	2024-07-12 15:29:41 +08:00
BurkeHulk	e88190184a	support fp8 communication in pipeline parallelism	2024-07-12 15:25:25 +08:00
BurkeHulk	1e1959467e	fix scaling algorithm in FP8 casting	2024-07-12 15:23:37 +08:00
GuangyaoZhang	dbfa7d39fc	fix typo	2024-07-10 08:13:26 +00:00
pre-commit-ci[bot]	e17f835df7	[pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci	2024-07-04 12:47:17 +00:00
Hanks	6991819a97	Merge branch 'hpcaitech:main' into feature/fp8_comm	2024-07-04 20:34:41 +08:00
Hongxin Liu	7afbc81d62	[quant] fix bitsandbytes version check (#5882 ) * [quant] fix bitsandbytes version check * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2024-07-04 11:33:23 +08:00
HangXu	f5a52e1600	fp8 operators for compressed communication cast_to_fp8, cast_from_fp8, all_reduce_fp8	2024-07-01 13:44:21 +08:00
linsj20	91fa553775	[Feature] qlora support (#5586 ) * [feature] qlora support * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * qlora follow commit * migrate qutization folder to colossalai/ * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor fixes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2024-04-28 10:51:27 +08:00

21 Commits (8241c0c054b38a109ed3ce7be1052a1e600b8471)