github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
3,520 Commits
32 Branches
48 Tags
232 MiB
Commit Graph

3520 Commits (783aafa327f1fc1ed75a42ab01b7418351110599)

Author SHA1 Message Date
hxwang 783aafa327
[moe] full test for deepseek and mixtral (pp + sp to fix) 2024-07-19 07:32:56 +00:00
hxwang 162e2d935c
[moe] finalize test (no pp) 2024-07-19 07:32:56 +00:00
haze188 b91cdccf2e
moe sp + ep bug fix 2024-07-19 07:32:55 +00:00
hxwang 8e85523a42
[moe] init moe plugin comm setting with sp 2024-07-19 07:32:54 +00:00
hxwang f0599a0c19
[chore] minor fix 2024-07-19 07:32:02 +00:00
Haze188 633849f438
[Feature] MoE Ulysses Support (#5918) 
* moe sp support

* moe sp bug solve

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-07-19 07:32:01 +00:00
hxwang c8bf2681e3
[moe] clean legacy code 2024-07-19 07:32:01 +00:00
hxwang 8d3d7f3cbd
[moe] test deepseek 2024-07-19 07:32:00 +00:00
botbw 335ad3c6fb
[moe] implement tp 2024-07-19 07:30:17 +00:00
botbw d4a64e355e
[test] add mixtral modelling test 2024-07-19 07:30:16 +00:00
hxwang 18be903ed9
[chore] arg pass & remove drop token 2024-07-19 07:30:16 +00:00
botbw cbcc818d5a
[chore] trivial fix 2024-07-19 07:30:15 +00:00
botbw 5bc085fc01
[chore] manually revert unintended commit 2024-07-19 07:30:14 +00:00
botbw 1b15cc97f5
[moe] add mixtral dp grad scaling when not all experts are activated 2024-07-19 07:30:14 +00:00
botbw 2f9bce6686
[moe] implement submesh initialization 2024-07-19 07:30:13 +00:00
haze188 a613edd517
solve hang when parallel mode = pp + dp 2024-07-19 07:30:13 +00:00
haze188 0210bead8c
[misc] solve booster hang by rename the variable 2024-07-19 07:29:36 +00:00
botbw b303ffe9f3
[zero] solve hang 2024-07-19 07:29:36 +00:00
botbw 2431694564
[moe] implement transit between non moe tp and ep 2024-07-19 07:29:35 +00:00
botbw dec6e25e99
[test] pass mixtral shardformer test 2024-07-19 07:29:35 +00:00
hxwang 61109c7843
[zero] solve hang 2024-07-19 07:29:07 +00:00
hxwang 000456bf94
[chore] handle non member group 2024-07-19 07:29:07 +00:00
hxwang 4fc6f9aa98
[test] mixtra pp shard test 2024-07-19 07:29:06 +00:00
hxwang 5a9490a46b
[moe] fix plugin 2024-07-19 07:29:06 +00:00
hxwang 6a9164a477
[test] add mixtral transformer test 2024-07-19 07:29:05 +00:00
hxwang 229db4bc16
[test] add mixtral for sequence classification 2024-07-19 07:29:05 +00:00
Tong Li f585d4e38e
[ColossalChat] Hotfix for ColossalChat (#5910) 
* add ignore and tiny llama

* fix path issue

* run style

* fix issue

* update bash

* add ignore and tiny llama

* fix path issue

* run style

* fix issue

* update bash

* fix ddp issue

* add Qwen 1.5 32B
 2024-07-19 13:40:07 +08:00
Edenzzzz 8cc8f645cd
[Examples] Add lazy init to OPT and GPT examples (#5924) 
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
 2024-07-19 10:10:08 +08:00
Hongxin Liu e86127925a
[plugin] support all-gather overlap for hybrid parallel (#5919) 
* [plugin] fixed all-gather overlap support for hybrid parallel
 2024-07-18 15:33:03 +08:00
Hongxin Liu 73494de577
[release] update version (#5912) 2024-07-17 17:29:59 +08:00
Hongxin Liu 27a72f0de1 [misc] support torch2.3 (#5893) 
* [misc] support torch2.3

* [devops] update compatibility ci

* [devops] update compatibility ci

* [devops] add debug

* [devops] add debug

* [devops] add debug

* [devops] add debug

* [devops] remove debug

* [devops] remove debug
 2024-07-16 13:59:25 +08:00
アマデウス 530283dba0 fix object_to_tensor usage when torch>=2.3.0 (#5820) 2024-07-16 13:59:25 +08:00
Guangyao Zhang 2e28c793ce [compatibility] support torch 2.2 (#5875) 
* Support Pytorch 2.2.2

* keep build_on_pr file and update .compatibility
 2024-07-16 13:59:25 +08:00
YeAnbang d8bf7e09a2
Merge pull request #5901 from hpcaitech/colossalchat 
[Chat] fix eval: add in training evaluation, fix orpo sft loss bug
 2024-07-16 11:07:32 +08:00
Guangyao Zhang 1c961b20f3
[ShardFormer] fix qwen2 sp (#5903) 2024-07-15 13:58:06 +08:00
Stephan Kö 45c49dde96
[Auto Parallel]: Speed up intra-op plan generation by 44% (#5446) 
* Remove unnecessary calls to deepcopy

* Build DimSpec's difference dict only once

This change considerably speeds up construction speed of DimSpec objects. The difference_dict is the same for each DimSpec object, so a single copy of it is enough.

* Fix documentation of DimSpec's difference method
 2024-07-15 12:05:06 +08:00
YeAnbang b3594d4d68 fix orpo cross entropy loss 2024-07-15 02:12:05 +00:00
Hongxin Liu c068ef0fa0
[zero] support all-gather overlap (#5898) 
* [zero] support all-gather overlap

* [zero] add overlap all-gather flag

* [misc] fix typo

* [zero] update api
 2024-07-11 18:59:59 +08:00
YeAnbang 115c4cc5a4 hotfix citation 2024-07-11 06:05:05 +00:00
YeAnbang e7a8634636 fix eval 2024-07-11 03:35:03 +00:00
YeAnbang dd9e1cdafe
Merge pull request #5850 from hpcaitech/rlhf_SimPO 
[Chat] Rlhf support SimPO
 2024-07-11 09:14:12 +08:00
pre-commit-ci[bot] 8a9721bafe [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2024-07-10 10:44:32 +00:00
YeAnbang 33f15203d3 Merge branch 'main' of https://github.com/hpcaitech/ColossalAI into rlhf_SimPO 2024-07-10 10:39:34 +00:00
YeAnbang f6ef5c3609 fix style 2024-07-10 10:37:17 +00:00
YeAnbang d888c3787c add benchmark for sft, dpo, simpo, orpo. Add benchmarking result. Support lora with gradient checkpoint 2024-07-10 10:17:08 +00:00
Guangyao Zhang 669849d74b
[ShardFormer] Add Ulysses Sequence Parallelism support for Command-R, Qwen2 and ChatGLM (#5897) 2024-07-10 11:34:25 +08:00
YeAnbang 16f3451fe2 Merge branch 'main' of https://github.com/hpcaitech/ColossalAI into rlhf_SimPO 2024-07-10 02:32:07 +00:00
Edenzzzz fbf33ecd01
[Feature] Enable PP + SP for llama (#5868) 
* fix cross-PP-stage position id length diff bug

* fix typo

* fix typo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use a one cross entropy func for all shardformer models

---------

Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-07-09 18:05:20 +08:00
Runyu Lu 66abf1c6e8
[HotFix] CI,import,requirements-test for #5838 (#5892) 
* [Hot Fix] CI,import,requirements-test

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-07-08 22:32:06 +08:00
Runyu Lu cba20525a8
[Feat] Diffusion Model(PixArtAlpha/StableDiffusion3) Support (#5838) 
* Diffusion Model Inference support

* Stable Diffusion 3 Support

* pixartalpha support
 2024-07-08 16:02:07 +08:00