ff14144d9c
[tmp] add write_tensor
1 month ago
ad6558e91c
[chore] refactor
1 month ago
162251ab78
[ckpt] add safetensors util
1 month ago
4c8e85ee0d
[Coati] Train DPO using PP ( #6054 )
...
* update dpo
* remove unsupport plugin
* update msg
* update dpo
* remove unsupport plugin
* update msg
* update template
* update dataset
* add pp for dpo
* update dpo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add dpo fn
* update dpo
* update dpo
* update dpo
* update dpo
* minor update
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update loss
* update help
* polish code
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 month ago
dc2cdaf3e8
[shardformer] optimize seq parallelism ( #6086 )
...
* [shardformer] optimize seq parallelism
* [shardformer] fix gpt2 fused linear col
* [plugin] update gemini plugin
* [plugin] update moe hybrid plugin
* [test] update gpt2 fused linear test
* [shardformer] fix gpt2 fused linear reduce
1 month ago
6b2c506fc5
Update README.md ( #6087 )
...
add HPC-AI.COM activity
2 months ago
646b3c5a90
[shardformer] fix linear 1d row and support uneven splits for fused qkv linear ( #6084 )
...
* [tp] hotfix linear row
* [tp] support uneven split for fused linear
* [tp] support sp for fused linear
* [tp] fix gpt2 mlp policy
* [tp] fix gather fused and add fused linear row
2 months ago
f4daf04270
add funding news ( #6072 )
...
* add funding news
* add funding news
* add funding news
2 months ago
cbaa104216
release FP8 news ( #6068 )
...
* add FP8 news
* release FP8 news
* release FP8 news
2 months ago
dabc2e7430
[release] update version ( #6062 )
2 months ago
f9546ba0be
[ColossalEval] support for vllm ( #6056 )
...
* support vllm
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* modify vllm and update readme
* run pre-commit
* remove dupilicated lines and refine code
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update param name
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refine code
* update readme
* refine code
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2 months ago
4fa6b9509c
[moe] add parallel strategy for shared_expert && fix test for deepseek ( #6063 )
2 months ago
63314ce4e4
Merge pull request #6064 from wangbluo/fix_attn
...
[sp] : fix the attention kernel for sp
2 months ago
10e4f7da72
fix
2 months ago
37e35230ff
Merge pull request #6061 from wangbluo/sp_fix
...
[sp] : fix the attention kernel for sp
2 months ago
827ef3ee9a
fix
2 months ago
bdb125f83f
[doc] FP8 training and communication document ( #6050 )
...
* Add FP8 training and communication document
* add fp8 docstring for plugins
* fix typo
* fix typo
2 months ago
f20b066c59
[fp8] Disable all_gather intranode. Disable Redundant all_gather fp8 ( #6059 )
...
* all_gather only internode, fix pytest
* fix cuda arch <89 compile pytest error
* fix pytest failure
* disable all_gather_into_tensor_flat_fp8
* fix fp8 format
* fix pytest
* fix conversations
* fix chunk tuple to list
2 months ago
b582319273
fix
2 months ago
0ad3129cb9
fix
2 months ago
0b14a5512e
fix
2 months ago
696fced0d7
[fp8] fix missing fp8_comm flag in mixtral ( #6057 )
2 months ago
dc032172c3
fix
2 months ago
f393867cff
fix
2 months ago
6eb8832366
fix
2 months ago
683179cefd
fix
2 months ago
0a01e2a453
fix the attn
2 months ago
216d54e374
[pre-commit.ci] auto fixes from pre-commit.com hooks
...
for more information, see https://pre-commit.ci
2 months ago
fdd84b9087
fix the sp
2 months ago
a35a078f08
[doc] update sp doc ( #6055 )
...
* update sp doc
* fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix
* fix
* fix
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2 months ago
13946c4448
[fp8] hotfix backward hook ( #6053 )
...
* [fp8] hotfix backward hook
* [fp8] hotfix pipeline loss accumulation
2 months ago
c54c4fcd15
[hotfix] moe hybrid parallelism benchmark & follow-up fix ( #6048 )
...
* [example] pass use_fp8_comm flag to all plugins
* [example] add mixtral benchmark
* [moe] refine assertion and check
* [moe] fix mixtral & add more tests
* [moe] consider checking dp * sp group and moe_dp_group
* [mixtral] remove gate tp & add more tests
* [deepseek] fix tp & sp for deepseek
* [mixtral] minor fix
* [deepseek] add deepseek benchmark
3 months ago
8fd25d6e09
[Feature] Split cross-entropy computation in SP ( #5959 )
...
* halfway
* fix cross-PP-stage position id length diff bug
* fix typo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unified cross entropy func for all shardformer models
* remove redundant lines
* add basic ring attn; debug cross entropy
* fwd bwd logic complete
* fwd bwd logic complete; add experimental triton rescale
* precision tests passed
* precision tests passed
* fix typos and remove misc files
* update softmax_lse shape by new interface
* change tester name
* remove buffer clone; support packed seq layout
* add varlen tests
* fix typo
* all tests passed
* add dkv_group; fix mask
* remove debug statements
* adapt chatglm, command-R, qwen
* debug
* halfway
* fix cross-PP-stage position id length diff bug
* fix typo
* fix typo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unified cross entropy func for all shardformer models
* remove redundant lines
* add basic ring attn; debug cross entropy
* fwd bwd logic complete
* fwd bwd logic complete; add experimental triton rescale
* precision tests passed
* precision tests passed
* fix typos and remove misc files
* add sp_mode to benchmark; fix varlen interface
* update softmax_lse shape by new interface
* add varlen tests
* fix typo
* all tests passed
* add dkv_group; fix mask
* remove debug statements
* add comments
* q1 index only once
* remove events to simplify stream sync
* simplify forward/backward logic
* 2d ring forward passed
* 2d ring backward passed
* fixes
* fix ring attn loss
* 2D ring backward + llama passed
* merge
* update logger
* fix typo
* rebase
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix typo
* remove typos
* fixes
* support GPT
---------
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
3 months ago
b3db1058ec
[release] update version ( #6041 )
...
* [release] update version
* [devops] update comp test
* [devops] update comp test debug
* [devops] debug comp test
* [devops] debug comp test
* [devops] debug comp test
* [devops] debug comp test
* [devops] debug comp test
3 months ago
5ce6dd75bf
[fp8] disable all_to_all_fp8 in intranode ( #6045 )
...
* enhance all_to_all_fp8 with internode comm control
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* disable some fp8 ops due to performance issue
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
3 months ago
26e553937b
[fp8] fix linear hook ( #6046 )
3 months ago
c3b5caff0e
[fp8] optimize all-gather ( #6043 )
...
* [fp8] optimize all-gather
* [fp8] fix all gather fp8 ring
* [fp8] enable compile
* [fp8] fix all gather fp8 ring
3 months ago
c650a906db
[Hotfix] Remove deprecated install ( #6042 )
...
* remove deprecated install
* remove unused folder
3 months ago
e9032fb0b2
[colossalai/checkpoint_io/...] fix bug in load_state_dict_into_model; format error msg ( #6020 )
...
* fix bug in load_state_dict_into_model; format error msg
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update utils.py
to support checking missing_keys
* Update general_checkpoint_io.py
fix bug in missing_keys error message
* retrigger tests
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
3 months ago
e96a0761ea
[FP8] unsqueeze scale to make it compatible with torch.compile ( #6040 )
3 months ago
0d3a85d04f
add fused norm ( #6038 )
3 months ago
4a68efb7da
[Colossal-LLaMA] Refactor latest APIs ( #6030 )
...
* refactor latest code
* update api
* add dummy dataset
* update Readme
* add setup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update files
* add PP support
* update arguments
* update argument
* reorg folder
* update version
* remove IB infor
* update utils
* update readme
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update save for zero
* update save
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add apex
* update
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
3 months ago
cc1b0efc17
[plugin] hotfix zero plugin ( #6036 )
...
* [plugin] hotfix zero plugin
* [plugin] hotfix zero plugin
3 months ago
d383449fc4
[CI] Remove triton version for compatibility bug; update req torch >=2.2 ( #6018 )
...
* remove triton version
* remove torch 2.2
* remove torch 2.1
* debug
* remove 2.1 build tests
* require torch >=2.2
---------
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
3 months ago
17904cb5bf
Merge pull request #6012 from hpcaitech/feature/fp8_comm
...
[fp8] support fp8 communication and fp8 training for Colossalai
3 months ago
4a6f31eb0c
Merge pull request #6033 from wangbluo/fix
...
[fp8] fix the merge
3 months ago
80d24ae519
[pre-commit.ci] auto fixes from pre-commit.com hooks
...
for more information, see https://pre-commit.ci
3 months ago
dae39999d7
fix
3 months ago
7cf9df07bc
[Hotfix] Fix llama fwd replacement bug ( #6031 )
...
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
3 months ago
0bf46c54af
Merge pull request #6029 from hpcaitech/flybird11111-patch-1
...
Update train_dpo.py
3 months ago
botbw