fdd84b9087
fix the sp
2024-09-13 02:32:03 +00:00
9bc3b6e220
[feat] moehybrid support zerobubble;
2024-09-12 02:51:46 +00:00
a35a078f08
[doc] update sp doc ( #6055 )
...
* update sp doc
* fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix
* fix
* fix
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-11 17:25:14 +08:00
13946c4448
[fp8] hotfix backward hook ( #6053 )
...
* [fp8] hotfix backward hook
* [fp8] hotfix pipeline loss accumulation
2024-09-11 16:11:25 +08:00
11ae6848c6
[zerobubble]Support ZeroBubble Pipeline ( #6034 )
...
* [feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble;
* [feat] add dw test;
* [fix] fix weight not close;
* [update] update text;
* [feat] add test run_fwd_bwd automatic scheduling;
* [feat] split communication and calculation; fix pop empty send_bwd_buffer error;
* [feat] add test for p & p grad;
* [feat] add comments for ZBV func;
* [fix] rm useless assign and comments;
* [fix] fix ci test; add pytest;
* [feat] add run_fwd_bwd_with_microbatch (replace input) & test; add p&p.grad assert close test & all pass;
* [feat] add apply v_schedule graph; p & p.grad assert err exist;
* [fix] update
* [feat] fix ci; add assert;
* [feat] fix poc format
* [feat] fix func name & ci; add comments;
* [fix] fix poc test; add comments in poc;
* [feat] add optim backward_b_by_grad
* [feat] fix optimizer bwd b & w; support return accum loss & output
* [feat] add fwd_bwd_step, run_fwd_only;
* [fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;
* [fix] fix communication_map;
* [feat] update test; rm comments;
* [fix] rm zbv in hybridplugin
* [fix] fix optim bwd;
* [fix] fix optim bwd;
* [fix] rm output.data after send fwd;
* [fix] fix bwd step if condition; remove useless comments and format info;
* [fix] fix detach output & release output;
* [fix] rm requir_grad for output;
* [fix] fix requir grad position and detach position and input&output local buffer append position;
* [feat] add memory assertation;
* [fix] fix mem check;
* [fix] mem assertation'
* [fix] fix mem assertation
* [fix] fix mem; use a new model shape; only assert mem less and equal than theo;
* [fix] fix model zoo import;
* [fix] fix redundant detach & clone; add buffer assertation in the end;
* [fix] add output_obj_grad assert None at bwd b step; replace input_obj.require_grad_ with treemap;
* [fix] update optim state dict assert (include param group & state); fix mem assert after add optim;
* [fix] add testcase with microbatch 4;
2024-09-10 17:33:09 +08:00
c54c4fcd15
[hotfix] moe hybrid parallelism benchmark & follow-up fix ( #6048 )
...
* [example] pass use_fp8_comm flag to all plugins
* [example] add mixtral benchmark
* [moe] refine assertion and check
* [moe] fix mixtral & add more tests
* [moe] consider checking dp * sp group and moe_dp_group
* [mixtral] remove gate tp & add more tests
* [deepseek] fix tp & sp for deepseek
* [mixtral] minor fix
* [deepseek] add deepseek benchmark
2024-09-10 17:30:53 +08:00
8fd25d6e09
[Feature] Split cross-entropy computation in SP ( #5959 )
...
* halfway
* fix cross-PP-stage position id length diff bug
* fix typo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unified cross entropy func for all shardformer models
* remove redundant lines
* add basic ring attn; debug cross entropy
* fwd bwd logic complete
* fwd bwd logic complete; add experimental triton rescale
* precision tests passed
* precision tests passed
* fix typos and remove misc files
* update softmax_lse shape by new interface
* change tester name
* remove buffer clone; support packed seq layout
* add varlen tests
* fix typo
* all tests passed
* add dkv_group; fix mask
* remove debug statements
* adapt chatglm, command-R, qwen
* debug
* halfway
* fix cross-PP-stage position id length diff bug
* fix typo
* fix typo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unified cross entropy func for all shardformer models
* remove redundant lines
* add basic ring attn; debug cross entropy
* fwd bwd logic complete
* fwd bwd logic complete; add experimental triton rescale
* precision tests passed
* precision tests passed
* fix typos and remove misc files
* add sp_mode to benchmark; fix varlen interface
* update softmax_lse shape by new interface
* add varlen tests
* fix typo
* all tests passed
* add dkv_group; fix mask
* remove debug statements
* add comments
* q1 index only once
* remove events to simplify stream sync
* simplify forward/backward logic
* 2d ring forward passed
* 2d ring backward passed
* fixes
* fix ring attn loss
* 2D ring backward + llama passed
* merge
* update logger
* fix typo
* rebase
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix typo
* remove typos
* fixes
* support GPT
---------
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-10 12:06:50 +08:00
b3db1058ec
[release] update version ( #6041 )
...
* [release] update version
* [devops] update comp test
* [devops] update comp test debug
* [devops] debug comp test
* [devops] debug comp test
* [devops] debug comp test
* [devops] debug comp test
* [devops] debug comp test
2024-09-10 10:31:09 +08:00
6c2a120bed
[fix] add testcase with microbatch 4;
2024-09-09 10:16:03 +00:00
8366a7855f
[fix] update optim state dict assert (include param group & state); fix mem assert after add optim;
2024-09-09 09:27:13 +00:00
ce58d8e8bf
[fix] add output_obj_grad assert None at bwd b step; replace input_obj.require_grad_ with treemap;
2024-09-09 08:19:58 +00:00
7568b34626
[fix] fix redundant detach & clone; add buffer assertation in the end;
2024-09-09 08:04:28 +00:00
fed8b1587d
[fix] fix model zoo import;
2024-09-09 06:39:33 +00:00
a5ec3d4285
[fix] fix mem; use a new model shape; only assert mem less and equal than theo;
2024-09-09 06:38:31 +00:00
5ce6dd75bf
[fp8] disable all_to_all_fp8 in intranode ( #6045 )
...
* enhance all_to_all_fp8 with internode comm control
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* disable some fp8 ops due to performance issue
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-09 13:47:17 +08:00
35a7b636b3
[fix] fix mem assertation
2024-09-09 05:41:39 +00:00
400e5e5b23
[fix] mem assertation'
2024-09-09 02:58:06 +00:00
4a358348c7
[fix] fix mem check;
2024-09-04 10:57:38 +00:00
2f09c374f3
[feat] add memory assertation;
2024-09-04 06:34:18 +00:00
e6e1a97a6d
[fix] fix requir grad position and detach position and input&output local buffer append position;
2024-09-04 03:31:08 +00:00
20503cdfdf
[fix] rm requir_grad for output;
2024-09-03 09:24:40 +00:00
b4103f125c
[fix] fix detach output & release output;
2024-09-03 09:09:41 +00:00
4c1f81c683
[fix] fix bwd step if condition; remove useless comments and format info;
2024-09-03 08:56:08 +00:00
26e553937b
[fp8] fix linear hook ( #6046 )
2024-09-03 16:37:16 +08:00
c3b5caff0e
[fp8] optimize all-gather ( #6043 )
...
* [fp8] optimize all-gather
* [fp8] fix all gather fp8 ring
* [fp8] enable compile
* [fp8] fix all gather fp8 ring
2024-09-03 15:45:17 +08:00
ab643c9af7
[fix] rm output.data after send fwd;
2024-09-03 14:12:17 +08:00
a48afc4a66
[fix] fix optim bwd;
2024-09-03 02:40:26 +00:00
c650a906db
[Hotfix] Remove deprecated install ( #6042 )
...
* remove deprecated install
* remove unused folder
2024-09-03 10:33:18 +08:00
591a13bf7e
[fix] fix optim bwd;
2024-09-02 11:19:42 +00:00
77fe44286c
[fix] rm zbv in hybridplugin
2024-09-02 10:00:43 +00:00
6d18d38d5c
[feat] update test; rm comments;
2024-09-02 09:50:47 +00:00
e9032fb0b2
[colossalai/checkpoint_io/...] fix bug in load_state_dict_into_model; format error msg ( #6020 )
...
* fix bug in load_state_dict_into_model; format error msg
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update utils.py
to support checking missing_keys
* Update general_checkpoint_io.py
fix bug in missing_keys error message
* retrigger tests
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-02 16:56:35 +08:00
a7b767b071
[fix] fix communication_map;
2024-08-30 05:56:02 +00:00
8eb6eac225
[fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;
2024-08-30 05:42:43 +00:00
6af81d8c0d
[feat] add fwd_bwd_step, run_fwd_only;
2024-08-30 02:47:52 +00:00
48ba22dbfd
[feat] fix optimizer bwd b & w; support return accum loss & output
2024-08-29 08:54:45 +00:00
e96a0761ea
[FP8] unsqueeze scale to make it compatible with torch.compile ( #6040 )
2024-08-29 14:49:23 +08:00
4c4b01b859
[feat] add optim backward_b_by_grad
2024-08-29 03:16:59 +00:00
0d3a85d04f
add fused norm ( #6038 )
2024-08-28 17:12:51 +08:00
4a68efb7da
[Colossal-LLaMA] Refactor latest APIs ( #6030 )
...
* refactor latest code
* update api
* add dummy dataset
* update Readme
* add setup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update files
* add PP support
* update arguments
* update argument
* reorg folder
* update version
* remove IB infor
* update utils
* update readme
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update save for zero
* update save
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add apex
* update
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-28 17:01:58 +08:00
b1419ef76a
[fix] fix poc test; add comments in poc;
2024-08-28 05:47:53 +00:00
582ba0d6ff
[feat] fix func name & ci; add comments;
2024-08-28 03:40:50 +00:00
b5f7b4d228
[feat] fix poc format
2024-08-28 03:08:35 +00:00
d6e3d7d2a3
[feat] fix ci; add assert;
2024-08-28 02:41:05 +00:00
29383b2de0
[fix] update
2024-08-28 02:33:42 +00:00
cc1b0efc17
[plugin] hotfix zero plugin ( #6036 )
...
* [plugin] hotfix zero plugin
* [plugin] hotfix zero plugin
2024-08-28 10:16:48 +08:00
fe209164f1
[feat] add apply v_schedule graph; p & p.grad assert err exist;
2024-08-27 10:29:39 +00:00
8b37323f16
[feat] add run_fwd_bwd_with_microbatch (replace input) & test; add p&p.grad assert close test & all pass;
2024-08-27 09:31:38 +00:00
9e0bd1af00
[fix] fix ci test; add pytest;
2024-08-27 08:00:23 +00:00
283c9ff5d2
[fix] rm useless assign and comments;
2024-08-27 07:31:58 +00:00
wangbluo