github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
3,652 Commits
35 Branches
48 Tags
232 MiB
Commit Graph

44 Commits (45f17fc6ccb239b64c010831ddbcabe32984e4f3)

Author SHA1 Message Date
duanjunwen 45f17fc6cc [fix] rm comments; 2024-09-26 06:13:56 +00:00
duanjunwen a92e16719b [fix] fix zerobubble; support shardformer model type; 2024-09-26 06:11:56 +00:00
duanjunwen fc8b016887 [fix] fix stage_indices; 2024-09-25 06:15:45 +00:00
duanjunwen a875212a42 [fix] fix ci --> oom in 4096 hidden dim; 2024-09-23 05:55:16 +00:00
duanjunwen 1739df423c [fix] fix fwd branch, fwd pass both micro_batch & internal_inputs' 2024-09-20 07:34:43 +00:00
duanjunwen 4753bf7add [fix] fix mem assert; 2024-09-19 08:27:47 +00:00
duanjunwen a115106f8d [fix] fix bwd w input; 2024-09-19 08:10:05 +00:00
duanjunwen 349272c71f [fix] updatw bwd b&w input; dict --> list[torch.Tensor] 2024-09-19 07:47:01 +00:00
duanjunwen 1f5c7258aa Merge remote-tracking branch 'upstream/feature/zerobubble' into dev/zero_bubble 2024-09-19 03:52:13 +00:00
duanjunwen af2c2f8092 [feat] add more test; 2024-09-18 07:51:54 +00:00
duanjunwen 3dbad102cf [fix] fix zerobubble pp for shardformer type input; 2024-09-18 07:14:34 +00:00
duanjunwen 9bc3b6e220 [feat] moehybrid support zerobubble; 2024-09-12 02:51:46 +00:00
duanjunwen 11ae6848c6
[zerobubble]Support ZeroBubble Pipeline (#6034) 
* [feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble;

* [feat] add dw test;

* [fix] fix weight not close;

* [update] update text;

* [feat] add test run_fwd_bwd automatic scheduling;

* [feat] split communication and calculation; fix pop empty send_bwd_buffer error;

* [feat] add test for p & p grad;

* [feat] add comments for ZBV func;

* [fix] rm useless assign and comments;

* [fix] fix ci test; add pytest;

* [feat] add run_fwd_bwd_with_microbatch  (replace input) & test; add p&p.grad assert close test & all pass;

* [feat] add apply v_schedule graph; p & p.grad assert err exist;

* [fix] update

* [feat] fix ci; add assert;

* [feat] fix poc format

* [feat] fix func name & ci; add comments;

* [fix] fix poc test; add comments in poc;

* [feat] add optim backward_b_by_grad

* [feat] fix optimizer bwd b & w; support return accum loss & output

* [feat] add fwd_bwd_step, run_fwd_only;

* [fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;

* [fix] fix communication_map;

* [feat] update test; rm comments;

* [fix] rm zbv in hybridplugin

* [fix] fix optim bwd;

* [fix] fix optim bwd;

* [fix] rm output.data after send fwd;

* [fix] fix bwd step if condition; remove useless comments and format info;

* [fix] fix detach output & release output;

* [fix] rm requir_grad for output;

* [fix] fix requir grad position and detach position and input&output local buffer append position;

* [feat] add memory assertation;

* [fix] fix mem check;

* [fix] mem assertation'

* [fix] fix mem assertation

* [fix] fix mem; use a new model shape; only assert mem less and equal than theo;

* [fix] fix model zoo import;

* [fix] fix redundant detach & clone; add buffer assertation in the end;

* [fix] add output_obj_grad assert None at bwd b step; replace input_obj.require_grad_ with treemap;

* [fix] update optim state dict assert (include param group & state); fix mem assert after add optim;

* [fix] add testcase with microbatch 4;
 2024-09-10 17:33:09 +08:00
duanjunwen 6c2a120bed [fix] add testcase with microbatch 4; 2024-09-09 10:16:03 +00:00
duanjunwen 8366a7855f [fix] update optim state dict assert (include param group & state); fix mem assert after add optim; 2024-09-09 09:27:13 +00:00
duanjunwen 7568b34626 [fix] fix redundant detach & clone; add buffer assertation in the end; 2024-09-09 08:04:28 +00:00
duanjunwen a5ec3d4285 [fix] fix mem; use a new model shape; only assert mem less and equal than theo; 2024-09-09 06:38:31 +00:00
duanjunwen 35a7b636b3 [fix] fix mem assertation 2024-09-09 05:41:39 +00:00
duanjunwen 400e5e5b23 [fix] mem assertation' 2024-09-09 02:58:06 +00:00
duanjunwen 4a358348c7 [fix] fix mem check; 2024-09-04 10:57:38 +00:00
duanjunwen 2f09c374f3 [feat] add memory assertation; 2024-09-04 06:34:18 +00:00
duanjunwen e6e1a97a6d [fix] fix requir grad position and detach position and input&output local buffer append position; 2024-09-04 03:31:08 +00:00
duanjunwen 4c1f81c683 [fix] fix bwd step if condition; remove useless comments and format info; 2024-09-03 08:56:08 +00:00
duanjunwen ab643c9af7 [fix] rm output.data after send fwd; 2024-09-03 14:12:17 +08:00
duanjunwen 591a13bf7e [fix] fix optim bwd; 2024-09-02 11:19:42 +00:00
duanjunwen 77fe44286c [fix] rm zbv in hybridplugin 2024-09-02 10:00:43 +00:00
duanjunwen 6d18d38d5c [feat] update test; rm comments; 2024-09-02 09:50:47 +00:00
duanjunwen 6af81d8c0d [feat] add fwd_bwd_step, run_fwd_only; 2024-08-30 02:47:52 +00:00
duanjunwen 48ba22dbfd [feat] fix optimizer bwd b & w; support return accum loss & output 2024-08-29 08:54:45 +00:00
duanjunwen 4c4b01b859 [feat] add optim backward_b_by_grad 2024-08-29 03:16:59 +00:00
duanjunwen b1419ef76a [fix] fix poc test; add comments in poc; 2024-08-28 05:47:53 +00:00
duanjunwen 582ba0d6ff [feat] fix func name & ci; add comments; 2024-08-28 03:40:50 +00:00
duanjunwen d6e3d7d2a3 [feat] fix ci; add assert; 2024-08-28 02:41:05 +00:00
duanjunwen 29383b2de0 [fix] update 2024-08-28 02:33:42 +00:00
duanjunwen fe209164f1 [feat] add apply v_schedule graph; p & p.grad assert err exist; 2024-08-27 10:29:39 +00:00
duanjunwen 8b37323f16 [feat] add run_fwd_bwd_with_microbatch (replace input) & test; add p&p.grad assert close test & all pass; 2024-08-27 09:31:38 +00:00
duanjunwen 9e0bd1af00 [fix] fix ci test; add pytest; 2024-08-27 08:00:23 +00:00
duanjunwen 283c9ff5d2 [fix] rm useless assign and comments; 2024-08-27 07:31:58 +00:00
duanjunwen f1c1a87246 [feat] add test for p & p grad; 2024-08-27 06:37:26 +00:00
duanjunwen 5e09c8b4e1 [feat] split communication and calculation; fix pop empty send_bwd_buffer error; 2024-08-27 06:29:13 +00:00
duanjunwen 1d75045c37 [feat] add test run_fwd_bwd automatic scheduling; 2024-08-26 11:21:56 +00:00
duanjunwen 203033ea16 [fix] fix weight not close; 2024-08-23 08:57:27 +00:00
duanjunwen c18ef060cf [feat] add dw test; 2024-08-23 06:04:12 +00:00
duanjunwen ee9baedadf [feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble; 2024-08-22 10:25:34 +00:00