8eb6eac225
[fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;
2024-08-30 05:42:43 +00:00
6af81d8c0d
[feat] add fwd_bwd_step, run_fwd_only;
2024-08-30 02:47:52 +00:00
48ba22dbfd
[feat] fix optimizer bwd b & w; support return accum loss & output
2024-08-29 08:54:45 +00:00
4c4b01b859
[feat] add optim backward_b_by_grad
2024-08-29 03:16:59 +00:00
b1419ef76a
[fix] fix poc test; add comments in poc;
2024-08-28 05:47:53 +00:00
582ba0d6ff
[feat] fix func name & ci; add comments;
2024-08-28 03:40:50 +00:00
b5f7b4d228
[feat] fix poc format
2024-08-28 03:08:35 +00:00
d6e3d7d2a3
[feat] fix ci; add assert;
2024-08-28 02:41:05 +00:00
29383b2de0
[fix] update
2024-08-28 02:33:42 +00:00
fe209164f1
[feat] add apply v_schedule graph; p & p.grad assert err exist;
2024-08-27 10:29:39 +00:00
8b37323f16
[feat] add run_fwd_bwd_with_microbatch (replace input) & test; add p&p.grad assert close test & all pass;
2024-08-27 09:31:38 +00:00
9e0bd1af00
[fix] fix ci test; add pytest;
2024-08-27 08:00:23 +00:00
283c9ff5d2
[fix] rm useless assign and comments;
2024-08-27 07:31:58 +00:00
1b4bb2beeb
[feat] add comments for ZBV func;
2024-08-27 07:11:50 +00:00
f1c1a87246
[feat] add test for p & p grad;
2024-08-27 06:37:26 +00:00
5e09c8b4e1
[feat] split communication and calculation; fix pop empty send_bwd_buffer error;
2024-08-27 06:29:13 +00:00
1d75045c37
[feat] add test run_fwd_bwd automatic scheduling;
2024-08-26 11:21:56 +00:00
fd5526b76e
Merge branch 'main' into dev/zero_bubble
2024-08-26 04:03:20 +00:00
107230d27a
[update] update text;
2024-08-26 04:00:51 +00:00
203033ea16
[fix] fix weight not close;
2024-08-23 08:57:27 +00:00
7cf9df07bc
[Hotfix] Fix llama fwd replacement bug ( #6031 )
...
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
2024-08-23 15:44:27 +08:00
c18ef060cf
[feat] add dw test;
2024-08-23 06:04:12 +00:00
ee9baedadf
[feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble;
2024-08-22 10:25:34 +00:00
39e2597426
[ColossalChat] Add PP support ( #6001 )
...
* support pp training
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update rm
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refactor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update test case
* fix
* change to 4
* fix eval
* test
* add pp
* hotfix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* support pp training
* update rm
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refactor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update test case
* fix
* change to 4
* fix eval
* test
* add pp
* hotfix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update
* skip pp eval
* update all reduce
* update sft
* update ignore
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update no cache
* add eval
* remove fi
* remove debug
* remove parentheses to avoid warning
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert "add eval"
This reverts commit 3ab2f6fa32
2024-08-21 10:47:39 +08:00
0d3b0bd864
[plugin] add cast inputs option for zero ( #6003 ) ( #6022 )
2024-08-21 10:21:26 +08:00
dcc44aab8d
[misc] Use dist logger in plugins ( #6011 )
...
* use dist logger in plugins
* remove trash
* print on rank 0
---------
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
2024-08-20 10:32:41 +08:00
f1c3266a94
overlap kv comm with output rescale ( #6017 )
...
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
2024-08-19 14:08:17 +08:00
26493b97d3
[misc] update compatibility ( #6008 )
...
* [misc] update compatibility
* [misc] update requirements
* [devops] disable requirements cache
* [test] fix torch ddp test
* [test] fix rerun on address in use
* [test] fix lazy init
2024-08-16 18:49:14 +08:00
f5c84af0b0
[Feature] Zigzag Ring attention ( #5905 )
...
* halfway
* fix cross-PP-stage position id length diff bug
* fix typo
* fix typo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unified cross entropy func for all shardformer models
* remove redundant lines
* add basic ring attn; debug cross entropy
* fwd bwd logic complete
* fwd bwd logic complete; add experimental triton rescale
* precision tests passed
* precision tests passed
* fix typos and remove misc files
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add sp_mode to benchmark; fix varlen interface
* update softmax_lse shape by new interface
* change tester name
* remove buffer clone; support packed seq layout
* add varlen tests
* fix typo
* all tests passed
* add dkv_group; fix mask
* remove debug statements
---------
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-16 13:56:38 +08:00
887d2d579b
[misc] Bypass the huggingface bug to solve the mask mismatch problem ( #5991 )
2024-08-15 14:40:26 +08:00
4dd03999ec
[pre-commit.ci] pre-commit autoupdate ( #5995 )
...
updates:
- [github.com/psf/black-pre-commit-mirror: 24.4.2 → 24.8.0](https://github.com/psf/black-pre-commit-mirror/compare/24.4.2...24.8.0 )
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-15 14:40:03 +08:00
406f984063
[plugin] add cast inputs option for zero ( #6003 )
2024-08-15 10:41:22 +08:00
ceb1e262e7
fix sync condition ( #6000 )
2024-08-14 11:22:39 +08:00
ed97d3a5d3
[Chat] fix readme ( #5989 )
...
* fix readme
* fix readme, tokenization fully tested
* fix readme, tokenization fully tested
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: root <root@notebook-8f919155-6035-47b4-9c6f-1be133b9e2c9-0.notebook-8f919155-6035-47b4-9c6f-1be133b9e2c9.colossal-ai.svc.cluster.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-12 14:55:17 +08:00
b4d2377d4c
[Hotfix] Avoid fused RMSnorm import error without apex ( #5985 )
...
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
2024-08-09 18:17:09 +08:00
ad3fa4f49c
[Hotfix] README link ( #5966 )
...
* update ignore
* update readme
* run style
* update readme
2024-08-08 18:04:47 +08:00
9179d4088e
[Docs] clarify launch port
...
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
2024-08-07 13:53:48 +08:00
fe71917851
Merge pull request #5962 from hpcaitech/colossalchat
...
[Chat] Support overall loss, update KTO logging
2024-08-02 17:32:41 +08:00
0b2d55c4ab
Support overall loss, update KTO logging
2024-08-02 06:51:38 +00:00
75c963686f
[lora] lora support hybrid parallel plugin ( #5956 )
...
* lora support hybrid plugin
* fix
* fix
* fix
* fix
2024-08-02 10:36:58 +08:00
19d1510ea2
[feat] Dist Loader for Eval ( #5950 )
...
* support auto distributed data loader
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* support auto distributed data loader
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix tp error
* remove unused parameters
* remove unused
* update inference
* update docs
* update inference
---------
Co-authored-by: Michelle <qianranma8@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-02 10:06:25 +08:00
62cdac6b7b
[chore] remove redundant test case, print string & reduce test tokens
2024-08-01 10:06:59 +08:00
d1d1ab871e
[moe] solve dp axis issue
2024-08-01 10:06:59 +08:00
65daa87627
[doc] add MoeHybridParallelPlugin docstring
2024-08-01 10:06:59 +08:00
7bedd03739
[moe] remove force_overlap_comm flag and add warning instead
2024-08-01 10:06:59 +08:00
f7c5485ed6
[chore] docstring
2024-08-01 10:06:59 +08:00
7e737df5ad
[misc] remove useless condition
2024-08-01 10:06:59 +08:00
70793ce9ed
[misc] fix ci failure: change default value to false in moe plugin
2024-08-01 10:06:59 +08:00
12d043ca00
[misc] remove incompatible test config
2024-08-01 10:06:59 +08:00
606b0891ed
[chore] change moe_pg_mesh to private
2024-08-01 10:06:59 +08:00
duanjunwen