1d75045c37
[feat] add test run_fwd_bwd automatic scheduling;
2024-08-26 11:21:56 +00:00
fd5526b76e
Merge branch 'main' into dev/zero_bubble
2024-08-26 04:03:20 +00:00
107230d27a
[update] update text;
2024-08-26 04:00:51 +00:00
203033ea16
[fix] fix weight not close;
2024-08-23 08:57:27 +00:00
7cf9df07bc
[Hotfix] Fix llama fwd replacement bug ( #6031 )
...
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
2024-08-23 15:44:27 +08:00
c18ef060cf
[feat] add dw test;
2024-08-23 06:04:12 +00:00
ee9baedadf
[feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble;
2024-08-22 10:25:34 +00:00
39e2597426
[ColossalChat] Add PP support ( #6001 )
...
* support pp training
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update rm
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refactor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update test case
* fix
* change to 4
* fix eval
* test
* add pp
* hotfix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* support pp training
* update rm
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refactor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update test case
* fix
* change to 4
* fix eval
* test
* add pp
* hotfix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update
* skip pp eval
* update all reduce
* update sft
* update ignore
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update no cache
* add eval
* remove fi
* remove debug
* remove parentheses to avoid warning
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert "add eval"
This reverts commit 3ab2f6fa32
2024-08-21 10:47:39 +08:00
0d3b0bd864
[plugin] add cast inputs option for zero ( #6003 ) ( #6022 )
2024-08-21 10:21:26 +08:00
dcc44aab8d
[misc] Use dist logger in plugins ( #6011 )
...
* use dist logger in plugins
* remove trash
* print on rank 0
---------
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
2024-08-20 10:32:41 +08:00
f1c3266a94
overlap kv comm with output rescale ( #6017 )
...
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
2024-08-19 14:08:17 +08:00
26493b97d3
[misc] update compatibility ( #6008 )
...
* [misc] update compatibility
* [misc] update requirements
* [devops] disable requirements cache
* [test] fix torch ddp test
* [test] fix rerun on address in use
* [test] fix lazy init
2024-08-16 18:49:14 +08:00
f5c84af0b0
[Feature] Zigzag Ring attention ( #5905 )
...
* halfway
* fix cross-PP-stage position id length diff bug
* fix typo
* fix typo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unified cross entropy func for all shardformer models
* remove redundant lines
* add basic ring attn; debug cross entropy
* fwd bwd logic complete
* fwd bwd logic complete; add experimental triton rescale
* precision tests passed
* precision tests passed
* fix typos and remove misc files
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add sp_mode to benchmark; fix varlen interface
* update softmax_lse shape by new interface
* change tester name
* remove buffer clone; support packed seq layout
* add varlen tests
* fix typo
* all tests passed
* add dkv_group; fix mask
* remove debug statements
---------
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-16 13:56:38 +08:00
887d2d579b
[misc] Bypass the huggingface bug to solve the mask mismatch problem ( #5991 )
2024-08-15 14:40:26 +08:00
4dd03999ec
[pre-commit.ci] pre-commit autoupdate ( #5995 )
...
updates:
- [github.com/psf/black-pre-commit-mirror: 24.4.2 → 24.8.0](https://github.com/psf/black-pre-commit-mirror/compare/24.4.2...24.8.0 )
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-15 14:40:03 +08:00
406f984063
[plugin] add cast inputs option for zero ( #6003 )
2024-08-15 10:41:22 +08:00
ceb1e262e7
fix sync condition ( #6000 )
2024-08-14 11:22:39 +08:00
ed97d3a5d3
[Chat] fix readme ( #5989 )
...
* fix readme
* fix readme, tokenization fully tested
* fix readme, tokenization fully tested
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: root <root@notebook-8f919155-6035-47b4-9c6f-1be133b9e2c9-0.notebook-8f919155-6035-47b4-9c6f-1be133b9e2c9.colossal-ai.svc.cluster.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-12 14:55:17 +08:00
b4d2377d4c
[Hotfix] Avoid fused RMSnorm import error without apex ( #5985 )
...
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
2024-08-09 18:17:09 +08:00
ad3fa4f49c
[Hotfix] README link ( #5966 )
...
* update ignore
* update readme
* run style
* update readme
2024-08-08 18:04:47 +08:00
9179d4088e
[Docs] clarify launch port
...
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
2024-08-07 13:53:48 +08:00
fe71917851
Merge pull request #5962 from hpcaitech/colossalchat
...
[Chat] Support overall loss, update KTO logging
2024-08-02 17:32:41 +08:00
0b2d55c4ab
Support overall loss, update KTO logging
2024-08-02 06:51:38 +00:00
75c963686f
[lora] lora support hybrid parallel plugin ( #5956 )
...
* lora support hybrid plugin
* fix
* fix
* fix
* fix
2024-08-02 10:36:58 +08:00
19d1510ea2
[feat] Dist Loader for Eval ( #5950 )
...
* support auto distributed data loader
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* support auto distributed data loader
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix tp error
* remove unused parameters
* remove unused
* update inference
* update docs
* update inference
---------
Co-authored-by: Michelle <qianranma8@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-02 10:06:25 +08:00
62cdac6b7b
[chore] remove redundant test case, print string & reduce test tokens
2024-08-01 10:06:59 +08:00
d1d1ab871e
[moe] solve dp axis issue
2024-08-01 10:06:59 +08:00
65daa87627
[doc] add MoeHybridParallelPlugin docstring
2024-08-01 10:06:59 +08:00
7bedd03739
[moe] remove force_overlap_comm flag and add warning instead
2024-08-01 10:06:59 +08:00
f7c5485ed6
[chore] docstring
2024-08-01 10:06:59 +08:00
7e737df5ad
[misc] remove useless condition
2024-08-01 10:06:59 +08:00
70793ce9ed
[misc] fix ci failure: change default value to false in moe plugin
2024-08-01 10:06:59 +08:00
12d043ca00
[misc] remove incompatible test config
2024-08-01 10:06:59 +08:00
606b0891ed
[chore] change moe_pg_mesh to private
2024-08-01 10:06:59 +08:00
5b4c12381b
Revert "[moe] implement submesh initialization"
...
This reverts commit 2f9bce6686
2024-08-01 10:06:59 +08:00
cb01c0d5ce
[moe] refactor mesh assignment
2024-08-01 10:06:59 +08:00
034020bd04
[misc] remove debug/print code
2024-08-01 10:06:59 +08:00
59bcf56c60
[misc] skip redunant test
2024-08-01 10:06:59 +08:00
c3dc9b4dba
[deepseek] replace attn (a workaround for bug in transformers)
2024-08-01 10:06:59 +08:00
6c39f0b144
[test] add check
2024-08-01 10:06:59 +08:00
b2952a5982
[moe] deepseek moe sp support
2024-08-01 10:06:59 +08:00
96d0fbc531
[bug] fix: somehow logger hangs the program
2024-08-01 10:06:59 +08:00
067e18f7e9
[test] fix test: test_zero1_2
2024-08-01 10:06:59 +08:00
74b03de3f9
[moe] remove ops
2024-08-01 10:06:59 +08:00
70c9924d0d
[chore] solve moe ckpt test failure and some other arg pass failure
2024-08-01 10:06:59 +08:00
52d346f2a5
[pre-commit.ci] auto fixes from pre-commit.com hooks
...
for more information, see https://pre-commit.ci
2024-08-01 10:06:59 +08:00
46037c2ccd
[chore] minor fix after rebase
2024-08-01 10:06:59 +08:00
803878b2fd
[moe] full test for deepseek and mixtral (pp + sp to fix)
2024-08-01 10:06:59 +08:00
7077d38d5a
[moe] finalize test (no pp)
2024-08-01 10:06:59 +08:00
2cddeac717
moe sp + ep bug fix
2024-08-01 10:06:59 +08:00
duanjunwen