ColossalAI/colossalai/shardformer/modeling
Edenzzzz fbf33ecd01
[Feature] Enable PP + SP for llama (#5868)
* fix cross-PP-stage position id length diff bug

* fix typo

* fix typo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use a one cross entropy func for all shardformer models

---------

Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-07-09 18:05:20 +08:00
..
chatglm2_6b
__init__.py
bert.py
blip2.py
bloom.py [Feature] Enable PP + SP for llama (#5868) 2024-07-09 18:05:20 +08:00
chatglm2.py
command.py [Feature] Enable PP + SP for llama (#5868) 2024-07-09 18:05:20 +08:00
deepseek.py
falcon.py
gpt2.py [Feature] Enable PP + SP for llama (#5868) 2024-07-09 18:05:20 +08:00
gptj.py
jit.py
llama.py [Feature] Enable PP + SP for llama (#5868) 2024-07-09 18:05:20 +08:00
mistral.py [Feature] Enable PP + SP for llama (#5868) 2024-07-09 18:05:20 +08:00
mixtral.py
opt.py [Feature] Enable PP + SP for llama (#5868) 2024-07-09 18:05:20 +08:00
qwen2.py [Feature] Enable PP + SP for llama (#5868) 2024-07-09 18:05:20 +08:00
sam.py
t5.py
vit.py
whisper.py