github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
3,394 Commits
32 Branches
47 Tags
229 MiB
Commit Graph

273 Commits (colossalchat_upgrade)

Author SHA1 Message Date
YeAnbang 82aecd6374 add SimPO 2024-06-24 02:12:20 +00:00
YeAnbang 84eab13078 update sft trainning script 2024-06-11 02:44:20 +00:00
YeAnbang 2abdede1d7 fix readme 2024-06-10 01:08:42 +00:00
YeAnbang 77db21610a replace the customized dataloader setup with the build-in one 2024-06-07 09:44:25 +00:00
YeAnbang 0d7ff10ea5 replace the customized dataloader setup with the build-in one 2024-06-07 09:43:42 +00:00
YeAnbang 790e1362a6 merge 2024-06-07 07:01:32 +00:00
YeAnbang ac1520cb8f remove baichuan from template test due to transformer version conflict 2024-06-07 07:01:32 +00:00
YeAnbang e16ccc272a update ci 2024-06-07 07:01:32 +00:00
YeAnbang 45195ac53d remove local data path 2024-06-07 07:01:31 +00:00
YeAnbang bf57b13dda remove models that require huggingface auth from ci 2024-06-07 07:01:31 +00:00
YeAnbang 0bbac158ed fix datasets version 2024-06-07 07:01:31 +00:00
YeAnbang 62eb28b929 remove duplicated test 2024-06-07 07:01:31 +00:00
YeAnbang b8b5cacf38 fix transformers version 2024-06-07 07:01:31 +00:00
pre-commit-ci[bot] 1b880ce095 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2024-06-07 07:01:31 +00:00
YeAnbang 7ae87b3159 fix training script 2024-06-07 07:01:31 +00:00
YeAnbang 0b4a33548c moupdate ci tests, st ci test cases passed, tp failed in generation for ppo, sp is buggy 2024-06-07 07:01:31 +00:00
YeAnbang 7e65b71815 run pre-commit 2024-06-07 07:01:30 +00:00
YeAnbang 929e1e3da4 upgrade ppo dpo rm script 2024-06-07 07:01:30 +00:00
YeAnbang 7a7e86987d upgrade colossal-chat support tp_group>1, add sp for sft 2024-06-07 07:01:30 +00:00
Tong Li 913c920ecc
[Colossal-LLaMA] Fix sft issue for llama2 (#5719) 
* fix minor issue

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-05-15 10:52:11 +08:00
Hongxin Liu 7f8b16635b
[misc] refactor launch API and tensor constructor (#5666) 
* [misc] remove config arg from initialize

* [misc] remove old tensor contrusctor

* [plugin] add npu support for ddp

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [devops] fix doc test ci

* [test] fix test launch

* [doc] update launch doc

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-04-29 10:40:11 +08:00
linsj20 91fa553775 [Feature] qlora support (#5586) 
* [feature] qlora support

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* qlora follow commit

* migrate qutization folder to colossalai/

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor fixes

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-04-28 10:51:27 +08:00
Tong Li 862fbaaa62
[Feature] Support LLaMA-3 CPT and ST (#5619) 
* support LLaMA-3

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Run pre-commit

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-04-23 13:54:05 +08:00
Camille Zhong 89049b0d89
[doc] fix ColossalMoE readme (#5599) 
* fix readme

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-04-15 18:06:18 +08:00
Hongxin Liu 641b1ee71a
[devops] remove post commit ci (#5566) 
* [devops] remove post commit ci

* [misc] run pre-commit on all files

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-04-08 15:09:40 +08:00
digger yu a799ca343b
[fix] fix typo s/muiti-node /multi-node etc. (#5448) 2024-04-07 18:42:15 +08:00
Wenhao Chen e614aa34f3
[shardformer, pipeline] add `gradient_checkpointing_ratio` and heterogenous shard policy for llama (#5508) 
* feat: add `GradientCheckpointConfig` and `PipelineGradientCheckpointConfig`

* feat: apply `GradientCheckpointConfig` to policy and llama_forward

* feat: move `distribute_layer` and `get_stage_index` to PipelineStageManager

* fix: add optional args for `distribute_layer` and `get_stage_index`

* fix: fix changed API calls

* test: update llama tests

* style: polish `GradientCheckpointConfig`

* fix: fix pipeline utils tests
 2024-04-01 11:34:58 +08:00
YeAnbang df5e9c53cf
[ColossalChat] Update RLHF V2 (#5286) 
* Add dpo. Fix sft, ppo, lora. Refactor all

* fix and tested ppo

* 2 nd round refactor

* add ci tests

* fix ci

* fix ci

* fix readme, style

* fix readme style

* fix style, fix benchmark

* reproduce benchmark result, remove useless files

* rename to ColossalChat

* use new image

* fix ci workflow

* fix ci

* use local model/tokenizer for ci tests

* fix ci

* fix ci

* fix ci

* fix ci timeout

* fix rm progress bar. fix ci timeout

* fix ci

* fix ci typo

* remove 3d plugin from ci temporary

* test environment

* cannot save optimizer

* support chat template

* fix readme

* fix path

* test ci locally

* restore build_or_pr

* fix ci data path

* fix benchmark

* fix ci, move ci tests to 3080, disable fast tokenizer

* move ci to 85

* support flash attention 2

* add all-in-one data preparation script. Fix colossal-llama2-chat chat template

* add hardware requirements

* move ci test data

* fix save_model, add unwrap

* fix missing bos

* fix missing bos; support grad accumulation with gemini

* fix ci

* fix ci

* fix ci

* fix llama2 chat template config

* debug sft

* debug sft

* fix colossalai version requirement

* fix ci

* add sanity check to prevent NaN loss

* fix requirements

* add dummy data generation script

* add dummy data generation script

* add dummy data generation script

* add dummy data generation script

* update readme

* update readme

* update readme and ignore

* fix logger bug

* support parallel_output

* modify data preparation logic

* fix tokenization

* update lr

* fix inference

* run pre-commit

---------

Co-authored-by: Tong Li <tong.li352711588@gmail.com>
 2024-03-29 14:12:29 +08:00
Insu Jang 00525f7772
[shardformer] fix pipeline forward error if custom layer distribution is used (#5189) 
* Use self.[distribute_layers|get_stage_index] to exploit custom layer distribution

* Change static methods for t5 layer distribution to member functions

* Change static methods for whisper layer distribution to member functions

* Replace whisper policy usage with self one

* Fix test case to use non-static layer distribution methods

* fix: fix typo

---------

Co-authored-by: Wenhao Chen <cwher@outlook.com>
 2024-03-27 13:57:00 +08:00
Wenhao Chen bb0a668fee
[hotfix] set return_outputs=False in examples and polish code (#5404) 
* fix: simplify merge_batch

* fix: use return_outputs=False to eliminate extra memory consumption

* feat: add return_outputs warning

* style: remove `return_outputs=False` as it is the default value
 2024-03-25 12:31:09 +08:00
binmakeswell d158fc0e64
[doc] update open-sora demo (#5479) 
* [doc] update open-sora demo

* [doc] update open-sora demo

* [doc] update open-sora demo
 2024-03-20 16:08:41 +08:00
digger yu 385e85afd4
[hotfix] fix typo s/keywrods/keywords etc. (#5429) 2024-03-12 11:25:16 +08:00
Camille Zhong da885ed540
fix tensor data update for gemini loss caluculation (#5442) 2024-03-11 13:49:58 +08:00
Camille Zhong 743e7fad2f
[colossal-llama2] add stream chat examlple for chat version model (#5428) 
* add stream chat for chat version

* remove os.system clear

* modify function name
 2024-03-07 14:58:56 +08:00
hugo-syn c8003d463b
[doc] Fix typo s/infered/inferred/ (#5288) 
Signed-off-by: hugo-syn <hugo.vincent@synacktiv.com>
 2024-03-05 22:02:08 +08:00
Dongruixuan Li a7ae2b5b4c
[eval-hotfix] set few_shot_data to None when few shot is disabled (#5422) 2024-03-05 21:48:55 +08:00
binmakeswell 822241a99c
[doc] sora release (#5425) 
* [doc] sora release

* [doc] sora release

* [doc] sora release

* [doc] sora release
 2024-03-05 12:08:58 +08:00
Camille Zhong 4b8312c08e
fix sft single turn inference example (#5416) 2024-03-01 17:27:50 +08:00
Tong Li a28c971516
update requirements (#5407) 2024-02-28 17:46:27 +08:00
CZYCW b833153fd5
[hotfix] fix variable type for top_p (#5313) 
Co-authored-by: binmakeswell <binmakeswell@gmail.com>
 2024-02-19 18:25:44 +08:00
Hongxin Liu 7303801854
[llama] fix training and inference scripts (#5384) 
* [llama] refactor inference example to fit sft

* [llama] fix training script to fit gemini

* [llama] fix inference script
 2024-02-19 16:41:04 +08:00
Frank Lee efef43b53c
Merge pull request #5372 from hpcaitech/exp/mixtral 2024-02-08 16:30:05 +08:00
Hongxin Liu 65e5d6baa5 [moe] fix mixtral optim checkpoint (#5344) 2024-02-07 19:21:02 +08:00
Hongxin Liu 956b561b54 [moe] fix mixtral forward default value (#5329) 2024-02-07 19:21:02 +08:00
Hongxin Liu b60be18dcc [moe] fix mixtral checkpoint io (#5314) 2024-02-07 19:21:02 +08:00
Hongxin Liu da39d21b71 [moe] support mixtral (#5309) 
* [moe] add mixtral block for single expert

* [moe] mixtral block fwd support uneven ep

* [moe] mixtral block bwd support uneven ep

* [moe] add mixtral moe layer

* [moe] simplify replace

* [meo] support save sharded mixtral

* [meo] support load sharded mixtral

* [meo] support save sharded optim

* [meo] integrate moe manager into plug

* [meo] fix optimizer load

* [meo] fix mixtral layer
 2024-02-07 19:21:02 +08:00
Hongxin Liu c904d2ae99 [moe] update capacity computing (#5253) 
* [moe] top2 allow uneven input

* [moe] update capacity computing

* [moe] remove debug info

* [moe] update capacity computing

* [moe] update capacity computing
 2024-02-07 19:21:02 +08:00
Xuanlei Zhao 7d8e0338a4 [moe] init mixtral impl 2024-02-07 19:21:02 +08:00
Hongxin Liu 084c91246c
[llama] fix memory issue (#5371) 
* [llama] fix memory issue

* [llama] add comment
 2024-02-06 19:02:37 +08:00
Hongxin Liu eb4f2d90f9
[llama] polish training script and fix optim ckpt (#5368) 2024-02-06 11:52:17 +08:00