0ab3de8994
fix bugs with compute moe norm
2023-08-22 14:00:07 +08:00
716131e477
introduce how to deploy 4-bit quantized internlm model ( #207 )
2023-08-22 11:31:01 +08:00
8407c203a3
refactor code
2023-08-22 10:53:21 +08:00
ac243e5b33
refactor code
2023-08-22 10:42:39 +08:00
b01e20adc8
update moe training cfg on real-dataset
2023-08-22 10:36:17 +08:00
a8dd77ce76
fix bug on logger
2023-08-22 10:35:17 +08:00
075648cd70
update readme related to internlm-chat-7v-v1.1 ( #214 )
2023-08-22 08:08:44 +08:00
58108413bd
Update readme for news of InternLM-Chat-7B-v1.1 and Lagent ( #213 )
...
* update readme
* fix typo
2023-08-22 07:46:01 +08:00
cc3c48ae47
test(ci_scripts): add load ckpt cases ( #208 )
...
* fix format
* add scripts for load ckpt case
* update test config
* debug:use var in json
* fix syntax error
* export pythonpath
* use absolute path
* use father path of workspace
* debug load new ckpt
* change data path
* add train folder
* fix code format
* fix pylint warning
---------
Co-authored-by: wangmengke <wangmengke@pjlab.org.cn>
2023-08-21 15:24:43 +08:00
53648dc0e9
feat(train.py): support torch profiler ( #201 )
...
* feat(train.py): support torch profiling
* feat(train.py): optimize initialize_llm_profile
* feat(train.py): profiling with tp0 and dp0
* move sequence parallel context manager to evalation func
* fix lint
* move the process for type_ids to load_new_batch
* fix lint
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
2023-08-21 15:23:38 +08:00
4832671abe
fix(pipeline_scheduler.py): fix tensor shape err and comm block ( #210 )
2023-08-21 12:09:27 +08:00
05a3b2a3be
Merge branch 'feature_add_moe' of github.com:blankde/InternLM into feature_add_moe_pp_zl
2023-08-21 10:00:43 +08:00
db685e8a31
fix the pp moe bugs
2023-08-21 09:59:58 +08:00
08532dc20b
fix bugs with merge
2023-08-18 18:03:44 +08:00
7b1709a7ff
Merge branch 'feature_add_moe' of github.com:blankde/InternLM into feature_add_moe_pp_zl
...
Conflicts:
train.py
2023-08-17 17:00:04 +08:00
2983076d89
add logger for moe_loss
2023-08-17 16:52:11 +08:00
c76182b2d6
Merge branch 'develop' into feature_add_moe
2023-08-17 16:37:06 +08:00
c412ad39de
Merge branch 'feature_add_moe' of https://github.com/blankde/InternLM into feature_add_moe
2023-08-17 16:14:25 +08:00
754f1d961a
fix moe bugs in zero optimizer
2023-08-17 16:11:34 +08:00
48ed81d2e6
fix more moe bugs in zero optimizer
2023-08-17 15:44:36 +08:00
9ee57e6c8a
fix moe bugs in zero optimizer
2023-08-17 15:31:27 +08:00
f5f5446560
Merge main to develop ( #203 )
...
* fix/fix_submodule_err (#61 )
* fix/fix_submodule_err
---------
Co-authored-by: ChenQiaoling00 <qiaoling_chen@u.nus.edu>
* fix issue templates (#65 )
* fix(tokenizer): refactor tokenizer and update usage in readme (#51 )
* update tokenizer example
* fix(readme, requirements): fix typo at Chinese readme and select a lower version of transformers (#73 )
* fix a typo in readme
* in order to find InternLMTokenizer, select a lower version of Transformers
---------
Co-authored-by: gouhchangjiang <gouhchangjiang@gmail.com>
* [Doc] Add wechat and discord link in readme (#78 )
* Doc：add wechat and discord link
* Doc：update wechat and discord link
* Doc：update wechat and discord link
* Doc：update wechat and discord link
* Doc：update wechat and discord link
* Doc：update wechat and discord link
* Doc：update wechat and discord link
* Doc：update wechat and discord link
* Doc：update wechat and discord link
* Doc：update wechat and discord link
* Doc：update wechat and discord link
* [Docs]: add Japanese README (#43 )
* Add Japanese README
* Update README-ja-JP.md
replace message
* Update README-ja-JP.md
* add repetition_penalty in GenerationConfig in web_demo.py (#48 )
Co-authored-by: YWMditto <862779238@qq.com>
* use fp16 in instruction (#80 )
* [Enchancement] add more options for issue template (#77 )
* [Enchancement] add more options for issue template
* update qustion icon
* fix link
* Use tempfile for convert2hf.py (#23 )
Fix https://github.com/InternLM/InternLM/issues/50
* delete torch_dtype of README's example code (#100 )
* set the value of repetition_penalty to 1.0 to avoid random outputs (#99 )
* Update web_demo.py (#97 )
Remove meaningless log.
* [Fix]Fix wrong string cutoff in the script for sft text tokenizing (#106 )
* docs(install.md): update dependency package transformers version to >= 4.28.0 (#124 )
Co-authored-by: 黄婷 <huangting3@CN0014010744M.local>
* docs(LICENSE): add license (#125 )
* add license of colossalai and flash-attn
* fix lint
* modify the name
* fix AutoModel map in convert2hf.py (#116 )
* variables are not printly as expect (#114 )
* feat(solver): fix code to adapt to torch2.0 and provide docker images (#128 )
* feat(solver): fix code to adapt to torch2.0
* docs(install.md): publish internlm environment image
* docs(install.md): update dependency packages version
* docs(install.md): update default image
---------
Co-authored-by: 黄婷 <huangting3@CN0014010744M.local>
* add demo test (#132 )
Co-authored-by: qa-caif-cicd <qa-caif-cicd@pjlab.org.cn>
* fix web_demo cache accelerate (#133 )
* Doc: add twitter link (#141 )
* Feat add checkpoint fraction (#151 )
* feat(config): add checkpoint_fraction into config
* feat: remove checkpoint_fraction from configs/7B_sft.py
---------
Co-authored-by: wangguoteng.p <wangguoteng925@qq.com>
* [Doc] update deployment guide to keep consistency with lmdeploy (#136 )
* update deployment guide
* fix error
* use llm partition (#159 )
Co-authored-by: qa-caif-cicd <qa-caif-cicd@pjlab.org.cn>
* test(ci_scripts): clean test data after test, remove unnecessary global variables, and other optimizations (#165 )
* test: optimization of ci scripts(variables, test data cleaning, etc).
* chore(workflows): disable ci job on push.
* fix: update partition
* test(ci_scripts): add install requirements automaticlly,trigger event about lint check and other optimizations (#174 )
* add pull_request in lint check
* use default variables in ci_scripts
* fix format
* check and install requirements automaticlly
* fix format
---------
Co-authored-by: qa-caif-cicd <qa-caif-cicd@pjlab.org.cn>
* feat(profiling): add a simple memory profiler (#89 )
* feat(profiling): add simple memory profiler
* feat(profiling): add profiling argument
* feat(CI_workflow): Add PR & Issue auto remove workflow (#184 )
* feat(ci_workflow): Add PR & Issue auto remove workflow
Add a workflow for stale PR & Issue auto remove
- pr & issue well be labeled as stale for inactive in 7 days
- staled PR & Issue well be remove in 7 days
- run this workflow every day on 1:30 a.m.
* Update stale.yml
* feat(bot): Create .owners.yml for Auto Assign (#176 )
* Create .owners.yml: for issue/pr assign automatically
* Update .owners.yml
* Update .owners.yml
fix typo
* [feat]: add pal reasoning script (#163 )
* [Feat] Add PAL inference script
* Update README.md
* Update tools/README.md
Co-authored-by: BigDong <yudongwang1226@gmail.com>
* Update tools/pal_inference.py
Co-authored-by: BigDong <yudongwang1226@gmail.com>
* Update pal script
* Update README.md
* restore .ore-commit-config.yaml
* Update tools/README.md
Co-authored-by: BigDong <yudongwang1226@gmail.com>
* Update tools/README.md
Co-authored-by: BigDong <yudongwang1226@gmail.com>
* Update pal inference script
* Update READMD.md
* Update internlm/utils/interface.py
Co-authored-by: Wenwei Zhang <40779233+ZwwWayne@users.noreply.github.com>
* Update pal script
* Update pal script
* Update script
* Add docstring
* Update format
* Update script
* Update script
* Update script
---------
Co-authored-by: BigDong <yudongwang1226@gmail.com>
Co-authored-by: Wenwei Zhang <40779233+ZwwWayne@users.noreply.github.com>
* test(ci_scripts): add timeout settings and clean work after the slurm job (#185 )
* restore pr test on develop branch
* add mask
* add post action to cancel slurm job
* remove readonly attribute on job log
* add debug info
* debug job log
* try stdin
* use stdin
* set default value avoid error
* try setting readonly on job log
* performance echo
* remove debug info
* use squeue to check slurm job status
* restore the lossed parm
* litmit retry times
* use exclusive to avoid port already in use
* optimize loop body
* remove partition
* add {} for variables
* set env variable for slurm partition
---------
Co-authored-by: qa-caif-cicd <qa-caif-cicd@pjlab.org.cn>
* refactor(tools): move interface.py and import it to web_demo (#195 )
* move interface.py and import it to web_demo
* typo
* fix(ci): fix lint error
* fix(ci): fix lint error
---------
Co-authored-by: Sun Peng <sunpengsdu@gmail.com>
Co-authored-by: ChenQiaoling00 <qiaoling_chen@u.nus.edu>
Co-authored-by: Kai Chen <chenkaidev@gmail.com>
Co-authored-by: Yang Gao <Gary1546308416AL@gmail.com>
Co-authored-by: Changjiang GOU <gouchangjiang@gmail.com>
Co-authored-by: gouhchangjiang <gouhchangjiang@gmail.com>
Co-authored-by: vansin <msnode@163.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
Co-authored-by: YWMditto <46778265+YWMditto@users.noreply.github.com>
Co-authored-by: YWMditto <862779238@qq.com>
Co-authored-by: WRH <12756472+wangruohui@users.noreply.github.com>
Co-authored-by: liukuikun <24622904+Harold-lkk@users.noreply.github.com>
Co-authored-by: x54-729 <45304952+x54-729@users.noreply.github.com>
Co-authored-by: Shuo Zhang <zhangshuolove@live.com>
Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com>
Co-authored-by: 黄婷 <huangting3@CN0014010744M.local>
Co-authored-by: ytxiong <45058324+yingtongxiong@users.noreply.github.com>
Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
Co-authored-by: kkscilife <126147887+kkscilife@users.noreply.github.com>
Co-authored-by: qa-caif-cicd <qa-caif-cicd@pjlab.org.cn>
Co-authored-by: hw <45089338+MorningForest@users.noreply.github.com>
Co-authored-by: Guoteng <32697156+SolenoidWGT@users.noreply.github.com>
Co-authored-by: wangguoteng.p <wangguoteng925@qq.com>
Co-authored-by: lvhan028 <lvhan_028@163.com>
Co-authored-by: zachtzy <141206206+zachtzy@users.noreply.github.com>
Co-authored-by: cx <759046501@qq.com>
Co-authored-by: Jaylin Lee <61487970+APX103@users.noreply.github.com>
Co-authored-by: del-zhenwu <dele.zhenwu@gmail.com>
Co-authored-by: Shaoyuan Xie <66255889+Daniel-xsy@users.noreply.github.com>
Co-authored-by: BigDong <yudongwang1226@gmail.com>
Co-authored-by: Wenwei Zhang <40779233+ZwwWayne@users.noreply.github.com>
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-08-16 15:57:26 +08:00
f3664bfbab
fix(train.py): fix scheduler metric hook skip error ( #204 )
2023-08-16 15:47:05 +08:00
8cdd1abb35
suppport interleaved pp
2023-08-16 12:02:59 +08:00
5f2381af62
fix/ci train error ( #200 )
...
* fix(ci): fix ci train error
* fix(ci): fix ci train error
* fix(ci): fix ci train error
2023-08-16 11:11:27 +08:00
db13bc46bc
fix(ci): fix ci train error ( #199 )
2023-08-15 20:09:54 +08:00
ef851d16c6
Feat/optimizer ( #194 )
...
* feat(optimier.py): reduce memory footprint and avoid _check_overflow call
* feat(optimier.py): reduce memory footprint and avoid _check_overflow call
* feat(optimizer.py): overlap compute norm with allreduce
* update var and function name
* update function compute norm (#197 )
Co-authored-by: ChenQiaoling00 <qiaoling_chen@u.nus.edu>
* feat(optimizer/hybrid_zero_optim.py): overlap gradients last bucket allreduce and compute norm (#196 )
* support gradients allreduce and compute norm overlap
* fix para set error
* remove timer cal_norm for testing
* feat(optimizer/hybrid_zero_optim.py): support group global norm
* format(lint): fix lint error
* feat(optimizer/store.py): update code based on comment
---------
Co-authored-by: ChenQiaoling00 <qiaoling_chen@u.nus.edu>
Co-authored-by: huangting4201 <1538303371@qq.com>
2023-08-15 18:55:10 +08:00
7b4933de0d
Merge branch 'feature_add_moe' of github.com:blankde/InternLM into feature_add_moe_pp_zl
2023-08-15 18:49:55 +08:00
92a31732f9
fix the moe_loss_coeff bug
2023-08-15 18:47:19 +08:00
3bfaad895a
Merge branch 'feature_add_moe' of https://github.com/blankde/InternLM into feature_add_moe
2023-08-15 17:58:52 +08:00
929dd36cf2
avoid moe parameter partition in zero optimizer
2023-08-15 17:52:51 +08:00
8c7d868f01
support zero_overlap_communication
2023-08-15 16:18:20 +08:00
0600b42c01
refactor(tools): move interface.py and import it to web_demo ( #195 )
...
* move interface.py and import it to web_demo
* typo
2023-08-14 22:32:29 +08:00
1accc9f08d
add no-interleaved & no-overlapped moe pp support
2023-08-14 11:10:37 +08:00
d8e5397159
Merge branch 'develop' into feature_add_moe
2023-08-11 19:05:18 +08:00
4e8bd39d8f
refactor(solver/optimizer): improve optimizer memory ( #193 )
...
* refactor(solver/optimizer): improve optimizer memory
* feat(data): remove useless dataset type ids map
2023-08-11 17:46:07 +08:00
5f3133fac8
Revert "feat(ckpt): add auto ckpt load and singal quit ( #189 )" ( #192 )
...
This reverts commit a45a91bb84
2023-08-11 17:12:26 +08:00
ccb06a98e4
test(ci_scripts): add timeout settings and clean work after the slurm job ( #185 )
...
* restore pr test on develop branch
* add mask
* add post action to cancel slurm job
* remove readonly attribute on job log
* add debug info
* debug job log
* try stdin
* use stdin
* set default value avoid error
* try setting readonly on job log
* performance echo
* remove debug info
* use squeue to check slurm job status
* restore the lossed parm
* litmit retry times
* use exclusive to avoid port already in use
* optimize loop body
* remove partition
* add {} for variables
* set env variable for slurm partition
---------
Co-authored-by: qa-caif-cicd <qa-caif-cicd@pjlab.org.cn>
2023-08-11 17:09:56 +08:00
a45a91bb84
feat(ckpt): add auto ckpt load and singal quit ( #189 )
...
Co-authored-by: wangguoteng.p <wangguoteng925@qq.com>
2023-08-11 17:08:01 +08:00
73998a0bb7
modified: internlm/model/modeling_internlm.py
2023-08-11 11:58:22 +08:00
7cfea534e7
[feat]: add pal reasoning script ( #163 )
...
* [Feat] Add PAL inference script
* Update README.md
* Update tools/README.md
Co-authored-by: BigDong <yudongwang1226@gmail.com>
* Update tools/pal_inference.py
Co-authored-by: BigDong <yudongwang1226@gmail.com>
* Update pal script
* Update README.md
* restore .ore-commit-config.yaml
* Update tools/README.md
Co-authored-by: BigDong <yudongwang1226@gmail.com>
* Update tools/README.md
Co-authored-by: BigDong <yudongwang1226@gmail.com>
* Update pal inference script
* Update READMD.md
* Update internlm/utils/interface.py
Co-authored-by: Wenwei Zhang <40779233+ZwwWayne@users.noreply.github.com>
* Update pal script
* Update pal script
* Update script
* Add docstring
* Update format
* Update script
* Update script
* Update script
---------
Co-authored-by: BigDong <yudongwang1226@gmail.com>
Co-authored-by: Wenwei Zhang <40779233+ZwwWayne@users.noreply.github.com>
2023-08-10 17:53:46 +08:00
ac168bd9c1
Update .pre-commit-config.yaml
2023-08-10 16:10:51 +08:00
c7f9d4f48c
add expert data support and fix bugs
2023-08-10 16:07:35 +08:00
d28816a499
feat(bot): Create .owners.yml for Auto Assign ( #176 )
...
* Create .owners.yml: for issue/pr assign automatically
* Update .owners.yml
* Update .owners.yml
fix typo
2023-08-10 12:11:30 +08:00
e4c0651b96
feat(CI_workflow): Add PR & Issue auto remove workflow ( #184 )
...
* feat(ci_workflow): Add PR & Issue auto remove workflow
Add a workflow for stale PR & Issue auto remove
- pr & issue well be labeled as stale for inactive in 7 days
- staled PR & Issue well be remove in 7 days
- run this workflow every day on 1:30 a.m.
* Update stale.yml
2023-08-09 16:26:05 +08:00
5f2e082b21
reformat code
2023-08-09 16:03:47 +08:00
4a5cf5d1df
Merge pull request #1 from blankde/feature_add_moe_zl
...
add residual and other moe features
2023-08-09 15:40:52 +08:00
d1c7b607fa
add param arguments
2023-08-09 15:37:53 +08:00
0905da3e7b
Merge branch 'feature_add_moe' of github.com:blankde/InternLM into feature_add_moe_zl
2023-08-09 15:15:12 +08:00
e21ddc364b
modify grad clipping due to moe
2023-08-09 15:04:19 +08:00
Wenwen Qu