github
/
InternLM
mirror of https://github.com/InternLM/InternLM
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
184 Commits
4 Branches
8 Tags
10 MiB
Commit Graph

184 Commits (655e9dae40ba20df20db61bfb89cdae742f1a5f8)

Author SHA1 Message Date
Wenwen Qu 655e9dae40
Feat(norm)/support fused precision (#319) 
* add fused precision support for norm

* refactor code

* refactor code

* change the granularity of hook

* fix bugs if self.model is ModuleList

* add dtype condition for post hook

* refactor code for split group

* refactor code for pre/post hook

* refactor code for split group

* remove fp32 hook for norm

* unit tests for fused precision

* add doc for fused precision

* add doc for En. version

* reformat docs

* Update mixed_precision.rst

* Update mixed_precision.po

* update mixed_precision.po
 2023-09-26 20:39:55 +08:00
YWMditto 96b20cd43f
doc(usage): add dynamic ntk into doc (#367) 
* add long text generation in doc/usage.md

* add long text generation in doc/usage.md

* add long text generation in doc/usage.md

---------

Co-authored-by: YWMditto <862779238@qq.com>
 2023-09-26 16:58:46 +08:00
jiaxingli c1e30cff2c
feat(numa): bind numa if possible (#320) 
* feat:add numa

* feat:add bind numa

* feat:add bind numa

* feat:add bind numa

* feat: bind numa

* feat: bind numa

* feat: add numa

* feat:add numa

* feat:add numa

* try_bind_numa should not raise exception

---------

Co-authored-by: 877825076@qq.com <877825076@qq.com>
 2023-09-25 19:34:52 +08:00
jiaopenglong 9284303a6d
doc(monitor): add light monitoring doc (#352) 
* add light monitoring doc

* update light monitoring doc

* update light monitoring doc

* update light monitoring doc

* update light monitoring doc continue

* update light monitoring doc continue

* update monitor config doc

* update monitor config doc continue

* update monitor config doc continue
 2023-09-25 19:28:09 +08:00
jiaopenglong 847cc819dd
fix(monitor): add volc and aliyun jobid (#338) 
* add volc and aliyun jobid

* rm workspaceid
 2023-09-25 17:58:32 +08:00
jiaopenglong 064965527b
fix(config): monitor config key error when args_check is False (#362) 
* add monitor switch

* add switch to light monitor

* fix alert_address is empty

* fix light monitor heartbeat

* init light_monitor on rank_log only

* add comments to the monitoring config

* optimize config

* fix monitor config key error when args_check is False
 2023-09-25 17:30:36 +08:00
Guoteng 26a7397752
fix(storage): fix try_get_storage_backend (#359) 
* fix(storage): fix try_get_storage_backend

* fix typo and print infos only in log rank

* fix typo and print infos only in log rank

---------

Co-authored-by: gaoyang07 <Gary1546308416AL@gmail.com>
 2023-09-25 15:16:25 +08:00
huangting4201 a86c4bbbfd Merge branch 'main' into develop 2023-09-22 19:24:03 +08:00
Guoteng d1e52f0c03
feat(doc/code-docs): add checkpoint save/load usage doc (#311) 
* feat(doc): add checkpoint doc

* fix checkpoint doc

* fix comment

* fix(doc/code-docs): remove fuzzy

* fix(doc/code-docs): fix some errors

* fix(doc/code-docs): minor fix

---------

Co-authored-by: li126com <li126com2@126.com>
Co-authored-by: huangting4201 <1538303371@qq.com>
 2023-09-22 18:45:33 +08:00
huangting4201 1ed36754df
feat(.github/workflows): update ci e2e tests and add ci unit tests (#324) 
* feat(.github/workflows/e2e_test.yaml): update e2e yaml

* feat(.github/workflows/e2e_test.yaml): update e2e yaml

* test e2e

* test e2e

* test e2e

* test e2e

* test e2e

* fix(ci): test ci

* fix(ci): test ci

* fix(ci): test ci

* fix(ci): test ci

* fix(ci): test ci

* fix(ci): add weekly tests

---------

Co-authored-by: huangting4201 <huangting3@sensetime.com>
 2023-09-22 14:07:14 +08:00
jiaxingli f5337f6e02
Feat(PythonGC): Do garbage collection manually (#326) 
* feat:add gc control

* feat:add gc control

* feat:add gc control

* feat:add gc

* re-lint
 2023-09-22 13:52:25 +08:00
huangting4201 3b0eff0c8a
fix(model/embedding.py): ci lint check error (#345) 
* fix(ci): fix ci lint error

* fix(ci): fix ci lint error
 2023-09-21 14:46:22 +08:00
YWMditto 8464425a7b
feat(mdoel): add DynamicNTKScalingRotaryEmbedding (#339) 
* add dynamic ntk rope

* update dynamic ntk rope

* fix lint check

* fix lint check

* add more desc

---------

Co-authored-by: YWMditto <862779238@qq.com>
 2023-09-20 23:31:47 +08:00
Shuo Zhang e611817442
fix(doc): add 20b releasing info to readme (#330) 
* fix(eval): StreamingDataset does not have an __len__ method.

* doc(readme): update readme

* update readme

* update readme

* update readme

* update readme

* update readme

* update readme
 2023-09-20 16:46:45 +08:00
Shuo Zhang 5e5d160685
fix(readme): fix readme about 20B releasing (#329) 
* fix(eval): StreamingDataset does not have an __len__ method.

* doc(readme): update readme

* update readme

* update readme

* update readme

* update readme

* update readme
 2023-09-20 16:26:43 +08:00
Shuo Zhang 2a09ebd5c1
doc(readme): update readme, add 20B releasing info (#328) 
* fix(eval): StreamingDataset does not have an __len__ method.

* doc(readme): update readme

* update readme
 2023-09-20 16:04:43 +08:00
huangting4201 67eda4cbe1 fix(.github/workflows/e2e_test.yaml): update ci runner name 2023-09-19 18:13:20 +08:00
yingtongxiong 30b21075e8 merge main 2023-09-19 18:04:47 +08:00
ytxiong 6a5915bf0d
feat(linear): optimize mlp by using jit (#321) 
* fuse silu op

* refactor code

* fix lint

* fix lint
 2023-09-19 14:57:43 +08:00
huangting4201 025ca55dfe
test(tests/test_training): add training e2e tests for loss spike and loss accuracy (#304) 
* tests(test_training): add test case for loss accuracy

* tests(test_training): update test cases

* ci(.github/workflows/e2e_test.yaml): remove pull submodule

* ci(.github/workflows/e2e_test.yaml): update ci env and remove useless env var

* test(tests/test_training): add 16 GPUs test cases

* test(tests/test_training): fix training_16GPU_8DP2PP test case error

* test(tests/test_training): add new case for interleaved pp

* test(tests/test_training): remove redundant code

* test(tests/test_training): update ci job timeout minutes to 30m

* feat(initialize/launch.py): check num_chunks and interleaved_overlap

---------

Co-authored-by: huangting4201 <huangting3@sensetime.com>
 2023-09-19 14:55:40 +08:00
kkscilife bfefc4ea3c
test(ci_scripts): move ci env (#317) 
* change partition and runner label

* change rm action to mv

* use spot

* use rsync to move test files

* remove *

* remove *

* change into llm_s partition

---------

Co-authored-by: wangmengke <wangmengke@pjlab.org.cn>
 2023-09-19 14:52:32 +08:00
x54-729 b9824fab89
fix(tools): fix yield bug in stream_chat (#315) 2023-09-19 14:18:02 +08:00
x54-729 cd6426a249
feat(tools): support openai api (#313) 
* fix(chat): fix stream_chat to return generator (#123)

* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302)

* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299)

* support openai api to deploy internlm

* update README for information os openai_api.py

* change example in README_EN.md to English

* delete unnecessary print; fix model card typo; fix chat epoch

---------

Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: huangting4201 <1538303371@qq.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
 2023-09-19 13:49:48 +08:00
huangting4201 2710fa7343
Merge develop to main (#314) 
* feat: add unitest for model (#300)

* feat: add unitest for model

* feat:add model test

* Merge main to develop (#309)

* fix(chat): fix stream_chat to return generator (#123)

* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302)

* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299)

---------

Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>

* docs(doc/code-docs): add figure for training docs (#307)

* add training image for docs

* docs(doc/code-docs): add training img for en doc

* docs(doc/code-docs): fix en docs for initialize

* docs(doc/code-docs): update conf file for readthedocs

* docs(doc/code-docs): fix typos

* docs(doc/code-docs): fix typos for reathedocs

* docs(doc/code-docs): minor typo fix for reathedocs

* docs(doc/code-docs): fix readthedocs conf file

* docs(doc/code-docs): update training image

* docs(doc/code-docs): fix typos

* docs(doc/code-docs): update training image

* docs(doc/code-docs): move training image to section initialize

* docs(doc/code-docs): fix lint

* add badge about reathedocs status

* Merge main to develop (#312)

* fix(chat): fix stream_chat to return generator (#123)

* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302)

* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299)

* docs(doc/code-docs): update quickstart usage (#301)

* docs(usage.md): update usage.md

* docs(doc/code-docs): update en usage

---------

Co-authored-by: huangting4201 <huangting3@sensetime.com>

* docs(doc/code-docs): update en usage

---------

Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
Co-authored-by: huangting4201 <huangting3@sensetime.com>

* feat: more tgs (#310)

* feat:more tgs

* feat:add more tgs

* feat:more tgs

* feat: add optimizer_unitest (#303)

* feat: add optimizer_unitest

* feat: add optimizer test

* feat: add optimizer test

* feat:add optimizer test

* fianl change

* feat:add optimizer test

* feat:add optimizer test

* feat:add optimizer test

---------

Co-authored-by: jiaxingli <43110891+li126com@users.noreply.github.com>
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
Co-authored-by: Season <caizheng@pjlab.org.cn>
Co-authored-by: huangting4201 <huangting3@sensetime.com>
 2023-09-15 19:12:38 +08:00
jiaxingli ab513e1ddd
feat: add optimizer_unitest (#303) 
* feat: add optimizer_unitest

* feat: add optimizer test

* feat: add optimizer test

* feat:add optimizer test

* fianl change

* feat:add optimizer test

* feat:add optimizer test

* feat:add optimizer test
 2023-09-15 18:56:56 +08:00
jiaxingli 794a484666
feat: more tgs (#310) 
* feat:more tgs

* feat:add more tgs

* feat:more tgs
 2023-09-15 18:56:11 +08:00
huangting4201 607f691e16
Merge main to develop (#312) 
* fix(chat): fix stream_chat to return generator (#123)

* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302)

* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299)

* docs(doc/code-docs): update quickstart usage (#301)

* docs(usage.md): update usage.md

* docs(doc/code-docs): update en usage

---------

Co-authored-by: huangting4201 <huangting3@sensetime.com>

* docs(doc/code-docs): update en usage

---------

Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
Co-authored-by: huangting4201 <huangting3@sensetime.com>
 2023-09-15 16:19:26 +08:00
huangting4201 42802a2b31
docs(doc/code-docs): update quickstart usage (#301) 
* docs(usage.md): update usage.md

* docs(doc/code-docs): update en usage

---------

Co-authored-by: huangting4201 <huangting3@sensetime.com>
 2023-09-15 15:29:58 +08:00
Season de68cc5007
docs(doc/code-docs): add figure for training docs (#307) 
* add training image for docs

* docs(doc/code-docs): add training img for en doc

* docs(doc/code-docs): fix en docs for initialize

* docs(doc/code-docs): update conf file for readthedocs

* docs(doc/code-docs): fix typos

* docs(doc/code-docs): fix typos for reathedocs

* docs(doc/code-docs): minor typo fix for reathedocs

* docs(doc/code-docs): fix readthedocs conf file

* docs(doc/code-docs): update training image

* docs(doc/code-docs): fix typos

* docs(doc/code-docs): update training image

* docs(doc/code-docs): move training image to section initialize

* docs(doc/code-docs): fix lint

* add badge about reathedocs status
 2023-09-15 15:22:22 +08:00
huangting4201 07fc5f674a
Merge main to develop (#309) 
* fix(chat): fix stream_chat to return generator (#123)

* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302)

* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299)

---------

Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
 2023-09-14 16:32:15 +08:00
jiaxingli 882a07011c
feat: add unitest for model (#300) 
* feat: add unitest for model

* feat:add model test
 2023-09-14 13:18:34 +08:00
jiangtann 09e71cebf3
fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299) 2023-09-11 20:17:11 +08:00
huangting4201 e354410bd2
fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302) 2023-09-11 20:06:22 +08:00
zhjunqin 8420115b5e
fix(chat): fix stream_chat to return generator (#123) 2023-09-10 23:46:45 +08:00
yingtongxiong 2ec20707d0 Merge remote-tracking branch 'origin/develop' 2023-09-08 20:42:55 +08:00
Guoteng 85e39aae67
fix(ckpt): fix snapshot none load error and remove file lock (#298) 2023-09-08 20:41:53 +08:00
yingtongxiong 9481df976f Merge remote-tracking branch 'origin/develop' 2023-09-08 17:58:04 +08:00
Sun Peng 1ee31ff9b1
feat: add runtime diag (#297) 
* feat: add runtime diag

* add diag_outlier_ratio

---------

Co-authored-by: yingtongxiong <974106207@qq.com>
 2023-09-08 17:56:46 +08:00
Season 06807a6fd5
docs(doc/code-docs): refine profiler docs (#295) 
* add detailed profiler guide

* added torch profiler detailed docs

* add english docs for profiler page

* docs(code-docs/source/profiler.rst): resize profiler trace image

* docs(code-docs/source/profiler.rst): fix typo

* docs(doc/imgs/torch_profiler_trace.png): update trace image
 2023-09-08 16:58:36 +08:00
Sun Peng 0423426c4c
fix: fix the bug to do bcast in a stream (#294) 
* fix: fix the bug to do bcast in a stream

* fix: fix the bug to do bcast in a stream

---------

Co-authored-by: yingtongxiong <974106207@qq.com>
 2023-09-08 13:53:40 +08:00
yingtongxiong 0c276d8de2 Merge remote-tracking branch 'origin/main' into develop 2023-09-08 10:19:54 +08:00
Sun Peng b7a8af8133
Feat/sync grad use async op (#277) 
* fix/brocast should not in commu stream

* fix/brocast should not in commu stream

* feat: support allreduce grad using async op

* fix bug of async op

* use reduceop.avg

* use torch flat

* delete unused stream

* delete unused stream

* feat: overap allreduce with memcapy

---------

Co-authored-by: yingtongxiong <974106207@qq.com>
 2023-09-07 21:51:30 +08:00
jiaopenglong 7c99e01ca7
fix(monitor): add alert switch and refactor monitor config (#285) 
* add monitor switch

* add switch to light monitor

* fix alert_address is empty

* fix light monitor heartbeat

* init light_monitor on rank_log only

* add comments to the monitoring config

* optimize config
 2023-09-07 21:49:05 +08:00
Guoteng 37b8c6684e
feat(utils): add timeout warpper for key functions (#286) 2023-09-07 17:26:17 +08:00
huangting4201 671c752de6
docs(doc/code-docs): support zh cn readthedocs (#289) 
* feat(code-docs): test auto doc

* feat(code-docs): test auto doc

* feat(code-docs): test auto doc

* feat(code-docs): test auto doc

* docs(doc/code-docs): add zh_CN structure

* docs(doc/code-docs): test install.md

* docs(doc/code-docs): source file to zh

* docs(doc/code-docs): update source files

* docs(doc/code-docs): add locales en

* docs(doc/code-docs): add locales en install

* docs(doc/code-docs): add locales en example

* docs(doc/code-docs): update en checkpoint

* add en translation for parallel.rst docs

* add en translation for profiler.po docs

* docs(doc/code-docs): update en monitor

* add en translation for monuitor, qa, training docs

* add en translation for quickstart docs

* docs(doc/code-docs): update monitor.po and usage.po

* docs(doc/code-docs): fix typos

* docs(doc/code-docs): update en parallel

* docs(doc/code-docs): update en parallel

* docs(doc/code-docs): update en usage

* docs(doc/code-docs): update en profilier

* docs(doc/code-docs): update en initialize

* docs(doc/code-docs): update en initialize

* docs(doc/code-docs): update en initialize

* docs(doc/code-docs): update en initialize

---------

Co-authored-by: zigzagcai <caizheng@pjlab.org.cn>
 2023-09-07 16:11:08 +08:00
Season b6d909d43e
docs(*): add documentation and reST files for readthedocs (#272) 
* add initial reST files for readthedocs

* fix typos

* docs refine and minor fix

* add references for parallel training section

* fix reST format

* fix reST format

* fix reST format

* add comments for trainer API

* add link to step-by-step quickstart guide

* docs(code-docs/source/parallel.rst): add paper link url

* docs(code-docs/source/parallel.rst): add paper link url

* use MyST to render markdown

* docs(code-docs/source/initialize.rst): update model init

* add requirements for myst-parser

* reuse install and usage markdown

* docs(code-docs/source/index.rst): add example and q&a

* docs(doc/code-docs/*): docs refine

* docs(code-docs/source/parallel.rst): update docs for zero config

* docs(code-docs/source/example.rst): fix typos for example.rst

* docs(code-docs/source/example.rst): refine docs

* docs(code-docs/source/example): update example

* docs(code-docs/source/example): delete useless example

* docs(code-docs/source/*): fix image display issue

* docs(code-docs/source/parallel.rst): add docs for communication overlap

* docs(code-docs/source/conf.py): update conf.py

* docs(code-docs/source/example): update example 30B demo

* docs(code-docs/source/parallel.rst): update pipeline parallel

* docs(code-docs/source/parallel.rst): update pipeline parallel

* docs(code-docs/source/parallel.rst): update pipeline parallel

* docs(code-docs/source/parallel.rst): update pipeline parallel

* docs(code-docs/source/parallel.rst): update ZeRO1.5

* docs(code-docs/source/parallel.rst): update ZeRO1.5

* docs(code-docs/source): fix word spelling error

---------

Co-authored-by: huangting4201 <huangting3@sensetime.com>
 2023-09-06 15:36:03 +08:00
Wenwen Qu 7f687bf4b3
fix(core/context): use dummy mode to generate random numbers in model construction (#266) 
* change mode to dummy in model construction and restore to data when done

* add comments

* move set_mode(.DATA) to initialize_model(.)
 2023-09-06 14:34:11 +08:00
Guoteng ff181bc5f8
fix(ckpt): fix checkpoint reload bug (#282) 
1. fix only_load tuple convert bug.
2. fix reload_zero_fp32_buff copy bug
 2023-09-06 04:05:04 +08:00
Guoteng 8acf823a04
fix(storage): fix and refactor storage api (#281) 2023-09-06 01:15:09 +08:00
jiaopenglong 8d8d811e10
feat(monitor): add light monitor (#275) 
* add light monitor

* filter key of metrics dict

* test no light_monitor case

* mv init_light_monitor to initialize_distributed_env
 2023-09-05 19:24:01 +08:00