c1e30cff2c
feat(numa): bind numa if possible ( #320 )
...
* feat:add numa
* feat:add bind numa
* feat:add bind numa
* feat:add bind numa
* feat: bind numa
* feat: bind numa
* feat: add numa
* feat:add numa
* feat:add numa
* try_bind_numa should not raise exception
---------
Co-authored-by: 877825076@qq.com <877825076@qq.com>
2023-09-25 19:34:52 +08:00
9284303a6d
doc(monitor): add light monitoring doc ( #352 )
...
* add light monitoring doc
* update light monitoring doc
* update light monitoring doc
* update light monitoring doc
* update light monitoring doc continue
* update light monitoring doc continue
* update monitor config doc
* update monitor config doc continue
* update monitor config doc continue
2023-09-25 19:28:09 +08:00
847cc819dd
fix(monitor): add volc and aliyun jobid ( #338 )
...
* add volc and aliyun jobid
* rm workspaceid
2023-09-25 17:58:32 +08:00
064965527b
fix(config): monitor config key error when args_check is False ( #362 )
...
* add monitor switch
* add switch to light monitor
* fix alert_address is empty
* fix light monitor heartbeat
* init light_monitor on rank_log only
* add comments to the monitoring config
* optimize config
* fix monitor config key error when args_check is False
2023-09-25 17:30:36 +08:00
26a7397752
fix(storage): fix try_get_storage_backend ( #359 )
...
* fix(storage): fix try_get_storage_backend
* fix typo and print infos only in log rank
* fix typo and print infos only in log rank
---------
Co-authored-by: gaoyang07 <Gary1546308416AL@gmail.com>
2023-09-25 15:16:25 +08:00
a86c4bbbfd
Merge branch 'main' into develop
2023-09-22 19:24:03 +08:00
d1e52f0c03
feat(doc/code-docs): add checkpoint save/load usage doc ( #311 )
...
* feat(doc): add checkpoint doc
* fix checkpoint doc
* fix comment
* fix(doc/code-docs): remove fuzzy
* fix(doc/code-docs): fix some errors
* fix(doc/code-docs): minor fix
---------
Co-authored-by: li126com <li126com2@126.com>
Co-authored-by: huangting4201 <1538303371@qq.com>
2023-09-22 18:45:33 +08:00
1ed36754df
feat(.github/workflows): update ci e2e tests and add ci unit tests ( #324 )
...
* feat(.github/workflows/e2e_test.yaml): update e2e yaml
* feat(.github/workflows/e2e_test.yaml): update e2e yaml
* test e2e
* test e2e
* test e2e
* test e2e
* test e2e
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): test ci
* fix(ci): add weekly tests
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-22 14:07:14 +08:00
f5337f6e02
Feat(PythonGC): Do garbage collection manually ( #326 )
...
* feat:add gc control
* feat:add gc control
* feat:add gc control
* feat:add gc
* re-lint
2023-09-22 13:52:25 +08:00
3b0eff0c8a
fix(model/embedding.py): ci lint check error ( #345 )
...
* fix(ci): fix ci lint error
* fix(ci): fix ci lint error
2023-09-21 14:46:22 +08:00
8464425a7b
feat(mdoel): add DynamicNTKScalingRotaryEmbedding ( #339 )
...
* add dynamic ntk rope
* update dynamic ntk rope
* fix lint check
* fix lint check
* add more desc
---------
Co-authored-by: YWMditto <862779238@qq.com>
2023-09-20 23:31:47 +08:00
e611817442
fix(doc): add 20b releasing info to readme ( #330 )
...
* fix(eval): StreamingDataset does not have an __len__ method.
* doc(readme): update readme
* update readme
* update readme
* update readme
* update readme
* update readme
* update readme
2023-09-20 16:46:45 +08:00
5e5d160685
fix(readme): fix readme about 20B releasing ( #329 )
...
* fix(eval): StreamingDataset does not have an __len__ method.
* doc(readme): update readme
* update readme
* update readme
* update readme
* update readme
* update readme
2023-09-20 16:26:43 +08:00
2a09ebd5c1
doc(readme): update readme, add 20B releasing info ( #328 )
...
* fix(eval): StreamingDataset does not have an __len__ method.
* doc(readme): update readme
* update readme
2023-09-20 16:04:43 +08:00
67eda4cbe1
fix(.github/workflows/e2e_test.yaml): update ci runner name
2023-09-19 18:13:20 +08:00
30b21075e8
merge main
2023-09-19 18:04:47 +08:00
6a5915bf0d
feat(linear): optimize mlp by using jit ( #321 )
...
* fuse silu op
* refactor code
* fix lint
* fix lint
2023-09-19 14:57:43 +08:00
025ca55dfe
test(tests/test_training): add training e2e tests for loss spike and loss accuracy ( #304 )
...
* tests(test_training): add test case for loss accuracy
* tests(test_training): update test cases
* ci(.github/workflows/e2e_test.yaml): remove pull submodule
* ci(.github/workflows/e2e_test.yaml): update ci env and remove useless env var
* test(tests/test_training): add 16 GPUs test cases
* test(tests/test_training): fix training_16GPU_8DP2PP test case error
* test(tests/test_training): add new case for interleaved pp
* test(tests/test_training): remove redundant code
* test(tests/test_training): update ci job timeout minutes to 30m
* feat(initialize/launch.py): check num_chunks and interleaved_overlap
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-19 14:55:40 +08:00
bfefc4ea3c
test(ci_scripts): move ci env ( #317 )
...
* change partition and runner label
* change rm action to mv
* use spot
* use rsync to move test files
* remove *
* remove *
* change into llm_s partition
---------
Co-authored-by: wangmengke <wangmengke@pjlab.org.cn>
2023-09-19 14:52:32 +08:00
b9824fab89
fix(tools): fix yield bug in stream_chat ( #315 )
2023-09-19 14:18:02 +08:00
cd6426a249
feat(tools): support openai api ( #313 )
...
* fix(chat): fix stream_chat to return generator (#123 )
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302 )
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299 )
* support openai api to deploy internlm
* update README for information os openai_api.py
* change example in README_EN.md to English
* delete unnecessary print; fix model card typo; fix chat epoch
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: huangting4201 <1538303371@qq.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
2023-09-19 13:49:48 +08:00
2710fa7343
Merge develop to main ( #314 )
...
* feat: add unitest for model (#300 )
* feat: add unitest for model
* feat:add model test
* Merge main to develop (#309 )
* fix(chat): fix stream_chat to return generator (#123 )
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302 )
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299 )
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
* docs(doc/code-docs): add figure for training docs (#307 )
* add training image for docs
* docs(doc/code-docs): add training img for en doc
* docs(doc/code-docs): fix en docs for initialize
* docs(doc/code-docs): update conf file for readthedocs
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): fix typos for reathedocs
* docs(doc/code-docs): minor typo fix for reathedocs
* docs(doc/code-docs): fix readthedocs conf file
* docs(doc/code-docs): update training image
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): update training image
* docs(doc/code-docs): move training image to section initialize
* docs(doc/code-docs): fix lint
* add badge about reathedocs status
* Merge main to develop (#312 )
* fix(chat): fix stream_chat to return generator (#123 )
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302 )
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299 )
* docs(doc/code-docs): update quickstart usage (#301 )
* docs(usage.md): update usage.md
* docs(doc/code-docs): update en usage
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
* docs(doc/code-docs): update en usage
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
Co-authored-by: huangting4201 <huangting3@sensetime.com>
* feat: more tgs (#310 )
* feat:more tgs
* feat:add more tgs
* feat:more tgs
* feat: add optimizer_unitest (#303 )
* feat: add optimizer_unitest
* feat: add optimizer test
* feat: add optimizer test
* feat:add optimizer test
* fianl change
* feat:add optimizer test
* feat:add optimizer test
* feat:add optimizer test
---------
Co-authored-by: jiaxingli <43110891+li126com@users.noreply.github.com>
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
Co-authored-by: Season <caizheng@pjlab.org.cn>
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-15 19:12:38 +08:00
ab513e1ddd
feat: add optimizer_unitest ( #303 )
...
* feat: add optimizer_unitest
* feat: add optimizer test
* feat: add optimizer test
* feat:add optimizer test
* fianl change
* feat:add optimizer test
* feat:add optimizer test
* feat:add optimizer test
2023-09-15 18:56:56 +08:00
794a484666
feat: more tgs ( #310 )
...
* feat:more tgs
* feat:add more tgs
* feat:more tgs
2023-09-15 18:56:11 +08:00
607f691e16
Merge main to develop ( #312 )
...
* fix(chat): fix stream_chat to return generator (#123 )
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302 )
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299 )
* docs(doc/code-docs): update quickstart usage (#301 )
* docs(usage.md): update usage.md
* docs(doc/code-docs): update en usage
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
* docs(doc/code-docs): update en usage
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-15 16:19:26 +08:00
42802a2b31
docs(doc/code-docs): update quickstart usage ( #301 )
...
* docs(usage.md): update usage.md
* docs(doc/code-docs): update en usage
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-15 15:29:58 +08:00
de68cc5007
docs(doc/code-docs): add figure for training docs ( #307 )
...
* add training image for docs
* docs(doc/code-docs): add training img for en doc
* docs(doc/code-docs): fix en docs for initialize
* docs(doc/code-docs): update conf file for readthedocs
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): fix typos for reathedocs
* docs(doc/code-docs): minor typo fix for reathedocs
* docs(doc/code-docs): fix readthedocs conf file
* docs(doc/code-docs): update training image
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): update training image
* docs(doc/code-docs): move training image to section initialize
* docs(doc/code-docs): fix lint
* add badge about reathedocs status
2023-09-15 15:22:22 +08:00
07fc5f674a
Merge main to develop ( #309 )
...
* fix(chat): fix stream_chat to return generator (#123 )
* fix(configs/7B_sft.py): model dtype float16 to bfloat16 (#302 )
* fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError (#299 )
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
Co-authored-by: zhjunqin <zhjunqin@users.noreply.github.com>
Co-authored-by: jiangtann <39088437+jiangtann@users.noreply.github.com>
2023-09-14 16:32:15 +08:00
882a07011c
feat: add unitest for model ( #300 )
...
* feat: add unitest for model
* feat:add model test
2023-09-14 13:18:34 +08:00
09e71cebf3
fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError ( #299 )
2023-09-11 20:17:11 +08:00
e354410bd2
fix(configs/7B_sft.py): model dtype float16 to bfloat16 ( #302 )
2023-09-11 20:06:22 +08:00
8420115b5e
fix(chat): fix stream_chat to return generator ( #123 )
2023-09-10 23:46:45 +08:00
2ec20707d0
Merge remote-tracking branch 'origin/develop'
2023-09-08 20:42:55 +08:00
85e39aae67
fix(ckpt): fix snapshot none load error and remove file lock ( #298 )
2023-09-08 20:41:53 +08:00
9481df976f
Merge remote-tracking branch 'origin/develop'
2023-09-08 17:58:04 +08:00
1ee31ff9b1
feat: add runtime diag ( #297 )
...
* feat: add runtime diag
* add diag_outlier_ratio
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
2023-09-08 17:56:46 +08:00
06807a6fd5
docs(doc/code-docs): refine profiler docs ( #295 )
...
* add detailed profiler guide
* added torch profiler detailed docs
* add english docs for profiler page
* docs(code-docs/source/profiler.rst): resize profiler trace image
* docs(code-docs/source/profiler.rst): fix typo
* docs(doc/imgs/torch_profiler_trace.png): update trace image
2023-09-08 16:58:36 +08:00
0423426c4c
fix: fix the bug to do bcast in a stream ( #294 )
...
* fix: fix the bug to do bcast in a stream
* fix: fix the bug to do bcast in a stream
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
2023-09-08 13:53:40 +08:00
0c276d8de2
Merge remote-tracking branch 'origin/main' into develop
2023-09-08 10:19:54 +08:00
b7a8af8133
Feat/sync grad use async op ( #277 )
...
* fix/brocast should not in commu stream
* fix/brocast should not in commu stream
* feat: support allreduce grad using async op
* fix bug of async op
* use reduceop.avg
* use torch flat
* delete unused stream
* delete unused stream
* feat: overap allreduce with memcapy
---------
Co-authored-by: yingtongxiong <974106207@qq.com>
2023-09-07 21:51:30 +08:00
7c99e01ca7
fix(monitor): add alert switch and refactor monitor config ( #285 )
...
* add monitor switch
* add switch to light monitor
* fix alert_address is empty
* fix light monitor heartbeat
* init light_monitor on rank_log only
* add comments to the monitoring config
* optimize config
2023-09-07 21:49:05 +08:00
37b8c6684e
feat(utils): add timeout warpper for key functions ( #286 )
2023-09-07 17:26:17 +08:00
671c752de6
docs(doc/code-docs): support zh cn readthedocs ( #289 )
...
* feat(code-docs): test auto doc
* feat(code-docs): test auto doc
* feat(code-docs): test auto doc
* feat(code-docs): test auto doc
* docs(doc/code-docs): add zh_CN structure
* docs(doc/code-docs): test install.md
* docs(doc/code-docs): source file to zh
* docs(doc/code-docs): update source files
* docs(doc/code-docs): add locales en
* docs(doc/code-docs): add locales en install
* docs(doc/code-docs): add locales en example
* docs(doc/code-docs): update en checkpoint
* add en translation for parallel.rst docs
* add en translation for profiler.po docs
* docs(doc/code-docs): update en monitor
* add en translation for monuitor, qa, training docs
* add en translation for quickstart docs
* docs(doc/code-docs): update monitor.po and usage.po
* docs(doc/code-docs): fix typos
* docs(doc/code-docs): update en parallel
* docs(doc/code-docs): update en parallel
* docs(doc/code-docs): update en usage
* docs(doc/code-docs): update en profilier
* docs(doc/code-docs): update en initialize
* docs(doc/code-docs): update en initialize
* docs(doc/code-docs): update en initialize
* docs(doc/code-docs): update en initialize
---------
Co-authored-by: zigzagcai <caizheng@pjlab.org.cn>
2023-09-07 16:11:08 +08:00
b6d909d43e
docs(*): add documentation and reST files for readthedocs ( #272 )
...
* add initial reST files for readthedocs
* fix typos
* docs refine and minor fix
* add references for parallel training section
* fix reST format
* fix reST format
* fix reST format
* add comments for trainer API
* add link to step-by-step quickstart guide
* docs(code-docs/source/parallel.rst): add paper link url
* docs(code-docs/source/parallel.rst): add paper link url
* use MyST to render markdown
* docs(code-docs/source/initialize.rst): update model init
* add requirements for myst-parser
* reuse install and usage markdown
* docs(code-docs/source/index.rst): add example and q&a
* docs(doc/code-docs/*): docs refine
* docs(code-docs/source/parallel.rst): update docs for zero config
* docs(code-docs/source/example.rst): fix typos for example.rst
* docs(code-docs/source/example.rst): refine docs
* docs(code-docs/source/example): update example
* docs(code-docs/source/example): delete useless example
* docs(code-docs/source/*): fix image display issue
* docs(code-docs/source/parallel.rst): add docs for communication overlap
* docs(code-docs/source/conf.py): update conf.py
* docs(code-docs/source/example): update example 30B demo
* docs(code-docs/source/parallel.rst): update pipeline parallel
* docs(code-docs/source/parallel.rst): update pipeline parallel
* docs(code-docs/source/parallel.rst): update pipeline parallel
* docs(code-docs/source/parallel.rst): update pipeline parallel
* docs(code-docs/source/parallel.rst): update ZeRO1.5
* docs(code-docs/source/parallel.rst): update ZeRO1.5
* docs(code-docs/source): fix word spelling error
---------
Co-authored-by: huangting4201 <huangting3@sensetime.com>
2023-09-06 15:36:03 +08:00
7f687bf4b3
fix(core/context): use dummy mode to generate random numbers in model construction ( #266 )
...
* change mode to dummy in model construction and restore to data when done
* add comments
* move set_mode(.DATA) to initialize_model(.)
2023-09-06 14:34:11 +08:00
ff181bc5f8
fix(ckpt): fix checkpoint reload bug ( #282 )
...
1. fix only_load tuple convert bug.
2. fix reload_zero_fp32_buff copy bug
2023-09-06 04:05:04 +08:00
8acf823a04
fix(storage): fix and refactor storage api ( #281 )
2023-09-06 01:15:09 +08:00
8d8d811e10
feat(monitor): add light monitor ( #275 )
...
* add light monitor
* filter key of metrics dict
* test no light_monitor case
* mv init_light_monitor to initialize_distributed_env
2023-09-05 19:24:01 +08:00
9445faf5be
fix(model): set tensor parallel attribute for mlp ( #271 )
...
* set is_tensor_parallel attribute for mlp
* fix lint
2023-09-05 19:03:02 +08:00
0fb8d4141f
Merge remote-tracking branch 'origin/main' into develop
2023-09-05 17:50:35 +08:00
jiaxingli