03654c0ce2
fix localhost measurement ( #4320 )
2023-08-01 10:14:00 +08:00
45b08f08cb
[zero] optimize the optimizer step time ( #4221 )
...
* optimize the optimizer step time
* fix corner case
* polish
* replace all-reduce with all-gather
* set comm device to cuda
2023-07-31 22:13:29 +08:00
1a49a5ea00
[zero] support shard optimizer state dict of zero ( #4194 )
...
* support shard optimizer of zero
* polish code
* support sync grad manually
2023-07-31 22:13:29 +08:00
dd7cc58299
[zero] add state dict for low level zero ( #4179 )
...
* add state dict for zero
* fix unit test
* polish
2023-07-31 22:13:29 +08:00
c668801d36
[zero] allow passing process group to zero12 ( #4153 )
...
* allow passing process group to zero12
* union tp-zero and normal-zero
* polish code
2023-07-31 22:13:29 +08:00
79cf1b5f33
[zero]support no_sync method for zero1 plugin ( #4138 )
...
* support no sync for zero1 plugin
* polish
* polish
2023-07-31 22:13:29 +08:00
c6ab96983a
[zero] refactor low level zero for shard evenly ( #4030 )
...
* refactor low level zero
* fix zero2 and support cpu offload
* avg gradient and modify unit test
* refactor grad store, support layer drop
* refactor bucket store, support grad accumulation
* fix and update unit test of zero and ddp
* compatible with tp, ga and unit test
* fix memory leak and polish
* add zero layer drop unittest
* polish code
* fix import err in unit test
* support diffenert comm dtype, modify docstring style
* polish code
* test padding and fix
* fix unit test of low level zero
* fix pad recording in bucket store
* support some models
* polish
2023-07-31 22:13:29 +08:00
5187c96b7c
support session-based training ( #4313 )
...
Co-authored-by: Yuanchen Xu <yuanchen.xu00@gmail.com>
2023-07-28 11:29:55 +08:00
ef4b99ebcd
add llama example CI
2023-07-26 14:12:57 +08:00
0991405361
[NFC] polish applications/Chat/coati/models/utils.py codestyle ( #4277 )
...
* [NFC] polish colossalai/context/random/__init__.py code style
* [NFC] polish applications/Chat/coati/models/utils.py code style
2023-07-26 14:12:57 +08:00
9e512938f6
[NFC] polish applications/Chat/coati/trainer/strategies/base.py code style ( #4278 )
2023-07-26 14:12:57 +08:00
c972d65311
applications/Chat/.gitignore ( #4279 )
...
Co-authored-by: henryqin1997 <henryqin1997@gamil.com>
2023-07-26 14:12:57 +08:00
709e121cd5
[NFC] polish applications/Chat/coati/models/generation.py code style ( #4275 )
2023-07-26 14:12:57 +08:00
dc1b6127f9
[NFC] polish applications/Chat/inference/server.py code style ( #4274 )
...
Co-authored-by: Yuanchen Xu <yuanchen.xu00@gmail.com>
2023-07-26 14:12:57 +08:00
caa4433072
[NFC] fix format of application/Chat/coati/trainer/utils.py ( #4273 )
2023-07-26 14:12:57 +08:00
1ce997daaf
[NFC] polish applications/Chat/examples/train_reward_model.py code style ( #4271 )
2023-07-26 14:12:57 +08:00
a50d39a143
[NFC] fix: format ( #4270 )
...
* [NFC] polish colossalai/fx/profiler/experimental/profiler_module/embedding.py code style
* [NFC] polish colossalai/communication/utils.py code style
---------
Co-authored-by: Minghao Huang <huangminghao@luchentech.com>
2023-07-26 14:12:57 +08:00
fee553288b
[NFC] polish runtime_preparation_pass style ( #4266 )
2023-07-26 14:12:57 +08:00
3883db452c
[NFC] polish unary_elementwise_generator.py code style ( #4267 )
...
Co-authored-by: aye42 <aye42@gatech.edu>
2023-07-26 14:12:57 +08:00
798cb72907
[NFC] polish applications/Chat/coati/trainer/base.py code style ( #4260 )
2023-07-26 14:12:57 +08:00
b2debdc09b
[NFC] polish applications/Chat/coati/dataset/sft_dataset.py code style ( #4259 )
2023-07-26 14:12:57 +08:00
abe4f971e0
[NFC] polish colossalai/booster/plugin/low_level_zero_plugin.py code style ( #4256 )
...
Co-authored-by: supercooledith <893754954@qq.com>
2023-07-26 14:12:57 +08:00
c614a99d28
[NFC] polish colossalai/auto_parallel/offload/amp_optimizer.py code style ( #4255 )
2023-07-26 14:12:57 +08:00
85774f0c1f
[NFC] polish colossalai/cli/benchmark/utils.py code style ( #4254 )
2023-07-26 14:12:57 +08:00
dee1c96344
[NFC] policy applications/Chat/examples/ray/mmmt_prompt.py code style ( #4250 )
2023-07-26 14:12:57 +08:00
77c469e1ba
[NFC] polish applications/Chat/coati/models/base/actor.py code style ( #4248 )
2023-07-26 14:12:57 +08:00
915ed8bed1
[NFC] polish applications/Chat/inference/requirements.txt code style ( #4265 )
2023-07-26 14:12:57 +08:00
86cf6aed5b
Fix/format ( #4261 )
...
* revise shardformer readme (#4246 )
* [example] add llama pretraining (#4257 )
* [NFC] polish colossalai/communication/p2p.py code style
---------
Co-authored-by: Jianghai <72591262+CjhHa1@users.noreply.github.com>
Co-authored-by: binmakeswell <binmakeswell@gmail.com>
Co-authored-by: Qianran Ma <qianranm@luchentech.com>
2023-07-26 14:12:57 +08:00
b366f1d99f
[NFC] Fix format for mixed precision ( #4253 )
...
* [NFC] polish colossalai/booster/mixed_precision/mixed_precision_base.py code style
2023-07-26 14:12:57 +08:00
02192a632e
[ci] support testmon core pkg change detection ( #4305 )
2023-07-21 18:36:35 +08:00
c6f6005990
[checkpointio] Sharded Optimizer Checkpoint for Gemini Plugin ( #4302 )
...
* sharded optimizer checkpoint for gemini plugin
* modify test to reduce testing time
* update doc
* fix bug when keep_gatherd is true under GeminiPlugin
2023-07-21 14:39:01 +08:00
fc5cef2c79
[lazy] support init on cuda ( #4269 )
...
* [lazy] support init on cuda
* [test] update lazy init test
* [test] fix transformer version
2023-07-19 16:43:01 +08:00
4b977541a8
[Kernels] added triton-implemented of self attention for colossal-ai ( #4241 )
...
* added softmax kernel
* added qkv_kernel
* added ops
* adding tests
* upload tets
* fix tests
* debugging
* debugging tests
* debugging
* added
* fixed errors
* added softmax kernel
* clean codes
* added tests
* update tests
* update tests
* added attention
* add
* fixed pytest checking
* add cuda check
* fix cuda version
* fix typo
2023-07-18 23:53:38 +08:00
7ff11b5537
[example] add llama pretraining ( #4257 )
2023-07-17 21:07:44 +08:00
9a4842c571
revise shardformer readme ( #4246 )
2023-07-17 17:30:57 +08:00
4e9b09c222
Automated submodule synchronization ( #4217 )
...
Co-authored-by: github-actions <github-actions@github.com>
2023-07-12 17:35:58 +08:00
c1cf752021
[docker] fixed ninja build command ( #4203 )
...
* [docker] fixed ninja build command
* polish code
2023-07-10 11:48:27 +08:00
58913441a1
Next commit [checkpointio] Unsharded Optimizer Checkpoint for Gemini Plugin ( #4141 )
...
* [checkpointio] unsharded optimizer checkpoint for Gemini plugin
* [checkpointio] unsharded optimizer checkpoint for Gemini using all_gather
2023-07-07 16:33:06 +08:00
fee32a3b78
[docker] added ssh and rdma support for docker ( #4192 )
2023-07-07 15:31:51 +08:00
190a6ea9c2
[dtensor] fixed readme file name and removed deprecated file ( #4162 )
2023-07-04 18:21:11 +08:00
cc3cbe9f6f
[workflow] show test duration ( #4159 )
2023-07-04 18:11:46 +08:00
1908caad38
[cli] hotfix launch command for multi-nodes ( #4165 )
2023-07-04 17:54:40 +08:00
2ac24040eb
fix some typo colossalai/shardformer ( #4160 )
2023-07-04 17:53:39 +08:00
c77b3b19be
[format] applied code formatting on changed files in pull request 4152 ( #4157 )
...
Co-authored-by: github-actions <github-actions@github.com>
2023-07-04 16:07:47 +08:00
f447ca1811
[chat] removed cache file ( #4155 )
2023-07-04 16:05:01 +08:00
89f45eda5a
[shardformer] added development protocol for standardization ( #4149 )
2023-07-04 16:05:01 +08:00
1fb0d95df0
[shardformer] made tensor parallelism configurable ( #4144 )
...
* [shardformer] made tensor parallelism configurable
* polish code
2023-07-04 16:05:01 +08:00
74257cb446
[shardformer] refactored some doc and api ( #4137 )
...
* [shardformer] refactored some doc and api
* polish code
2023-07-04 16:05:01 +08:00
7f9b30335b
[shardformer] write an shardformer example with bert finetuning ( #4126 )
...
* [shardformer] add benchmark of shardformer
* [shardformer] add benchmark of shardformer
2023-07-04 16:05:01 +08:00
ae035d305d
[shardformer] added embedding gradient check ( #4124 )
2023-07-04 16:05:01 +08:00
LuGY