github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
533 Commits
32 Branches
46 Tags
228 MiB
Commit Graph

533 Commits (11f54c7b6b2a4f515e03d48d1259256e2302daf7)

Author SHA1 Message Date
Jiarui Fang 53cb584808
[utils] correct cpu memory used and capacity in the context of multi-process (#726) 2022-04-12 14:57:54 +08:00
Jiarui Fang 7db3ccc79b
[hotfix] remove duplicated param register to stateful tensor manager (#728) 2022-04-12 13:55:25 +08:00
binmakeswell 600e769a42
add video (#732) 2022-04-12 13:41:56 +08:00
Frank Lee a5c3f072f6
[bug] removed zero installation requirements (#731) 2022-04-12 13:27:25 +08:00
HELSON b9b469ea50
[moe] add checkpoint for moe zero test (#729) 2022-04-12 12:11:54 +08:00
Frank Lee 6f7d1362c9
[doc] removed outdated installation command (#730) 2022-04-12 11:56:45 +08:00
FrankLeeeee e88a498c9c [test] removed trivial outdated test 2022-04-12 11:08:15 +08:00
FrankLeeeee 62b4ce7326 [test] added missing decorators to model checkpointing tests 2022-04-12 11:08:15 +08:00
Frank Lee 1cb7bdad3b
[util] fixed communication API depth with PyTorch 1.9 (#721) 2022-04-12 09:44:40 +08:00
Frank Lee 2412429d54
[util] fixed activation checkpointing on torch 1.9 (#719) 2022-04-12 09:35:45 +08:00
Frank Lee 04ff5ea546
[utils] support detection of number of processes on current node (#723) 2022-04-12 09:28:19 +08:00
Jiarui Fang 4d90a7b513
[refactor] zero directory (#724) 2022-04-11 23:13:02 +08:00
Frank Lee 20ab1f5520
[bug] fixed broken test_found_inf (#725) 2022-04-11 22:00:27 +08:00
Jiarui Fang 193dc8dacb
[refactor] refactor the memory utils (#715) 2022-04-11 16:47:57 +08:00
HELSON dbd96fe90a
[zero] check whether gradients have inf and nan in gpu (#712) 2022-04-11 15:40:13 +08:00
ver217 715b86eadd
[hotfix] fix stm cuda model data size (#710) 2022-04-11 15:10:39 +08:00
LuGY 140263a394
[hotfix]fixed bugs of assigning grad states to non leaf nodes (#711) 
* fixed bugs of assigning grad states to non leaf nodes

* use detach()
 2022-04-11 14:04:58 +08:00
Frank Lee eda30a058e
[compatibility] fixed tensor parallel compatibility with torch 1.9 (#700) 2022-04-11 13:44:50 +08:00
HELSON a9b8300d54
[zero] improve adaptability for not-shard parameters (#708) 
* adapt post grad hooks for not-shard parameters
* adapt optimizer for not-shard parameters
* offload gradients for not-replicated parameters
 2022-04-11 13:38:51 +08:00
ver217 ab8c6b4a0e
[zero] refactor memstats collector (#706) 
* refactor memstats collector

* fix disposable

* polish code
 2022-04-11 10:46:08 +08:00
アマデウス 3fc8a204dc
[]Corrected 3d vocab parallel embedding (#707) 2022-04-11 10:17:55 +08:00
HELSON ee112fe1da
[zero] adapt zero hooks for unsharded module (#699) 2022-04-08 20:23:26 +08:00
binmakeswell 896ade15d6
add PaLM link (#704) (#705) 2022-04-08 18:42:12 +08:00
binmakeswell 270157e9e7
add PaLM link (#704) 
* add PaLM link
 2022-04-08 18:26:59 +08:00
ver217 3c9cd5bb5e
[zero] stateful tensor manager (#687) 
* [WIP] stateful tensor manager

* add eviction strategy

* polish code

* polish code

* polish comment

* add unit test

* fix sampler bug

* polish code

* fix max sampling cnt resetting bug

* fix sampler bug

* polish code

* fix bug

* fix unit test

Co-authored-by: jiaruifang <fangjiarui123@gmail.com>
 2022-04-08 17:51:34 +08:00
ver217 70e8dd418b
[hotfix] update requirements-test (#701) 2022-04-08 16:52:36 +08:00
Frank Lee 1ae94ea85a
[ci] remove ipc config for rootless docker (#694) 2022-04-08 10:15:52 +08:00
github-actions[bot] d878d843ad
Automated submodule synchronization (#695) 
Co-authored-by: github-actions <github-actions@github.com>
 2022-04-08 10:03:53 +08:00
github-actions[bot] d50cdabbc9
Automated submodule synchronization (#556) 
Co-authored-by: github-actions <github-actions@github.com>
 2022-04-07 22:11:00 +08:00
Frank Lee dbe8e030fb
[ci] added missing field in workflow (#692) 2022-04-07 18:07:15 +08:00
Frank Lee 0372ed7951
[ci] update workflow trigger condition and support options (#691) 2022-04-07 17:53:03 +08:00
HELSON d7ecaf362b
[zero] fix init bugs in zero context (#686) 
* adapt model weight initialization for methods in Pytorch nn.init
 2022-04-07 17:38:45 +08:00
YuliangLiu0306 0ed7042f42
[pipeline] refactor pipeline (#679) 
* refactor pipeline---put runtime schedule into engine.

* add type hint for schedule Optional[BaseSchedule]

* preprocess schedule during engine initializing

* infer pipeline schedule params from config
 2022-04-07 15:54:14 +08:00
Frank Lee eace69387d
[ci] fixed compatibility workflow (#678) 2022-04-06 16:19:34 +08:00
Jiarui Fang 59bf2dc590
[zero] initialize a stateful tensor manager (#614) 2022-04-06 16:18:49 +08:00
Frank Lee cc236916c6
[ci] replace the dngc ocker image with self-built pytorch image (#672) 2022-04-06 14:10:17 +08:00
ver217 03e1d35931
[release] update version (#673) 2022-04-06 12:03:23 +08:00
encmps 79ccfa4310 [NFC] polish colossalai/kernel/cuda_native/csrc/multi_tensor_adam.cu code style (#667) 2022-04-06 11:40:59 +08:00
lucasliunju e4bcff9b0f [NFC] polish colossalai/builder/builder.py code style (#662) 2022-04-06 11:40:59 +08:00
shenggan 331683bf82 [NFC] polish colossalai/kernel/cuda_native/csrc/layer_norm_cuda_kernel.cu code style (#661) 2022-04-06 11:40:59 +08:00
FredHuang99 c336cd3066 [NFC] polish colossalai/communication/utils.py code style (#656) 2022-04-06 11:40:59 +08:00
MaxT 5ab9a71299 [NFC] polish colossalai/kernel/cuda_native/csrc/moe_cuda.cpp code style (#642) 2022-04-06 11:40:59 +08:00
Xue Fuzhao 10afec728f [NFC] polish colossalai/kernel/cuda_native/csrc/kernels/include/cuda_util.h code style (#641) 2022-04-06 11:40:59 +08:00
Cautiousss 055d0270c8 [NFC] polish colossalai/context/process_group_initializer/initializer_sequence.py colossalai/context/process_group_initializer initializer_tensor.py code style (#639) 
Co-authored-by: 何晓昕 <cautious@r-236-100-25-172.comp.nus.edu.sg>
 2022-04-06 11:40:59 +08:00
Ziheng Qin c7c224ee17 [NFC] polish colossalai/builder/pipeline.py code style (#638) 2022-04-06 11:40:59 +08:00
Sze-qq 10591ecdf9 [NFC] polish colossalai/kernel/cuda_native/csrc/cpu_adam.cpp code style (#636) 2022-04-06 11:40:59 +08:00
Wangbo Zhao 6fcb381801 [NFC] polish colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu code style (#635) 2022-04-06 11:40:59 +08:00
ExtremeViscent 8a5d526e95 [NFC] polish colossalai/kernel/cuda_native/csrc/kernels/dropout_kernels.cu and cross_entropy.cu code style (#634) 2022-04-06 11:40:59 +08:00
RichardoLuo ad1e7ab2b2 '[NFC] polish <colossalai/engine/_base_engine.py> code style' (#631) 
Co-authored-by: RichardoLuo <14049555596@qq.com>
 2022-04-06 11:40:59 +08:00
Zangwei 2e11853d04 [NFC] polish colossalai/communication/ring.py code style (#630) 2022-04-06 11:40:59 +08:00