github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
939 Commits
32 Branches
46 Tags
228 MiB
Commit Graph

666 Commits (89c434a0a67884a1978f1937aa2cb8c04b60866f)

Author SHA1 Message Date
Frank Lee 75abc75c15
[fx] fixed compatiblity issue with torch 1.10 (#1331) 2022-07-18 11:41:27 +08:00
ver217 7a05367101
[hotfix] shared model returns cpu state_dict (#1328) 2022-07-15 22:11:37 +08:00
Frank Lee b2475d8c5c
[fx] fixed unit tests for torch 1.12 (#1327) 2022-07-15 18:22:15 +08:00
HELSON d49708ae43
[hotfix] fix ddp for unit test test_gpt2 (#1326) 2022-07-15 18:19:52 +08:00
Frank Lee 250be4d31e
[utils] integrated colotensor with lazy init context (#1324) 
* [utils] integrated colotensor with lazy init context

* polish code

* polish code

* polish code
 2022-07-15 17:47:12 +08:00
YuliangLiu0306 e8acf55e8b
[fx] add balanced policy v2 (#1251) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [fx] add balanced policy v2

* add unittest
 2022-07-15 14:54:26 +08:00
XYE ca2d3f284f
[fx] Add unit test and fix bugs for transform_mlp_pass (#1299) 
* add test and fix bugs

* add functions back

* add comments
 2022-07-15 14:37:58 +08:00
HELSON 1b41686461
[hotfix] fix unit test test_module_spec (#1321) 2022-07-15 14:02:32 +08:00
Jiarui Fang 9e4c6449b0
[checkpoint] add ColoOptimizer checkpointing (#1316) 2022-07-15 09:52:55 +08:00
ver217 7c70bfbefa
[hotfix] fix PipelineSharedModuleGradientHandler (#1314) 2022-07-14 17:31:13 +08:00
Jiarui Fang 85f933b58b
[Optimizer] Remove useless ColoOptimizer (#1312) 2022-07-14 16:57:48 +08:00
Jiarui Fang 9f10524313
[Optimizer] polish the init method of ColoOptimizer (#1310) 2022-07-14 16:37:33 +08:00
Jiarui Fang 3ef3791a3b
[checkpoint] add test for bert and hotfix save bugs (#1297) 2022-07-14 15:38:18 +08:00
Frank Lee 4f4d8c3656
[fx] added apex normalization to patched modules (#1300) 
* [fx] added apex normalization to patched modules

* remove unused imports
 2022-07-14 14:24:13 +08:00
Jiarui Fang 4165eabb1e
[hotfix] remove potiential circle import (#1307) 
* make it faster

* [hotfix] remove circle import
 2022-07-14 13:44:26 +08:00
HELSON 260a55804a
[hotfix] fix shape error in backward when using ColoTensor (#1298) 2022-07-13 23:06:12 +08:00
runluo f83c4d6597
[NFC] polish colossalai/nn/layer/wrapper/pipeline_wrapper.py code style (#1303) 2022-07-13 19:01:07 +08:00
binmakeswell 7696cead8d Recover kernal files 2022-07-13 12:08:21 +08:00
XYE e83b2ce853 [NFC] polish colossalai/nn/layer/vanilla/layers.py code style (#1295) 2022-07-13 12:08:21 +08:00
Liping233 1000a41fd5 [NFC] polish colossalai/nn/layer/vanilla/__init__.py code style (#1293) 2022-07-13 12:08:21 +08:00
Maruyama_Aya 87f679aeae [NFC] polish colossalai/kernel/cuda_native/csrc/kernels/include/kernels.h code style (#1291) 2022-07-13 12:08:21 +08:00
Wangbo Zhao(黑色枷锁) 552667825b [NFC] polish colossalai/nn/layer/parallel_1d/layers.py code style (#1290) 2022-07-13 12:08:21 +08:00
doubleHU d6f5ef8860 [NFC] polish colossalai/kernel/cuda_native/csrc/kernels/transform_kernels.cu code style (#1286) 2022-07-13 12:08:21 +08:00
Ziheng Qin 6d6c01e94d [NFC] polish colossalai/__init__.py code style (#1285) 2022-07-13 12:08:21 +08:00
Jiatong Han 38e3ccd1e9 [NFC] polish colossalai/nn/layer/parallel_sequence/layers.py code style (#1280) 
Co-authored-by: JThh <jiatong.han@u.nus.edu>
 2022-07-13 12:08:21 +08:00
Boyuan Yao b414eaa5db [NFC] polish colossalai/nn/optimizer/lamb.py code style (#1275) 2022-07-13 12:08:21 +08:00
yuxuan-lou 5f6ab35d25 Hotfix/format (#1274) 
* [NFC] Polish colossalai/kernel/cuda_native/csrc/multi_tensor_lamb.cu code style. (#937)

* [NFC] polish colossalai/kernel/cuda_native/csrc/kernels/include/cuda_util.h code style

* [NFC] polish colossalai/kernel/cuda_native/csrc/scaled_masked_softmax.cpp code style

Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>
 2022-07-13 12:08:21 +08:00
Super Daniel 52d145a342 [NFC] polish colossalai/nn/lr_scheduler/onecycle.py code style (#1269) 2022-07-13 12:08:21 +08:00
Geng Zhang 0e06f62160 [NFC] polish colossalai/nn/layer/parallel_sequence/_operation.py code style (#1266) 2022-07-13 12:08:21 +08:00
binmakeswell c95e18cdb9 [NFC] polish colossalai/kernel/cuda_native/csrc/scaled_upper_triang_masked_softmax.h code style (#1270) 2022-07-13 12:08:21 +08:00
xyupeng 94bfd35184 [NFC] polish colossalai/builder/builder.py code style (#1265) 2022-07-13 12:08:21 +08:00
DouJS db13f96333 [NFC] polish colossalai/kernel/cuda_native/csrc/multi_tensor_apply.cuh code style (#1264) 2022-07-13 12:08:21 +08:00
shenggan 5d7366b144 [NFC] polish colossalai/kernel/cuda_native/csrc/scaled_masked_softmax.h code style (#1263) 2022-07-13 12:08:21 +08:00
Zangwei Zheng 197a2c89e2 [NFC] polish colossalai/communication/collective.py (#1262) 2022-07-13 12:08:21 +08:00
ziyu huang f1cafcc73a [NFC] polish colossalai/kernel/cuda_native/csrc/kernels/dropout_kernels.cu code style (#1261) 
Co-authored-by: “Arsmart123 <202476410arsmart@gmail.com>
 2022-07-13 12:08:21 +08:00
Sze-qq f8b9aaef47 [NFC] polish colossalai/kernel/cuda_native/csrc/type_shim.h code style (#1260) 2022-07-13 12:08:21 +08:00
superhao1995 f660152c73 [NFC] polish colossalai/nn/layer/parallel_3d/_operation.py code style (#1258) 
Co-authored-by: Research <research@soccf-snr3-017.comp.nus.edu.sg>
 2022-07-13 12:08:21 +08:00
Thunderbeee 9738fb0f78 [NFC] polish colossalai/nn/lr_scheduler/__init__.py (#1255) 
code style
 2022-07-13 12:08:21 +08:00
Kai Wang (Victor Kai) 50f2ad213f [NFC] polish colossalai/engine/ophooks/utils.py code style (#1256) 2022-07-13 12:08:21 +08:00
Ofey Chan 2dd4d556fb
[NFC] polish colossalai/nn/init.py code style (#1292) 2022-07-13 10:51:55 +08:00
Jiarui Fang 556b9b7e1a
[hotfix] Dist Mgr gather torch version (#1284) 
* make it faster

* [hotfix] torchvison fx tests

* [hotfix] rename duplicated named test_gpt.py

* [hotfix] dist mgr torch version
 2022-07-13 00:18:56 +08:00
HELSON abba4d84e1
[hotfix] fix bert model test in unitests (#1272) 2022-07-12 23:26:45 +08:00
ver217 7aadcbd070
hotfix colotensor _scan_for_pg_from_args (#1276) 2022-07-12 20:46:31 +08:00
oahzxl 0cf8e8e91c
[NFC] polish <colossalai/nn/lr_scheduler/poly.py> code style (#1267) 2022-07-12 18:18:14 +08:00
Jiarui Fang c92f84fcdb
[tensor] distributed checkpointing for parameters (#1240) 2022-07-12 15:51:06 +08:00
Frank Lee fb35460595
[fx] added ndim property to proxy (#1253) 2022-07-12 15:27:13 +08:00
Frank Lee 4a09fc0947
[fx] fixed tracing with apex-based T5 model (#1252) 
* [fx] fixed tracing with apex-based T5 model

* polish code

* polish code
 2022-07-12 15:19:25 +08:00
Frank Lee 7531c6271f
[fx] refactored the file structure of patched function and module (#1238) 
* [fx] refactored the file structure of patched function and module

* polish code
 2022-07-12 15:01:58 +08:00
YuliangLiu0306 17ed33350b
[hotfix] fix an assertion bug in base schedule. (#1250) 2022-07-12 14:20:02 +08:00
YuliangLiu0306 97d713855a
[fx] methods to get fx graph property. (#1246) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* manipulation

* [fx]add graph manipulation methods.

* [fx]methods to get fx graph property.

* add unit test

* add docstring to explain top node and leaf node in this context
 2022-07-12 14:10:37 +08:00
YuliangLiu0306 30b4fc0eb0
[fx]add split module pass and unit test from pipeline passes (#1242) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [fx]add split module pass and unit test from pipeline passes

* fix MNASNet bug

* polish
 2022-07-12 13:45:01 +08:00
Jiarui Fang 1aad903c15
[tensor] redistribute among different process groups (#1247) 
* make it faster

* [tensor] rename convert_to_dist -> redistribute

* [tensor] ShardSpec and ReplicaSpec

* [tensor] redistribute among diff pgs

* polish code
 2022-07-12 10:24:05 +08:00
Jiarui Fang 9bcd2fd4af
[tensor] a shorter shard and replicate spec (#1245) 2022-07-11 15:51:48 +08:00
Jiarui Fang 2699dfbbfd
[rename] convert_to_dist -> redistribute (#1243) 2022-07-11 13:05:44 +08:00
HELSON f6add9b720
[tensor] redirect .data.__get__ to a tensor instance (#1239) 2022-07-11 11:41:29 +08:00
Jiarui Fang 20da6e48c8
[checkpoint] save sharded optimizer states (#1237) 2022-07-08 16:33:13 +08:00
Jiarui Fang 4a76084dc9
[tensor] add zero_like colo op, important for Optimizer (#1236) 2022-07-08 14:55:27 +08:00
Jiarui Fang 3b500984b1
[tensor] fix some unittests (#1234) 2022-07-08 14:18:30 +08:00
ver217 a45ddf2d5f
[hotfix] fix sharded optim step and clip_grad_norm (#1226) 2022-07-08 13:34:48 +08:00
HELSON f071b500b6
[polish] polish __repr__ for ColoTensor, DistSpec, ProcessGroup (#1235) 2022-07-08 13:25:57 +08:00
HELSON 0453776def
[tensor] fix a assertion in colo_tensor cross_entropy (#1232) 2022-07-08 11:18:00 +08:00
Jiarui Fang 0e199d71e8
[hotfix] fx get comm size bugs (#1233) 
* init a checkpoint dir

* [checkpoint]support resume for cosinewarmuplr

* [checkpoint]add unit test

* fix some bugs but still not OK

* fix bugs

* make it faster

* [checkpoint]support generalized scheduler

* polish

* [tensor] torch function return colotensor

* polish

* fix bugs

* remove debug info

* polish

* polish

* [tensor] test_model pass unittests

* polish

* [hotfix] fx get comm size bug

Co-authored-by: ZhaoYi1222 <zhaoyi9499@gmail.com>
 2022-07-08 10:54:41 +08:00
HELSON 42ab36b762
[tensor] add unitest for colo_tensor 1DTP cross_entropy (#1230) 2022-07-07 19:17:23 +08:00
Yi Zhao 04537bf83e
[checkpoint]support generalized scheduler (#1222) 2022-07-07 18:16:38 +08:00
Jiarui Fang a98319f023
[tensor] torch function return colotensor (#1229) 2022-07-07 18:09:18 +08:00
YuliangLiu0306 2b7dca44b5
[fx]get communication size between partitions (#1224) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [fx]get communication size between partitions.

* polish
 2022-07-07 16:22:00 +08:00
Frank Lee 84f2298a96
[fx] added patches for tracing swin transformer (#1228) 2022-07-07 15:20:13 +08:00
Frank Lee b6cb5a47ad
[fx] added timm model tracing testing (#1221) 2022-07-07 14:02:17 +08:00
HELSON 280a81243d
[tensor] improve robustness of class 'ProcessGroup' (#1223) 2022-07-07 13:55:24 +08:00
Jiarui Fang 15d988f954
[tensor] sharded global process group (#1219) 2022-07-07 13:38:48 +08:00
Jiarui Fang db1bef9032
[hotfix] fx shard 1d pass bug fixing (#1220) 2022-07-07 13:37:31 +08:00
Frank Lee 11973d892d
[fx] added torchvision model tracing testing (#1216) 
* [fx] added torchvision model tracing testing

* remove unused imports
 2022-07-06 21:37:56 +08:00
Jiarui Fang 52736205d9
[checkpoint] make unitest faster (#1217) 2022-07-06 17:39:46 +08:00
Jiarui Fang f38006ea83
[checkpoint] checkpoint for ColoTensor Model (#1196) 2022-07-06 17:22:03 +08:00
XYE 291e22aac6
[fx] temporarily used (#1215) 2022-07-06 17:19:26 +08:00
Jiarui Fang ae7d3f4927
[refactor] move process group from _DistSpec to ColoTensor. (#1203) 2022-07-06 16:15:16 +08:00
Frank Lee 5da87ce35d
[fx] added testing for all albert variants (#1211) 2022-07-06 15:11:08 +08:00
Frank Lee 2d13a45a3b
[fx] added testing for all gpt variants (#1210) 
* [fx] added testing for all gpt variants

* polish code

* polish code
 2022-07-06 14:03:13 +08:00
YuliangLiu0306 189946c5c4
[fx]add uniform policy (#1208) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [fx]add uniform policy
 2022-07-06 13:48:11 +08:00
Frank Lee 426a279ce7
[fx] added testing for all bert variants (#1207) 
* [fx] added testing for all bert variants

* polish code
 2022-07-06 10:50:49 +08:00
Jiarui Fang b5f25eb32a
[Tensor] add cpu group to ddp (#1200) 2022-07-05 14:58:28 +08:00
Frank Lee f7878f465c
[fx] supported model tracing for huggingface bert (#1201) 
* [fx] supported model tracing for huggingface bert

* polish test
 2022-07-05 13:19:57 +08:00
Jiarui Fang 060b917daf
[refactor] remove gpc dependency in colotensor's _ops (#1189) 2022-07-04 18:54:37 +08:00
Frank Lee abf6a262dc
[fx] added module patch for pooling layers (#1197) 2022-07-04 15:21:26 +08:00
YuliangLiu0306 63d2a93878
[context]support arbitary module materialization. (#1193) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [context]support arbitary module materialization.

* [test]add numerical check for lazy init context.
 2022-07-04 10:12:02 +08:00
Jiarui Fang a444633d13
warmup ratio configration (#1192) 2022-06-30 15:23:50 +08:00
ver217 dba7e0cfb4
make AutoPlacementPolicy configurable (#1191) 2022-06-30 15:18:30 +08:00
YuliangLiu0306 2053e138a2
[context]use meta tensor to init model lazily. (#1187) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [context]use meta tensor to init model lazily.

* polish

* make module with device kwargs bypass the normal init.

* change unit test to adapt updated context.
 2022-06-29 21:02:30 +08:00
Frank Lee 2c8c05675d
[fx] patched conv and normalization (#1188) 2022-06-29 18:58:38 +08:00
Frank Lee 6d86f1bc91
[fx] supported data-dependent control flow in model tracing (#1185) 
* [fx] supported data-dependent control flow in model tracing

* polish code
 2022-06-29 15:05:25 +08:00
Jiarui Fang c463f8adf9
[tensor] remove gpc in tensor tests (#1186) 2022-06-29 14:08:40 +08:00
Jiarui Fang 372f791444
[refactor] move chunk and chunkmgr to directory gemini (#1182) 2022-06-29 13:31:02 +08:00
ver217 6b2f2ab9bb
[ddp] ColoDDP uses bucket all-reduce (#1177) 
* add reducer

* update colo ddp with reducer

* polish unit test

* polish unit test
 2022-06-29 10:34:13 +08:00
Jiarui Fang 7487215b95
[ColoTensor] add independent process group (#1179) 2022-06-29 10:03:09 +08:00
YuliangLiu0306 26ba87272d
[hotfix]fixed p2p process send stuck (#1181) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [hotfix]fixed p2p process send stuck
 2022-06-28 14:41:11 +08:00
Jiarui Fang 1b657f9ce1
[tensor] revert local view back (#1178) 2022-06-27 18:38:34 +08:00
Jiarui Fang 0dd4e2bbfb
[Tensor] rename some APIs in TensorSpec and Polish view unittest (#1176) 2022-06-27 15:56:11 +08:00
Ziyue Jiang dd0420909f
[Tensor] rename parallel_action (#1174) 
* rename parallel_action

* polish
 2022-06-27 10:04:45 +08:00
YuliangLiu0306 e27645376d
[hotfix]different overflow status lead to communication stuck. (#1175) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [hotfix]fix some bugs caused by refactored schedule.

* [hotfix]different overflow statu llead to communication stuck.
 2022-06-27 09:53:57 +08:00
Jiarui Fang aa7bef73d4
[Tensor] distributed view supports inter-process hybrid parallel (#1169) 2022-06-27 09:45:26 +08:00