github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
1,033 Commits
32 Branches
46 Tags
228 MiB
Commit Graph

408 Commits (ae71036cd2210b6e60805357f4bd059674e316bc)

Author SHA1 Message Date
HELSON 039b7ed3bc
[polish] add update directory in gemini; rename AgChunk to ChunkV2 (#1432) 2022-08-10 16:40:29 +08:00
Super Daniel f20cb4e893
[fx] modify the calculation of node_size in MetaInfoProp for activation checkpointing usages (#1425) 
* [fx] modify the calculation of node_size in MetaInfoProp for activation checkpointing usages

* [fx] modify the calculation of node_size in MetaInfoProp for activation checkpointing usages

* [fx] modify the calculation of node_size in MetaInfoProp for activation checkpointing usages
 2022-08-10 16:36:35 +08:00
Jiarui Fang 89c434a0a6
[polish] add test_ops directory (#1431) 2022-08-10 15:35:26 +08:00
Jiarui Fang 10b3df65c8
[FAW] move coloparam setting in test code. (#1429) 2022-08-10 14:31:53 +08:00
Jiarui Fang cb98cf5558
[FAW] parallel FreqAwareEmbedding (#1424) 2022-08-10 13:44:30 +08:00
HELSON 0d212183c4
[zero] add has_inf_or_nan in AgChunk; enhance the unit test of AgChunk (#1426) 2022-08-10 11:37:28 +08:00
YuliangLiu0306 33f0744d51
[tensor] add shape consistency feature to support auto spec transform (#1418) 
* [tensor] add shape consistency feature to supportauto sharding spec transform.

* [tensor] remove unused argument in simulator, add doc string for target pair.
 2022-08-10 11:29:17 +08:00
HELSON 4fb3c52cf0
[zero] add unit test for AgChunk's append, close, access (#1423) 2022-08-09 18:03:10 +08:00
Jiarui Fang d209aff684
Add FreqAwareEmbeddingBag (#1421) 2022-08-09 16:26:12 +08:00
Jiarui Fang 504419d261
[FAW] add cache manager for the cached embedding (#1419) 2022-08-09 15:17:17 +08:00
Kirigaya Kazuto 44fd3c83ab
[communication] add p2p_v2.py to support communication with List[Any] (#1407) 
* support p2p communication with any type of object | pass test

* reconstruct pipeline schedule with p2p_v2.py(support communication with List[Any]) | pass test

* [communication] add p2p_v2.py to support communication with List[Any]

* Delete _pipeline_schedule_v2.py

* Delete test_cifar_with_data_pipeline_tensor_v2.py

* [engin/schedule] use p2p_v2 to recontruct pipeline_schedule

* [communication] remove print code

* [communication] remove print code
 2022-08-09 11:40:04 +08:00
YuliangLiu0306 7c96055c68
[tensor]build sharding spec to replace distspec in future. (#1405) 2022-08-08 11:15:57 +08:00
ver217 12b4887097
[hotfix] fix CPUAdam kernel nullptr (#1410) 2022-08-05 19:45:45 +08:00
YuliangLiu0306 0442f940f0
[device] add DeviceMesh class to support logical device layout (#1394) 
* [device] add DeviceMesh class to support logical device layout

* polish code

* add doc string
 2022-08-02 19:23:48 +08:00
HELSON 4e98e938ce
[zero] alleviate memory usage in ZeRODDP state_dict (#1398) 2022-08-02 15:49:13 +08:00
Frank Lee adf5054ff8
[fx] fixed torchaudio conformer tracing (#1392) 2022-08-01 16:08:28 +08:00
Frank Lee 7d6293927f
[fx] patched torch.max and data movement operator (#1391) 
* [fx] patched torch.max and data movement operator

* polish code
 2022-08-01 15:31:50 +08:00
HELSON 527758b2ae
[hotfix] fix a running error in test_colo_checkpoint.py (#1387) 2022-07-29 15:58:06 +08:00
ver217 8dced41ad0
[zero] zero optim state_dict takes only_rank_0 (#1384) 
* zero optim state_dict takes only_rank_0

* fix unit test
 2022-07-29 13:22:50 +08:00
ver217 7d5d628e07
[DDP] test ddp state dict uses more strict threshold (#1382) 2022-07-28 17:29:04 +08:00
ver217 828b9e5e0d
[hotfix] fix zero optim save/load state dict (#1381) 2022-07-28 17:19:39 +08:00
Super Daniel be229217ce
[fx] add torchaudio test (#1369) 
* [fx]add torchaudio test

* [fx]add torchaudio test

* [fx] add torchaudio test

* [fx] add torchaudio test

* [fx] add torchaudio test

* [fx] add torchaudio test

* [fx] add torchaudio test

* [fx] add torchaudio test and test patches

* Delete ~

* [fx] add patches and patches test

* [fx] add patches and patches test

* [fx] fix patches

* [fx] fix rnn patches

* [fx] fix rnn patches

* [fx] fix rnn patches

* [fx] fix rnn patches

* [fx] merge upstream

* [fx] fix import errors
 2022-07-27 11:03:14 +08:00
Boyuan Yao bb640ec728
[fx] Add colotracer compatibility test on torchrec (#1370) 2022-07-26 17:54:39 +08:00
ver217 c415240db6
[nvme] CPUAdam and HybridAdam support NVMe offload (#1360) 
* impl nvme optimizer

* update cpu adam

* add unit test

* update hybrid adam

* update docstr

* add TODOs

* update CI

* fix CI

* fix CI

* fix CI path

* fix CI path

* fix CI path

* fix install tensornvme

* fix CI

* fix CI path

* fix CI env variables

* test CI

* test CI

* fix CI

* fix nvme optim __del__

* fix adam __del__

* fix nvme optim

* fix CI env variables

* fix nvme optim import

* test CI

* test CI

* fix CI
 2022-07-26 17:25:24 +08:00
HELSON 87775a0682
[colotensor] use cpu memory to store state_dict (#1367) 2022-07-26 14:13:38 +08:00
Frank Lee cd063ac37f
[fx] added activation checkpoint codegen support for torch < 1.12 (#1359) 2022-07-25 23:35:31 +08:00
HELSON 4417804129
[unit test] add megatron init test in zero_optim (#1358) 2022-07-25 11:18:08 +08:00
HELSON 7a065dc9f6
[hotfix] fix megatron_init in test_gpt2.py (#1357) 2022-07-25 10:28:19 +08:00
Frank Lee 644582eee9
[fx] added activation checkpoint codegen (#1355) 2022-07-25 09:39:10 +08:00
Frank Lee 05fae1fd56
[fx] added activation checkpointing annotation (#1349) 
* [fx] added activation checkpointing annotation

* polish code

* polish code
 2022-07-21 11:14:28 +08:00
HELSON 7a8702c06d
[colotensor] add Tensor.view op and its unit test (#1343) 
[colotensor] add megatron initialization for gpt2
 2022-07-21 10:53:15 +08:00
YuliangLiu0306 942c8cd1fb
[fx] refactor tracer to trace complete graph (#1342) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [fx] refactor tracer to trace complete graph

* add comments and solve conflicts.
 2022-07-20 11:20:38 +08:00
Frank Lee 2cc1175c76
[fx] tested the complete workflow for auto-parallel (#1336) 
* [fx] tested the complete workflow for auto-parallel

* polish code

* polish code

* polish code
 2022-07-20 10:45:17 +08:00
YuliangLiu0306 4631fef8a0
[fx]refactor tracer (#1335) 2022-07-19 15:50:42 +08:00
HELSON bf5066fba7
[refactor] refactor ColoTensor's unit tests (#1340) 2022-07-19 15:46:24 +08:00
HELSON f92c100ddd
[checkpoint] use gather_tensor in checkpoint and update its unit test (#1339) 2022-07-19 14:15:28 +08:00
Frank Lee f3ce7b8336
[fx] recovered skipped pipeline tests (#1338) 2022-07-19 09:49:50 +08:00
ver217 0c51ff2c13
[hotfix] ZeroDDP use new process group (#1333) 
* process group supports getting ranks in group

* chunk mgr receives a process group

* update unit test

* fix unit tests
 2022-07-18 14:14:52 +08:00
Frank Lee 75abc75c15
[fx] fixed compatiblity issue with torch 1.10 (#1331) 2022-07-18 11:41:27 +08:00
Frank Lee 169954f87e
[test] removed outdated unit test for meta context (#1329) 2022-07-15 23:16:23 +08:00
ver217 7a05367101
[hotfix] shared model returns cpu state_dict (#1328) 2022-07-15 22:11:37 +08:00
Frank Lee b2475d8c5c
[fx] fixed unit tests for torch 1.12 (#1327) 2022-07-15 18:22:15 +08:00
HELSON d49708ae43
[hotfix] fix ddp for unit test test_gpt2 (#1326) 2022-07-15 18:19:52 +08:00
Frank Lee 250be4d31e
[utils] integrated colotensor with lazy init context (#1324) 
* [utils] integrated colotensor with lazy init context

* polish code

* polish code

* polish code
 2022-07-15 17:47:12 +08:00
YuliangLiu0306 e8acf55e8b
[fx] add balanced policy v2 (#1251) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [fx] add balanced policy v2

* add unittest
 2022-07-15 14:54:26 +08:00
XYE ca2d3f284f
[fx] Add unit test and fix bugs for transform_mlp_pass (#1299) 
* add test and fix bugs

* add functions back

* add comments
 2022-07-15 14:37:58 +08:00
HELSON 1b41686461
[hotfix] fix unit test test_module_spec (#1321) 2022-07-15 14:02:32 +08:00
Jiarui Fang 9e4c6449b0
[checkpoint] add ColoOptimizer checkpointing (#1316) 2022-07-15 09:52:55 +08:00
Jiarui Fang 85f933b58b
[Optimizer] Remove useless ColoOptimizer (#1312) 2022-07-14 16:57:48 +08:00
Jiarui Fang 9f10524313
[Optimizer] polish the init method of ColoOptimizer (#1310) 2022-07-14 16:37:33 +08:00
HELSON 36086927e1
[hotfix] fix ColoTensor GPT2 unitest (#1309) 2022-07-14 16:37:20 +08:00
Jiarui Fang 3ef3791a3b
[checkpoint] add test for bert and hotfix save bugs (#1297) 2022-07-14 15:38:18 +08:00
Jiarui Fang bd71e2a88b
[hotfix] add missing file (#1308) 2022-07-14 14:43:15 +08:00
Frank Lee 4f4d8c3656
[fx] added apex normalization to patched modules (#1300) 
* [fx] added apex normalization to patched modules

* remove unused imports
 2022-07-14 14:24:13 +08:00
Jiarui Fang 4165eabb1e
[hotfix] remove potiential circle import (#1307) 
* make it faster

* [hotfix] remove circle import
 2022-07-14 13:44:26 +08:00
YuliangLiu0306 93a75433df
[hotfix] skip some unittest due to CI environment. (#1301) 2022-07-14 10:55:18 +08:00
HELSON 260a55804a
[hotfix] fix shape error in backward when using ColoTensor (#1298) 2022-07-13 23:06:12 +08:00
Frank Lee 7e8114a8dd
[hotfix] skipped unsafe test cases (#1282) 2022-07-13 00:08:59 +08:00
Jiarui Fang 79fe7b027a
[hotfix] test model unittest hotfix (#1281) 2022-07-12 23:45:29 +08:00
Jiarui Fang e56731e916 [hotfix] test_gpt.py duplicated (#1279) 
* make it faster

* [hotfix] torchvison fx tests

* [hotfix] rename duplicated named test_gpt.py
 2022-07-12 23:29:17 +08:00
HELSON abba4d84e1
[hotfix] fix bert model test in unitests (#1272) 2022-07-12 23:26:45 +08:00
YuliangLiu0306 01ea68b2e6
[tests] remove T5 test skip decorator (#1271) 2022-07-12 23:25:30 +08:00
Jiarui Fang ca9d5ee91c
[hotfix] torchvison fx unittests miss import pytest (#1277) 2022-07-12 23:04:06 +08:00
Jiarui Fang c92f84fcdb
[tensor] distributed checkpointing for parameters (#1240) 2022-07-12 15:51:06 +08:00
Frank Lee 4a09fc0947
[fx] fixed tracing with apex-based T5 model (#1252) 
* [fx] fixed tracing with apex-based T5 model

* polish code

* polish code
 2022-07-12 15:19:25 +08:00
YuliangLiu0306 97d713855a
[fx] methods to get fx graph property. (#1246) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* manipulation

* [fx]add graph manipulation methods.

* [fx]methods to get fx graph property.

* add unit test

* add docstring to explain top node and leaf node in this context
 2022-07-12 14:10:37 +08:00
YuliangLiu0306 30b4fc0eb0
[fx]add split module pass and unit test from pipeline passes (#1242) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [fx]add split module pass and unit test from pipeline passes

* fix MNASNet bug

* polish
 2022-07-12 13:45:01 +08:00
Jiarui Fang 1aad903c15
[tensor] redistribute among different process groups (#1247) 
* make it faster

* [tensor] rename convert_to_dist -> redistribute

* [tensor] ShardSpec and ReplicaSpec

* [tensor] redistribute among diff pgs

* polish code
 2022-07-12 10:24:05 +08:00
Jiarui Fang 9bcd2fd4af
[tensor] a shorter shard and replicate spec (#1245) 2022-07-11 15:51:48 +08:00
Jiarui Fang 2699dfbbfd
[rename] convert_to_dist -> redistribute (#1243) 2022-07-11 13:05:44 +08:00
HELSON f6add9b720
[tensor] redirect .data.__get__ to a tensor instance (#1239) 2022-07-11 11:41:29 +08:00
Jiarui Fang 20da6e48c8
[checkpoint] save sharded optimizer states (#1237) 2022-07-08 16:33:13 +08:00
Jiarui Fang 4a76084dc9
[tensor] add zero_like colo op, important for Optimizer (#1236) 2022-07-08 14:55:27 +08:00
Jiarui Fang 3b500984b1
[tensor] fix some unittests (#1234) 2022-07-08 14:18:30 +08:00
HELSON 0453776def
[tensor] fix a assertion in colo_tensor cross_entropy (#1232) 2022-07-08 11:18:00 +08:00
Jiarui Fang 0e199d71e8
[hotfix] fx get comm size bugs (#1233) 
* init a checkpoint dir

* [checkpoint]support resume for cosinewarmuplr

* [checkpoint]add unit test

* fix some bugs but still not OK

* fix bugs

* make it faster

* [checkpoint]support generalized scheduler

* polish

* [tensor] torch function return colotensor

* polish

* fix bugs

* remove debug info

* polish

* polish

* [tensor] test_model pass unittests

* polish

* [hotfix] fx get comm size bug

Co-authored-by: ZhaoYi1222 <zhaoyi9499@gmail.com>
 2022-07-08 10:54:41 +08:00
HELSON 42ab36b762
[tensor] add unitest for colo_tensor 1DTP cross_entropy (#1230) 2022-07-07 19:17:23 +08:00
Yi Zhao 04537bf83e
[checkpoint]support generalized scheduler (#1222) 2022-07-07 18:16:38 +08:00
Jiarui Fang a98319f023
[tensor] torch function return colotensor (#1229) 2022-07-07 18:09:18 +08:00
Frank Lee 5581170890
[fx] fixed huggingface OPT and T5 results misalignment (#1227) 2022-07-07 16:29:58 +08:00
YuliangLiu0306 2b7dca44b5
[fx]get communication size between partitions (#1224) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [fx]get communication size between partitions.

* polish
 2022-07-07 16:22:00 +08:00
Frank Lee 84f2298a96
[fx] added patches for tracing swin transformer (#1228) 2022-07-07 15:20:13 +08:00
Frank Lee 37fcf96b7f
[fx] fixed timm tracing result misalignment (#1225) 2022-07-07 14:45:15 +08:00
Frank Lee b6cb5a47ad
[fx] added timm model tracing testing (#1221) 2022-07-07 14:02:17 +08:00
Jiarui Fang 15d988f954
[tensor] sharded global process group (#1219) 2022-07-07 13:38:48 +08:00
Frank Lee 11973d892d
[fx] added torchvision model tracing testing (#1216) 
* [fx] added torchvision model tracing testing

* remove unused imports
 2022-07-06 21:37:56 +08:00
Jiarui Fang 52736205d9
[checkpoint] make unitest faster (#1217) 2022-07-06 17:39:46 +08:00
Jiarui Fang f38006ea83
[checkpoint] checkpoint for ColoTensor Model (#1196) 2022-07-06 17:22:03 +08:00
Jiarui Fang ae7d3f4927
[refactor] move process group from _DistSpec to ColoTensor. (#1203) 2022-07-06 16:15:16 +08:00
Frank Lee 5da87ce35d
[fx] added testing for all albert variants (#1211) 2022-07-06 15:11:08 +08:00
Frank Lee 2d13a45a3b
[fx] added testing for all gpt variants (#1210) 
* [fx] added testing for all gpt variants

* polish code

* polish code
 2022-07-06 14:03:13 +08:00
YuliangLiu0306 189946c5c4
[fx]add uniform policy (#1208) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [fx]add uniform policy
 2022-07-06 13:48:11 +08:00
Frank Lee 426a279ce7
[fx] added testing for all bert variants (#1207) 
* [fx] added testing for all bert variants

* polish code
 2022-07-06 10:50:49 +08:00
Frank Lee f7878f465c
[fx] supported model tracing for huggingface bert (#1201) 
* [fx] supported model tracing for huggingface bert

* polish test
 2022-07-05 13:19:57 +08:00
Jiarui Fang 060b917daf
[refactor] remove gpc dependency in colotensor's _ops (#1189) 2022-07-04 18:54:37 +08:00
Frank Lee abf6a262dc
[fx] added module patch for pooling layers (#1197) 2022-07-04 15:21:26 +08:00
YuliangLiu0306 63d2a93878
[context]support arbitary module materialization. (#1193) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [context]support arbitary module materialization.

* [test]add numerical check for lazy init context.
 2022-07-04 10:12:02 +08:00
YuliangLiu0306 2053e138a2
[context]use meta tensor to init model lazily. (#1187) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [context]use meta tensor to init model lazily.

* polish

* make module with device kwargs bypass the normal init.

* change unit test to adapt updated context.
 2022-06-29 21:02:30 +08:00
Frank Lee 2c8c05675d
[fx] patched conv and normalization (#1188) 2022-06-29 18:58:38 +08:00
Frank Lee 6d86f1bc91
[fx] supported data-dependent control flow in model tracing (#1185) 
* [fx] supported data-dependent control flow in model tracing

* polish code
 2022-06-29 15:05:25 +08:00