github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
792 Commits
32 Branches
47 Tags
229 MiB
Commit Graph

287 Commits (20da6e48c8ebd7e17f68308b0bc45e977ff80471)

Author SHA1 Message Date
ver217 f0a954f16d
[ddp] add set_params_to_ignore for ColoDDP (#1122) 
* add set_params_to_ignore for ColoDDP

* polish code

* fix zero hook v2

* add unit test

* polish docstr
 2022-06-16 12:54:46 +08:00
YuliangLiu0306 fcf55777dd
[fx]add autoparallel passes (#1121) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* feature/add autoparallel passes
 2022-06-15 16:36:46 +08:00
Frank Lee 16302a5359
[fx] added unit test for coloproxy (#1119) 
* [fx] added unit test for coloproxy

* polish code

* polish code
 2022-06-15 15:27:51 +08:00
ver217 7d14b473f0
[gemini] gemini mgr supports "cpu" placement policy (#1118) 
* update gemini mgr

* update chunk

* add docstr

* polish placement policy

* update test chunk

* update test zero

* polish unit test

* remove useless unit test
 2022-06-15 15:05:19 +08:00
Frank Lee 53297330c0
[test] fixed hybrid parallel test case on 8 GPUs (#1106) 2022-06-14 10:30:54 +08:00
ver217 1f894e033f
[gemini] zero supports gemini (#1093) 
* add placement policy

* add gemini mgr

* update mem stats collector

* update zero

* update zero optim

* fix bugs

* zero optim monitor os

* polish unit test

* polish unit test

* add assert
 2022-06-10 14:48:28 +08:00
Frank Lee 2b2dc1c86b
[pipeline] refactor the pipeline module (#1087) 
* [pipeline] refactor the pipeline module

* polish code
 2022-06-10 11:27:38 +08:00
Frank Lee bad5d4c0a1
[context] support lazy init of module (#1088) 
* [context] support lazy init of module

* polish code
 2022-06-10 10:09:48 +08:00
ver217 be01db37c8
[tensor] refactor chunk mgr and impl MemStatsCollectorV2 (#1077) 
* polish chunk manager

* polish unit test

* impl add_extern_static_tensor for chunk mgr

* add mem stats collector v2

* polish code

* polish unit test

* polish code

* polish get chunks
 2022-06-09 20:56:34 +08:00
Ziyue Jiang b3a03e4bfd
[Tensor] fix equal assert (#1091) 
* fix equal assert

* polish
 2022-06-09 17:36:15 +08:00
Frank Lee 50ec3a7e06
[test] skip tests when not enough GPUs are detected (#1090) 
* [test] skip tests when not enough GPUs are detected

* polish code

* polish code
 2022-06-09 17:19:13 +08:00
Frank Lee 65ee6dcc20
[test] ignore 8 gpu test (#1080) 
* [test] ignore 8 gpu test

* polish code

* polish workflow

* polish workflow
 2022-06-08 23:14:18 +08:00
Ziyue Jiang 0653c63eaa
[Tensor] 1d row embedding (#1075) 
* Add CPU 1d row embedding

* polish
 2022-06-08 12:04:59 +08:00
ver217 1b17859328
[tensor] chunk manager monitor mem usage (#1076) 2022-06-07 15:00:00 +08:00
Ziyue Jiang 4fc748f69b
[Tensor] fix optimizer for CPU parallel (#1069) 2022-06-06 17:36:11 +08:00
Jiarui Fang 49832b2344
[refactory] add nn.parallel module (#1068) 2022-06-06 15:34:41 +08:00
Jiarui Fang a00644079e
reorgnize colotensor directory (#1062) 
* reorgnize colotensor directory

* polish code
 2022-06-03 18:04:22 +08:00
Ziyue Jiang df9dcbbff6
[Tensor] add hybrid device demo and fix bugs (#1059) 2022-06-03 12:09:49 +08:00
YuliangLiu0306 b167258b6a
[pipeline]refactor ppschedule to support tensor list (#1050) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* refactor ppschedule to support tensor list

* polish
 2022-06-02 13:48:59 +08:00
ver217 51b9a49655
[zero] add zero optimizer for ColoTensor (#1046) 
* add zero optimizer

* torch ok

* unit test ok

* polish code

* fix bugs

* polish unit test

* polish zero optim

* polish colo ddp v2

* refactor folder structure

* add comment

* polish unit test

* polish zero optim

* polish unit test
 2022-06-02 12:13:15 +08:00
ver217 7faef93326
fix dist spec mgr (#1045) 2022-05-31 12:14:39 +08:00
ver217 9492a561c3
[tensor] ColoTensor supports ZeRo (#1015) 
* impl chunk manager

* impl param op hook

* add reduce_chunk

* add zero hook v2

* add zero dp

* fix TensorInfo

* impl load balancing when using zero without chunk

* fix zero hook

* polish chunk

* fix bugs

* ddp ok

* zero ok

* polish code

* fix bugs about load balancing

* polish code

* polish code

* add ene-to-end test

* polish code

* polish code

* polish code

* fix typo

* add test_chunk

* fix bugs

* fix bugs

* polish code
 2022-05-31 12:00:12 +08:00
YuliangLiu0306 9feff0f760
[titans]remove model zoo (#1042) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* rm model zoo
 2022-05-31 10:40:47 +08:00
Ziyue Jiang 7c530b9de2
[Tensor] add Parameter inheritance for ColoParameter (#1041) 
* add Parameter inheritance for ColoParameter

* remove tricks

* remove tricks

* polish

* polish
 2022-05-30 17:23:44 +08:00
Ziyue Jiang 6c5996a56e
[Tensor] add module check and bert test (#1031) 
* add Embedding

* Add bert test

* polish

* add check module test

* polish

* polish

* polish

* polish
 2022-05-26 18:15:42 +08:00
YuliangLiu0306 7106bd671d
[p2p]add object list send/recv (#1024) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [p2p]add object list send recv

* refactor for code reusability

* polish
 2022-05-26 14:28:46 +08:00
Ziyue Jiang 32291dd73f
[Tensor] add module handler for linear (#1021) 
* add module spec for linear

* polish

* polish

* polish
 2022-05-26 11:50:44 +08:00
ver217 cefc29ff06
[tensor] impl ColoDDP for ColoTensor (#1009) 
* impl ColoDDP for ColoTensor

* polish code
 2022-05-21 13:52:04 +08:00
ver217 a3b66f6def
[tensor] refactor parallel action (#1007) 
* refactor parallel action

* polish unit tests
 2022-05-20 20:19:58 +08:00
ver217 8e3d0ad8f1
[unit test] refactor test tensor (#1005) 
* polish test_gpt

* update op unit tests

* update test model
 2022-05-19 18:57:56 +08:00
ver217 ad536e308e
[tensor] refactor colo-tensor (#992) 
* refactor colo-tensor and update linear op

* polish code

* polish code

* update ops and unit tests

* update unit tests

* polish code

* rename dist_spec module

* polish code

* polish code

* remove unneeded import

* fix pipelinable
 2022-05-19 12:44:59 +08:00
ver217 c2fdc6a011
[tensor] derive compute pattern from dist spec (#971) 
* derive compute pattern from dist spec

* polish code
 2022-05-16 14:58:08 +08:00
Ziyue Jiang 797a9dc5a9
add DistSpec for loss and test_model (#947) 2022-05-13 20:29:50 +08:00
ver217 67c33f57eb
[tensor] design DistSpec and DistSpecManager for ColoTensor (#934) 
* add dist spec

* update linear op

* polish code

* polish code

* update embedding op

* polish unit tests

* polish unit tests

* polish comments

* polish code

* add test_dist_spec_mgr

* polish code

* refactor folder structure

* polish unit tests

* add get_process_group() for TensorSpec

* polish code
 2022-05-13 15:13:52 +08:00
Ziyue Jiang 830d3bca26
[Tensor] add optimizer to bert test (#933) 
* add optimizer to bert test

* polish
 2022-05-13 11:37:23 +08:00
Ziyue Jiang d73c2b1d79
[Tensor] fix init context (#931) 
* change torch.Parameter to ColoParameter

* fix post assignment for init context

* polish

* polish
 2022-05-11 15:48:12 +08:00
Ziyue Jiang dfc88b85ea
[Tensor] simplify named param (#928) 
* simplify ColoModulize

* simplify ColoModulize

* polish

* polish
 2022-05-11 10:54:19 +08:00
ver217 45b9124df4
[tensor] hijack addmm for colo tensor (#923) 
* hijack addmm for colo tensor

* fix bugs

* polish unit test

* polish comments
 2022-05-09 18:55:49 +08:00
Jiarui Fang 534afb018a
test pretrain loading on multi-process (#922) 2022-05-09 17:07:35 +08:00
Ziyue Jiang c195d2814c
[Tensor] add from_pretrained support and bert pretrained test (#921) 
* add from_pretrained support and test

* polish

* polish

* polish

* polish
 2022-05-09 16:11:47 +08:00
Jiarui Fang 845856ea29
[Graph] building computing graph with ColoTensor, Linear only (#917) 2022-05-07 17:10:37 +08:00
Ziyue Jiang 75d221918a
[Tensor] add 1d vocab loss (#918) 
* add 1d vocab loss

* polish
 2022-05-07 15:49:14 +08:00
Ziyue Jiang dfaff4e243
[Tensor] fix test_model (#916) 
* polish test_model

* polish
 2022-05-06 18:06:22 +08:00
Jiarui Fang ed6426c300
[Tensor] polish model test (#915) 2022-05-06 17:07:56 +08:00
Ziyue Jiang 0fab86b12a
[Tensor] add a basic bert. (#911) 
* add base bert test

* Add bert test

* polish

* remove test_bert

* polish
 2022-05-06 15:03:43 +08:00
Jiarui Fang ab95ec9aea
[Tensor] init ColoParameter (#914) 2022-05-06 12:57:14 +08:00
Ziyue Jiang 193d629311
update pytest.mark.parametrize in tensor tests (#913) 2022-05-06 11:16:40 +08:00
Ziyue Jiang f593a5637e
[Tensor] add embedding tp1d row (#904) 2022-04-29 14:10:05 +08:00
Ziyue Jiang 2c0d19d755
[Tensor] add ColoTensor TP1Dcol Embedding (#899) 2022-04-28 17:45:06 +08:00
Jiarui Fang d16671da75
[Tensor] initialize the ColoOptimizer (#898) 
* [Tensor] activation is an attr of ColoTensor

* [Tensor] add optimizer

* only detach parameters in context

* polish code
 2022-04-28 15:23:40 +08:00
Jiarui Fang e76f76c08b
[Tensor] test parameters() as member function (#896) 2022-04-28 10:57:14 +08:00
Ziyue Jiang cb182da7c5
[tensor] refine linear and add gather for laynorm (#893) 
* refine linear and add function to ColoTensor

* add gather for layernorm

* polish

* polish
 2022-04-28 10:55:40 +08:00
Jiarui Fang 26c49639d8
[Tensor] overriding paramters() for Module using ColoTensor (#889) 2022-04-27 15:28:59 +08:00
Ziyue Jiang 1d0aba4153
[tensor] add ColoTensor 1Dcol (#888) 2022-04-27 14:13:55 +08:00
Jiarui Fang a0e5971692
[Tensor] test model check results for a simple net (#887) 2022-04-27 12:00:18 +08:00
Jiarui Fang 72cdc06875
[Tensor] make ColoTensor more robust for getattr (#886) 
* [Tensor] make ColoTensor more robust for getattr

* polish

* polish
 2022-04-27 10:57:49 +08:00
Ziyue Jiang 9bc5a77c31
[tensor] wrap function in the torch_tensor to ColoTensor (#881) 2022-04-26 20:13:56 +08:00
Jiarui Fang 7f76517a85
[Tensor] make a simple net works with 1D row TP (#879) 2022-04-26 18:11:47 +08:00
ver217 c4d903e64a
[gemini] accelerate adjust_layout() (#878) 
* add lru cache

* polish code

* update unit test

* fix sharded optim
 2022-04-26 18:08:31 +08:00
Jiarui Fang 909211453b
[Tensor] Add some attributes to ColoTensor (#877) 
* [Tensor] add some function to ColoTensor

* torch.allclose

* rm torch.add
 2022-04-26 15:10:47 +08:00
Jiarui Fang e43f83aa5c
[Tensor] get named parameters for model using ColoTensors (#874) 2022-04-26 14:08:01 +08:00
Jiarui Fang 96211c2cc8
[tensor] customized op returns ColoTensor (#875) 
* [tensor] customized op returns ColoTensor

* polish

* polish code
 2022-04-26 13:23:59 +08:00
Ziyue Jiang 26d4ab8b03
[Tensor] Add function to spec and update linear 1Drow and unit tests (#869) 2022-04-26 10:15:26 +08:00
Jiarui Fang 1190b2c4a4
[tensor] add cross_entrophy_loss (#868) 2022-04-25 16:01:52 +08:00
HELSON 3107817172
[gemini] add stateful tensor container (#867) 2022-04-25 14:58:16 +08:00
Jiarui Fang d01d3b8cb0
colo init context add device attr. (#866) 2022-04-25 14:24:26 +08:00
Jiarui Fang 126ba573a8
[Tensor] add layer norm Op (#852) 2022-04-25 11:49:20 +08:00
Frank Lee 1258af71cc
[ci] cache cuda extension (#860) 2022-04-25 10:03:47 +08:00
Ziyue Jiang bcc8655021
[Tensor ] Add 1Drow weight reshard by spec (#854) 2022-04-24 18:30:20 +08:00
Jiarui Fang 62f059251b
[Tensor] init a tp network training unittest (#849) 2022-04-24 16:43:44 +08:00
Ziyue Jiang 2a0a427e04
[tensor]add assert for colo_tensor 1Drow (#846) 2022-04-24 14:12:45 +08:00
Ziyue Jiang 05023ecfee
[Tensor] TP Linear 1D row (#843) 2022-04-24 13:43:12 +08:00
HELSON e5ea3fdeef
[gemini] add GeminiMemoryManger (#832) 
* refactor StatefulTensor, tensor utilities

* add unitest for GeminiMemoryManager
 2022-04-24 13:08:48 +08:00
YuliangLiu0306 35ea6e1023
[pipelinable]use pipelinable context to initialize non-pipeline model (#816) 
* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4.

* [pipeline]add module lazy init feature to support large model initization.

* [pipeline]add to_layer_list and partition method to support arbitrary non-pp model

* refactor the module structure

* polish

* [pipelinable]add unit test for pipelinable

* polish

* polish

* Fix CodeFactor issues.
 2022-04-24 13:03:12 +08:00
Jiarui Fang ea0a2ed25f
[hotfix] the bug of numel() in ColoTensor (#845) 2022-04-24 12:32:10 +08:00
Jiarui Fang 8789850eea
Init Conext supports lazy allocate model memory (#842) 2022-04-22 18:03:35 +08:00
Frank Lee 943982d29a
[unittest] refactored unit tests for change in dependency (#838) 2022-04-22 15:39:07 +08:00
Frank Lee 01e9f834f5
[dependency] removed torchvision (#833) 
* [dependency] removed torchvision

* fixed transforms
 2022-04-22 15:24:35 +08:00
Jiarui Fang cb5a4778e1
Revert "[WIP] Applying ColoTensor on TP-1D-row Linear. (#831)" (#835) 
This reverts commit ac88de6dfc.
 2022-04-22 14:45:57 +08:00
Jiarui Fang ac88de6dfc
[WIP] Applying ColoTensor on TP-1D-row Linear. (#831) 
* revert zero tensors back

* [tensor] init row 1d linear
 2022-04-22 14:03:26 +08:00
Jiarui Fang 294a6060d0
[tensor] ZeRO use ColoTensor as the base class. (#828) 
* [refactor] moving InsertPostInitMethodToModuleSubClasses to utils.

* [tensor] ZeRO use ColoTensor as the base class.

* polish
 2022-04-22 12:00:48 +08:00
Ziyue Jiang 8e6fdb4f29
[tensor]fix test_linear (#826) 2022-04-21 17:18:56 +08:00
Ziyue Jiang 1a9e2c2dff
[tensor] fix kwargs in colo_tensor torch_funtion (#825) 2022-04-21 16:47:35 +08:00
Jiarui Fang 2ecc3d7a55
[tensor] lazy init (#823) 2022-04-21 15:40:23 +08:00
Jiarui Fang 660d2d1f1b
[Tensor] apply ColoTensor on Torch functions (#821) 
* Revert "[zero] add ZeroTensorShardStrategy (#793)"

This reverts commit 88759e289e.

* [gemini] set cpu memory capacity

* [log] local throughput collecting

* polish

* polish

* polish

* polish code

* polish

* polish code

* add a new tensor structure and override linear for it

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* [tensor] renaming and reorganize directory structure.

* rm useless dir

* polish

* polish

* [tensor] hander the function not wrapped
 2022-04-21 14:21:10 +08:00
Jiarui Fang 0ce8924ceb
[tensor] reorganize files (#820) 2022-04-21 14:15:48 +08:00
Jiarui Fang ab962b9735
[gemini] a new tensor structure (#818) 
* Revert "[zero] add ZeroTensorShardStrategy (#793)"

This reverts commit 88759e289e.

* [gemini] set cpu memory capacity

* [log] local throughput collecting

* polish

* polish

* polish

* polish code

* polish

* polish code

* add a new tensor structure and override linear for it

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish
 2022-04-21 11:42:37 +08:00
Jiarui Fang e761ad2cd7
Revert "[zero] add ZeroTensorShardStrategy (#793)" (#806) 2022-04-19 14:40:02 +08:00
HELSON 88759e289e
[zero] add ZeroTensorShardStrategy (#793) 2022-04-19 14:32:45 +08:00
Jiarui Fang 681addb512
[refactor] moving grad acc logic to engine (#804) 2022-04-19 14:03:21 +08:00
Jiarui Fang 4d9332b4c5
[refactor] moving memtracer to gemini (#801) 2022-04-19 10:13:08 +08:00
HELSON 4c4388c46e
[hotfix] fix memory leak in zero (#781) 2022-04-18 13:57:03 +08:00
Frank Lee 5a1a095b92
[test] refactored with the new rerun decorator (#763) 
* [test] refactored with the new rerun decorator

* polish test case
 2022-04-15 00:33:04 +08:00
Jiarui Fang 10ef8afdd2
[gemini] init genimi individual directory (#754) 2022-04-14 16:40:26 +08:00
ver217 dcca614eee
[hotfix] fix test_stateful_tensor_mgr (#762) 2022-04-14 15:50:09 +08:00
ver217 a93a7d7364
[hotfix] fix reuse_fp16_shard of sharded model (#756) 
* fix reuse_fp16_shard

* disable test stm

* polish code
 2022-04-14 14:56:46 +08:00
HELSON 84c6700b2a
[zero] refactor memstats_collector (#746) 2022-04-14 12:01:12 +08:00
ver217 e396bb71f2
[zero] add tensor placement policies (#743) 
* add tensor placement policies

* polish comments

* polish comments

* update moe unit tests
 2022-04-13 15:00:48 +08:00
HELSON 22c4b88d56
[zero] refactor ShardedParamV2 for convenience (#742) 2022-04-13 14:54:26 +08:00
Frank Lee f4f42d4c3c
[bug] fixed DDP compatibility with torch 1.8 (#739) 2022-04-13 00:08:46 +08:00