e9460b45c8
[engin/schedule] use p2p_v2 to recontruct pipeline_schedule ( #1408 )
...
* support p2p communication with any type of object | pass test
* reconstruct pipeline schedule with p2p_v2.py(support communication with List[Any]) | pass test
* [communication] add p2p_v2.py to support communication with List[Any]
* Delete _pipeline_schedule_v2.py
* Delete test_cifar_with_data_pipeline_tensor_v2.py
* [engin/schedule] use p2p_v2 to recontruct pipeline_schedule
* [engin/schedule] use p2p_v2 to recontruct pipeline_schedule
* [engin/schedule] use p2p_v2 to recontruct pipeline_schedule
* [engin/schedule] use p2p_v2 to recontruct pipeline_schedule
* [engin/schedule] use p2p_v2 to recontruct pipeline_schedule
* Delete p2p_v2.py
* Delete test_boardcast_send_recv_v2.py
* Delete test_object_list_p2p_v2.py
* [engin/schedule] use p2p_v2 to recontruct pipeline_schedule
* [communication] remove print code
* [communication] remove print code
* [engin/schedule] shorten the running time of testing file to prevent cancelling in CI
2022-08-12 11:33:26 +08:00
7c70bfbefa
[hotfix] fix PipelineSharedModuleGradientHandler ( #1314 )
2022-07-14 17:31:13 +08:00
4165eabb1e
[hotfix] remove potiential circle import ( #1307 )
...
* make it faster
* [hotfix] remove circle import
2022-07-14 13:44:26 +08:00
50f2ad213f
[NFC] polish colossalai/engine/ophooks/utils.py code style ( #1256 )
2022-07-13 12:08:21 +08:00
17ed33350b
[hotfix] fix an assertion bug in base schedule. ( #1250 )
2022-07-12 14:20:02 +08:00
f1f51990b9
[hotfix]fix some bugs caused by refactored schedule. ( #1148 )
...
* [CLI] add CLI launcher
* Revert "[CLI] add CLI launcher"
This reverts commit df7e6506d4
2022-06-21 22:46:30 +08:00
18091581c0
[pipeline]support more flexible pipeline ( #1138 )
...
* [CLI] add CLI launcher
* Revert "[CLI] add CLI launcher"
This reverts commit df7e6506d4
2022-06-21 14:40:50 +08:00
946dbd629d
[hotfix]fix bugs caused by refactored pipeline ( #1133 )
...
* [CLI] add CLI launcher
* Revert "[CLI] add CLI launcher"
This reverts commit df7e6506d4
2022-06-17 17:54:15 +08:00
3175bcb4d8
[pipeline]support List of Dict data ( #1125 )
...
* [CLI] add CLI launcher
* Revert "[CLI] add CLI launcher"
This reverts commit df7e6506d4
2022-06-16 11:19:48 +08:00
6f82ac9bcb
[pipeline] supported more flexible dataflow control for pipeline parallel training ( #1108 )
...
* [pipeline] supported more flexible dataflow control for pipeline parallel training
* polish code
* polish code
* polish code
2022-06-15 10:41:28 +08:00
1e9f9c227f
[hotfix]change to fit latest p2p ( #1100 )
...
* [CLI] add CLI launcher
* Revert "[CLI] add CLI launcher"
This reverts commit df7e6506d4
2022-06-13 14:57:25 +08:00
7f2d2b2b5b
[engine] fixed empty op hook check ( #1096 )
...
* [engine] fixed empty op hook check
* polish code
2022-06-10 17:27:27 +08:00
b167258b6a
[pipeline]refactor ppschedule to support tensor list ( #1050 )
...
* [CLI] add CLI launcher
* Revert "[CLI] add CLI launcher"
This reverts commit df7e6506d4
2022-06-02 13:48:59 +08:00
e4685832f8
[engine] fixed bug in gradient accumulation dataloader to keep the last step ( #1030 )
2022-05-26 14:28:23 +08:00
32a45cd7ef
[pipelinable]use pipelinable to support GPT model. ( #903 )
...
* [CLI] add CLI launcher
* Revert "[CLI] add CLI launcher"
This reverts commit df7e6506d4
2022-05-11 09:23:58 +08:00
11f54c7b6b
[doc] improved docstring and assertion messages for the engine module ( #871 )
2022-04-26 10:00:18 +08:00
681addb512
[refactor] moving grad acc logic to engine ( #804 )
2022-04-19 14:03:21 +08:00
4d9332b4c5
[refactor] moving memtracer to gemini ( #801 )
2022-04-19 10:13:08 +08:00
84c6700b2a
[zero] refactor memstats_collector ( #746 )
2022-04-14 12:01:12 +08:00
4d90a7b513
[refactor] zero directory ( #724 )
2022-04-11 23:13:02 +08:00
193dc8dacb
[refactor] refactor the memory utils ( #715 )
2022-04-11 16:47:57 +08:00
ee112fe1da
[zero] adapt zero hooks for unsharded module ( #699 )
2022-04-08 20:23:26 +08:00
3c9cd5bb5e
[zero] stateful tensor manager ( #687 )
...
* [WIP] stateful tensor manager
* add eviction strategy
* polish code
* polish code
* polish comment
* add unit test
* fix sampler bug
* polish code
* fix max sampling cnt resetting bug
* fix sampler bug
* polish code
* fix bug
* fix unit test
Co-authored-by: jiaruifang <fangjiarui123@gmail.com>
2022-04-08 17:51:34 +08:00
0ed7042f42
[pipeline] refactor pipeline ( #679 )
...
* refactor pipeline---put runtime schedule into engine.
* add type hint for schedule Optional[BaseSchedule]
* preprocess schedule during engine initializing
* infer pipeline schedule params from config
2022-04-07 15:54:14 +08:00
ad1e7ab2b2
'[NFC] polish <colossalai/engine/_base_engine.py> code style' ( #631 )
...
Co-authored-by: RichardoLuo <14049555596@qq.com>
2022-04-06 11:40:59 +08:00
f2da21a827
fix format ( #586 )
2022-04-06 11:40:59 +08:00
ffad81e1d1
fix format ( #585 )
...
Co-authored-by: fanjifu <FAN>
2022-04-06 11:40:59 +08:00
d2dc6049b5
fix format ( #580 )
2022-04-06 11:40:59 +08:00
cfb41297ff
'fix/format' ( #573 )
2022-04-06 11:40:59 +08:00
ade05a5d83
[refactor] pipeline, put runtime schedule into engine. ( #627 )
2022-04-03 20:46:45 +08:00
e956d93ac2
[refactor] memory utils ( #577 )
2022-04-01 09:22:33 +08:00
e6d50ec107
[zero] adapt zero for unsharded parameters ( #561 )
...
* support existing sharded and unsharded parameters in zero
* add unitest for moe-zero model init
* polish moe gradient handler
2022-03-31 18:34:11 +08:00
7675366fce
[polish] rename col_attr -> colo_attr ( #558 )
2022-03-31 12:25:45 +08:00
014bac0c49
[zero] hijack p.grad in sharded model ( #554 )
...
* hijack p.grad in sharded model
* polish comments
* polish comments
2022-03-30 18:14:50 +08:00
f552b11294
[zero] label state for param fp16 and grad ( #551 )
2022-03-30 15:57:46 +08:00
214da761d4
[zero] add stateful tensor ( #549 )
2022-03-30 13:51:37 +08:00
ec5086c49c
Refactored docstring to google style
2022-03-29 17:17:47 +08:00
73d36618a6
[profiler] add MemProfiler ( #356 )
...
* add memory trainer hook
* fix bug
* add memory trainer hook
* fix import bug
* fix import bug
* add trainer hook
* fix #370 git log bug
* modify `to_tensorboard` function to support better output
* remove useless output
* change the name of `MemProfiler`
* complete memory profiler
* replace error with warning
* finish trainer hook
* modify interface of MemProfiler
* modify `__init__.py` in profiler
* remove unnecessary pass statement
* add usage to doc string
* add usage to trainer hook
* new location to store temp data file
2022-03-29 12:48:34 +08:00
a30e2b4c24
[zero] adapt for no-leaf module in zero ( #535 )
...
only process module's own parameters in Zero context
add zero hooks for all modules that contrain parameters
gather parameters only belonging to module itself
2022-03-28 17:42:18 +08:00
705f56107c
[zero] refactor model data tracing ( #537 )
2022-03-28 16:38:18 +08:00
4d322b79da
[refactor] remove old zero code ( #517 )
2022-03-25 14:54:39 +08:00
920c5889a7
[zero] add colo move inline ( #521 )
2022-03-25 14:02:55 +08:00
a445e118cf
[polish] polish singleton and global context ( #500 )
2022-03-23 18:03:39 +08:00
b334822163
[zero] polish sharded param name ( #484 )
...
* [zero] polish sharded param name
* polish code
* polish
* polish code
* polish
* polsih
* polish
2022-03-22 14:36:16 +08:00
65c0f380c2
[format] polish name format for MOE ( #481 )
2022-03-21 23:19:47 +08:00
8d3250d74b
[zero] ZeRO supports pipeline parallel ( #477 )
2022-03-21 16:55:37 +08:00
aff9d354f7
[MOE] polish moe_env ( #467 )
2022-03-19 15:36:25 +08:00
84fd7c1d4d
add moe context, moe utilities and refactor gradient handler ( #455 )
2022-03-18 16:38:32 +08:00
a241f61b34
[zero] Update initialize for ZeRO ( #458 )
...
* polish code
* shard strategy receive pg in shard() / gather()
* update zero engine
* polish code
2022-03-18 16:18:31 +08:00
9506a8beb2
use double buffer to handle grad
2022-03-16 14:24:09 +08:00
Kirigaya Kazuto