ColossalAI/colossalai/nn/parallel
HELSON 1468e4bcfc
[zero] add constant placement policy (#1705)
* fixes memory leak when paramter is in fp16 in ZeroDDP init.
* bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release.
* adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.
2022-10-14 17:53:16 +08:00
..
layers [embedding] rename FreqAwareEmbedding -> CachedEmbedding (#1699) 2022-10-13 22:22:27 +08:00
__init__.py [ddp] refactor ColoDDP and ZeroDDP (#1146) 2022-06-21 16:35:23 +08:00
data_parallel.py [zero] add constant placement policy (#1705) 2022-10-14 17:53:16 +08:00
reducer.py [ddp] ColoDDP uses bucket all-reduce (#1177) 2022-06-29 10:34:13 +08:00
utils.py [feature] A new ZeRO implementation (#1644) 2022-10-09 09:18:51 +08:00