mirror of https://github.com/hpcaitech/ColossalAI
[readme] sync CN readme (#766)
parent
6e553748a7
commit
deaf99f4c9
|
@ -71,26 +71,30 @@
|
||||||
|
|
||||||
Colossal-AI 为您提供了一系列并行训练组件。我们的目标是让您的分布式 AI 模型训练像普通的单 GPU 模型一样简单。我们提供的友好工具可以让您在几行代码内快速开始分布式训练。
|
Colossal-AI 为您提供了一系列并行训练组件。我们的目标是让您的分布式 AI 模型训练像普通的单 GPU 模型一样简单。我们提供的友好工具可以让您在几行代码内快速开始分布式训练。
|
||||||
|
|
||||||
- 数据并行
|
- 并行化策略
|
||||||
- 流水线并行
|
- 数据并行
|
||||||
- 1维, 2维, 2.5维, 3维张量并行
|
- 流水线并行
|
||||||
- 序列并行
|
- 1维, [2维](https://arxiv.org/abs/2104.05343), [2.5维](https://arxiv.org/abs/2105.14500), [3维](https://arxiv.org/abs/2105.14450) 张量并行
|
||||||
- 友好的 trainer 和 engine
|
- [序列并行](https://arxiv.org/abs/2105.13120)
|
||||||
- 可扩展新的并行方式
|
- [零冗余优化器 (ZeRO)](https://arxiv.org/abs/2108.05818)
|
||||||
- 混合精度
|
- 异构内存管理
|
||||||
- 零冗余优化器 (ZeRO)
|
- [PatrickStar](https://arxiv.org/abs/2108.05818)
|
||||||
|
- 使用友好
|
||||||
|
- 基于参数文件的并行化
|
||||||
<p align="right">(<a href="#top">返回顶端</a>)</p>
|
<p align="right">(<a href="#top">返回顶端</a>)</p>
|
||||||
|
|
||||||
## 展示样例
|
## 展示样例
|
||||||
### ViT
|
### ViT
|
||||||
|
<p align="center">
|
||||||
<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/ViT.png" width="450" />
|
<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/ViT.png" width="450" />
|
||||||
|
</p>
|
||||||
|
|
||||||
- 14倍批大小和5倍训练速度(张量并行=64)
|
- 14倍批大小和5倍训练速度(张量并行=64)
|
||||||
|
|
||||||
### GPT-3
|
### GPT-3
|
||||||
|
<p align="center">
|
||||||
<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/GPT3.png" width=700/>
|
<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/GPT3.png" width=700/>
|
||||||
|
</p>
|
||||||
|
|
||||||
- 释放 50% GPU 资源占用, 或 10.7% 加速
|
- 释放 50% GPU 资源占用, 或 10.7% 加速
|
||||||
|
|
||||||
|
|
|
@ -76,15 +76,15 @@ distributed training in a few lines.
|
||||||
- Parallelism strategies
|
- Parallelism strategies
|
||||||
- Data Parallelism
|
- Data Parallelism
|
||||||
- Pipeline Parallelism
|
- Pipeline Parallelism
|
||||||
- 1D, [2D](https://arxiv.org/abs/2104.05343), [2.5D](https://arxiv.org/abs/2105.14500), 3D Tensor parallelism
|
- 1D, [2D](https://arxiv.org/abs/2104.05343), [2.5D](https://arxiv.org/abs/2105.14500), [3D](https://arxiv.org/abs/2105.14450) Tensor Parallelism
|
||||||
- [Sequence parallelism](https://arxiv.org/abs/2105.13120)
|
- [Sequence Parallelism](https://arxiv.org/abs/2105.13120)
|
||||||
- [Zero Redundancy Optimizer (ZeRO)](https://arxiv.org/abs/2108.05818)
|
- [Zero Redundancy Optimizer (ZeRO)](https://arxiv.org/abs/2108.05818)
|
||||||
|
|
||||||
- Heterogeneous Memory Menagement
|
- Heterogeneous Memory Menagement
|
||||||
- [PatrickStar](https://arxiv.org/abs/2108.05818)
|
- [PatrickStar](https://arxiv.org/abs/2108.05818)
|
||||||
|
|
||||||
- Friendly Usage
|
- Friendly Usage
|
||||||
- Configuration file based
|
- Parallelism based on configuration file
|
||||||
|
|
||||||
<p align="right">(<a href="#top">back to top</a>)</p>
|
<p align="right">(<a href="#top">back to top</a>)</p>
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue