From 936d0b0f7ba7f9b4e0d53c343bcf6afd10c63de1 Mon Sep 17 00:00:00 2001 From: Edenzzzz Date: Mon, 1 Jul 2024 17:07:22 +0800 Subject: [PATCH] [doc] Update llama + sp compatibility; fix dist optim table Co-authored-by: Edenzzzz --- .../en/features/distributed_optimizers.md | 36 +++++++++--------- docs/source/en/features/shardformer.md | 2 +- .../features/distributed_optimizers.md | 37 +++++++++---------- docs/source/zh-Hans/features/shardformer.md | 2 +- 4 files changed, 37 insertions(+), 40 deletions(-) diff --git a/docs/source/en/features/distributed_optimizers.md b/docs/source/en/features/distributed_optimizers.md index f95b23304..279bc8f9d 100644 --- a/docs/source/en/features/distributed_optimizers.md +++ b/docs/source/en/features/distributed_optimizers.md @@ -87,44 +87,42 @@ optim = DistGaloreAwamW( ## Plugin compatibility - - - - - + + + + + + - - + + + - + - + + - + - - - - - - - - - + + + + diff --git a/docs/source/en/features/shardformer.md b/docs/source/en/features/shardformer.md index 68d310f5c..40b8954b5 100644 --- a/docs/source/en/features/shardformer.md +++ b/docs/source/en/features/shardformer.md @@ -55,7 +55,7 @@ Model/Feature Compatibility Matrix: - + diff --git a/docs/source/zh-Hans/features/distributed_optimizers.md b/docs/source/zh-Hans/features/distributed_optimizers.md index 7a7068077..5761f8c55 100644 --- a/docs/source/zh-Hans/features/distributed_optimizers.md +++ b/docs/source/zh-Hans/features/distributed_optimizers.md @@ -84,44 +84,42 @@ optim = DistGaloreAwamW( ## 兼容性
Model/FeatureLambGaLoreAdafactorCAMEOptimizer/PluginHybrid Parallel PluginLow Level Zero PluginTorch DDP PluginGemini PluginMoe Hybrid Plugin
Hybrid Parallel
Plugin		✔️Lamb ✔️ ✔️ ✔️❌❌
Low Level Zero
Plugin		GaLore ✔️❌ ✔️ ✔️❌❌
Torch DDP
Plugin		Adafactor ✔️ ✔️ ✔️✔️
Gemini
Plugin		❌❌ ❌ ❌
Moe Hybrid
Plugin		❌❌CAME✔️✔️✔️ ❌ ❌
✔️ ✔️ ✔️❌✔️ ❌
- - - - - + + + + + + - - + + + - + - + + - - + - - - - - - - - + + + + @@ -130,6 +128,7 @@ optim = DistGaloreAwamW(
Model/FeatureLambGaLoreAdafactorCAMEOptimizer/PluginHybrid Parallel PluginLow Level Zero PluginTorch DDP PluginGemini PluginMoe Hybrid Plugin
Hybrid Parallel
Plugin		✔️Lamb ✔️ ✔️ ✔️❌❌
Low Level Zero
Plugin		GaLore ✔️❌ ✔️ ✔️❌❌
Torch DDP
Plugin		✔️Adafactor ✔️ ✔️ ✔️
Gemini
Plugin		❌❌ ❌ ❌
Moe Hybrid
Plugin		❌❌CAME✔️✔️✔️ ❌ ❌
+ ## API 参考 diff --git a/docs/source/zh-Hans/features/shardformer.md b/docs/source/zh-Hans/features/shardformer.md index 00e1a13d6..02290f3d6 100644 --- a/docs/source/zh-Hans/features/shardformer.md +++ b/docs/source/zh-Hans/features/shardformer.md @@ -51,7 +51,7 @@ Author: [Baizhou Zhang](https://github.com/Fridge003), [Bin Jia](https://github. ✔️ ✔️ ✔️ - ❌ + ✔️ ❌