github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

[chore] docstring

colossalchat
hxwang 2024-07-25 09:07:49 +00:00 committed by Hongxin Liu
parent 7e737df5ad
commit f7c5485ed6
1 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
colossalai/booster/plugin/moe_hybrid_parallel_plugin.py
View File

@ -115,8 +115,10 @@ class MoeHybridParallelPlugin(HybridParallelPlugin):
"""
Modified from colossalai.booster.plugin.hybrid_parallel_plugin.HybridParallelPlugin
Extra Args:
ep_size (int): The size of tensor parallelism. Tensor parallelism will not be used when tp_size is set to 1.
force_overlap_comm (bool): For LowLevelZeroOptimizer, it might causes program hang when some experts are routed and overlap_communication is True during training. This flag is used to force overlap_communication=True.
ep_size (int): The size of expert parallelism
force_overlap_comm (bool):
For LowLevelZeroOptimizer, it might causes program hang when some experts are routed and overlap_communication is True during training.
This flag is used to force overlap_communication=True. Make sure every expert are routed when you use this.
"""
def __init__(