pull/3810/head
jiangmingyan 2 years ago
parent 8aa1fb2c7f
commit 278fcbc444

@ -128,7 +128,7 @@ for idx, (img, label) in enumerate(train_dataloader):
### Step 6. Invoke Training Scripts
To verify gradient accumulation, we can just check the change of parameter values. When gradient accumulation is set, parameters are only updated in the last step. You can run the script using this command:
```shell
colossalai run --nproc_per_node 1 train.py --config config.py
colossalai run --nproc_per_node 1 train.py
```
You will see output similar to the text below. This shows gradient is indeed accumulated as the parameter is not updated

@ -136,7 +136,7 @@ for idx, (img, label) in enumerate(train_dataloader):
You can run the script using this command:
```shell
colossalai run --nproc_per_node 1 train.py --config config/config.py
colossalai run --nproc_per_node 1 train.py
```
<!-- doc-test-command: torchrun --standalone --nproc_per_node=1 gradient_clipping_with_booster.py -->

@ -131,7 +131,7 @@ for idx, (img, label) in enumerate(train_dataloader):
### 步骤 6. 启动训练脚本
为了验证梯度累积,我们可以只检查参数值的变化。当设置梯度累加时,仅在最后一步更新参数。您可以使用以下命令运行脚本:
```shell
colossalai run --nproc_per_node 1 train.py --config config.py
colossalai run --nproc_per_node 1 train.py
```
你将会看到类似下方的文本输出。这展现了梯度虽然在前3个迭代中被计算但直到最后一次迭代参数才被更新。

@ -135,6 +135,6 @@ for idx, (img, label) in enumerate(train_dataloader):
你可以使用以下命令运行脚本:
```shell
colossalai run --nproc_per_node 1 train.py --config config/config.py
colossalai run --nproc_per_node 1 train.py
```
<!-- doc-test-command: torchrun --standalone --nproc_per_node=1 gradient_clipping_with_booster.py -->

Loading…
Cancel
Save