github
/
InternLM
mirror of https://github.com/InternLM/InternLM
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Update train_performance.md

pull/39/head
Sun Peng 2023-07-08 18:09:02 +08:00 committed by GitHub
parent 0bfb937710
commit 3216328a67
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 15 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
doc/train_performance.md
View File

@ -59,4 +59,18 @@ activation checkpoint | tp | zero-1 | seq_len | micro_num | micro_bsz |
<div align="left">
<img src="../doc/imgs/flops.png" width="580"/>
</div>
</div>
| TP | ZERO1 | pack_sample | activation ckpt | gpu | seq_len | micro_bsz | micro_num | global tokens/step (M) | TGS | TFLOPS |
|-|-|-|-|-|-|-|-|-|-|-|
| 1 | 8 | TRUE | TRUE | 8 | 2048 | 8 | 1 | 0.125 | 3188 | 186 |
| 1 | 8 | TRUE | TRUE | 16 | 2048 | 8 | 1 | 0.25 | 3130 | 182 |
| 1 | 8 | TRUE | TRUE | 32 | 2048 | 8 | 1 | 0.5 | 3095 | 180 |
| 1 | 8 | TRUE | TRUE | 64 | 2048 | 8 | 1 | 1 | 3073 | 179 |
| 1 | 8 | TRUE | TRUE | 128 | 2048 | 8 | 1 | 2 | 3041 | 177 |
| 1 | 8 | TRUE | TRUE | 256 | 2048 | 8 | 1 | 4 | 3040 | 177 |
| 1 | 8 | TRUE | TRUE | 512 | 2048 | 8 | 1 | 8 | 3059 | 178 |
| 1 | 8 | TRUE | TRUE | 1024 | 2048 | 8 | 1 | 16 | 3081 | 179 |
| 1 | 8 | TRUE | TRUE | 512 | 2048 | 4 | 1 | 4 | 2865 | 165 |
| <bgcolor=#D0E8F2>1 | 8 | TRUE | TRUE | 1024 | 2048 | 2 | 1 | 4 | 1832 | 110</bgcolor> |
| <bgcolor=#D0E8F2>1 | 8 | TRUE | TRUE | 1024 | 2048 | 4 | 1 | 8 | 2796 | 160</bgcolor> |