github
/
InternLM
mirror of https://github.com/InternLM/InternLM
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

make seed in different tensor rank different (#405)

pull/408/head
ytxiong 2023-10-09 13:53:52 +08:00 committed by GitHub
parent 8b65e2e3c4
commit 9aef11e89c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
internlm/core/context/parallel_context.py
View File

@ -568,7 +568,8 @@ class ParallelContext(metaclass=SingletonMeta):
# during model construction), this is because the random state will be different in different tensor parallel
# device of the same data parallel group. The underlying reason is that the device of tp_rank = 0 will perform
# additional random operations during the RowParallelLinear module building process.
set_mode(ParallelMode.DUMMY)
# set_mode(ParallelMode.DUMMY)
set_mode(ParallelMode.TENSOR)
seeds = get_seeds()
seed_str = ", ".join([f"{k}: {v}" for k, v in seeds.items()])