github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

[ColoTensor] throw error when ColoInitContext meets meta parameter. (#2105)

pull/2106/head^2
Jiarui Fang 2022-12-09 11:39:46 +08:00 committed by GitHub
parent d87baa85d9
commit 05545bfee9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 7 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
colossalai/utils/model/colo_init_context.py
View File

@ -36,6 +36,11 @@ def _convert_to_coloparam(param: torch.nn.Parameter,
return param return param
# detaching tensor is necessary for optimizers. # detaching tensor is necessary for optimizers.
requires_grad = param.requires_grad requires_grad = param.requires_grad
if param.device.type == 'meta':
raise NotImplemented(
"ColoInitContext is initializing a model with meta parameters! This is not allowed right now!")
else:
# param is the global tensor. # param is the global tensor.
colo_param = ColoParameter(param.to(device=device, dtype=dtype), requires_grad=requires_grad) colo_param = ColoParameter(param.to(device=device, dtype=dtype), requires_grad=requires_grad)