5 Commits (1712da2800e8bc2b539583692668c13b267ed7af)

Author	SHA1	Message	Date
HELSON	1468e4bcfc	[zero] add constant placement policy (#1705) ... * fixes memory leak when paramter is in fp16 in ZeroDDP init. * bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release. * adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.	2 years ago
HELSON	95c35f73bd	[moe] initialize MoE groups by ProcessGroup (#1640)	2 years ago
Jiarui Fang	a445e118cf	[polish] polish singleton and global context (#500)	3 years ago
Jiarui Fang	65c0f380c2	[format] polish name format for MOE (#481)	3 years ago
HELSON	84fd7c1d4d	add moe context, moe utilities and refactor gradient handler (#455)	3 years ago