LuGY c6ab96983a [zero] refactor low level zero for shard evenly (#4030) ... * refactor low level zero * fix zero2 and support cpu offload * avg gradient and modify unit test * refactor grad store, support layer drop * refactor bucket store, support grad accumulation * fix and update unit test of zero and ddp * compatible with tp, ga and unit test * fix memory leak and polish * add zero layer drop unittest * polish code * fix import err in unit test * support diffenert comm dtype, modify docstring style * polish code * test padding and fix * fix unit test of low level zero * fix pad recording in bucket store * support some models * polish		2023-07-31 22:13:29 +08:00
..
__init__.py	[zero] reorganize zero/gemini folder structure (#3424)	2023-04-04 13:48:16 +08:00
base_store.py	[zero] reorganize zero/gemini folder structure (#3424)	2023-04-04 13:48:16 +08:00
bucket_store.py	[zero] refactor low level zero for shard evenly (#4030)	2023-07-31 22:13:29 +08:00
gradient_store.py	[zero] refactor low level zero for shard evenly (#4030)	2023-07-31 22:13:29 +08:00
parameter_store.py	[zero] refactor low level zero for shard evenly (#4030)	2023-07-31 22:13:29 +08:00
tensor_bucket.py	[zero] reorganize zero/gemini folder structure (#3424)	2023-04-04 13:48:16 +08:00