82 Commits (ce3c4eca7bc2c5b148dfe5db1ddb702558af4831)

Author	SHA1	Message	Date
Jiarui Fang	2827f41898	[Gemini] GeminiDPP convert to PyTorch Module. (#2151)	2 years ago
Jiarui Fang	9214d1fe28	[Gemini] chunk init using runtime visited param order (#2115)	2 years ago
Jiarui Fang	e5aa8333e4	[NFC] update chunk manager API (#2119)	2 years ago
Jiarui Fang	e99edfcb51	[NFC] polish comments for Chunk class (#2116)	2 years ago
HELSON	63fbba3c19	[zero] add L2 gradient clipping for ZeRO (#2112) ... * [zero] add L2 gradient clipping * [testing] add MlpModel * [zero] add unit test for grad clipping * fix atol	2 years ago
Jiarui Fang	1f99205827	[Gemini] remove static tracer (#2083)	2 years ago
Jiarui Fang	b3b89865e2	[Gemini] ParamOpHook -> ColoParamOpHook (#2080)	2 years ago
HELSON	e37f3db40c	[gemini] add arguments (#2046) ... * [zero] fix testing parameters * [gemini] add arguments * add docstrings	2 years ago
Jiarui Fang	96134e7be3	[hotfix] add bert test for gemini fwd bwd (#2035)	2 years ago
Jiarui Fang	8daf1b4db1	[Gemini] patch for supporting orch.add_ function for ColoTensor (#2003)	2 years ago
Jiarui Fang	cc0ed7cf33	[Gemini] ZeROHookV2 -> GeminiZeROHook (#1972)	2 years ago
Jiarui Fang	f7e276fa71	[Gemini] add GeminiAdamOptimizer (#1960)	2 years ago
Jiarui Fang	cd5a0d56fa	[Gemini] make gemini usage simple (#1821)	2 years ago
Zihao	20e255d4e8	MemStatsCollectorStatic (#1765)	2 years ago
HELSON	c6a1a62636	[hotfix] fix zero's incompatibility with checkpoint in torch-1.12 (#1786) ... * [hotfix] fix zero's incompatibility with checkpoint in torch-1.12 * [zero] add cpu shard init * [zero] add tiny example test * [colo_tensor] fix bugs for torch-1.11	2 years ago
HELSON	1468e4bcfc	[zero] add constant placement policy (#1705) ... * fixes memory leak when paramter is in fp16 in ZeroDDP init. * bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release. * adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.	2 years ago
Jiarui Fang	21962e1593	[embedding] rename FreqAwareEmbedding -> CachedEmbedding (#1699)	2 years ago
Jiarui Fang	363fc2861a	[embeddings] more detailed timer (#1692)	2 years ago
HELSON	b28991dd0a	[feature] A new ZeRO implementation (#1644)	2 years ago
Jiarui Fang	c638bec028	[embedding] polish async copy (#1657)	2 years ago
Jiarui Fang	988570e4a6	[embedding] add more detail profiling (#1656)	2 years ago
Jiarui Fang	e1f97fd2b8	[embedding] print profiling results (#1654)	2 years ago
Jiarui Fang	04443605a5	[embedding] non-blocking cpu-gpu copy (#1647)	2 years ago
CsRic	0767f67a0f	[embedding] isolate cache_op from forward (#1645) ... Co-authored-by: ric <mkkt_bkkt@mail.ustc.edu.cn>	2 years ago
Jiarui Fang	c5d39215f6	Revert "[feature] new zero implementation (#1623)" (#1643) ... This reverts commit 5be118f405.	2 years ago
HELSON	5be118f405	[feature] new zero implementation (#1623)	2 years ago
Jiarui Fang	e57df80325	[embeddings] cache option (#1635)	2 years ago
Jiarui Fang	38c68b5b9a	[embedding] rollback for better FAW performance (#1625)	2 years ago
Jiarui Fang	504ff1d101	[embeddings] use cache_ratio instead of cuda_row_num (#1611)	2 years ago
Jiarui Fang	a19eb80998	[embedding] updates some default parameters	2 years ago
CsRic	f3403ff98e	[embeddings] add already_split_along_rank flag for tablewise mode (#1584)	2 years ago
Jiatong Han	3263cdf57f	[NFC] polish colossalai/nn/parallel/data_parallel.py code style (#1570) ... Co-authored-by: JThh <jiatong.han@u.nus.edu>	2 years ago
CsRic	a389ac4ec9	[embedding] cache_embedding small improvement (#1564)	2 years ago
Jiarui Fang	64169f3e8f	[embedding] polish parallel embedding tablewise (#1545)	2 years ago
CsRic	964123ae0f	[embedding] freq_aware_embedding: add small functions for caller application (#1537)	2 years ago
Jiarui Fang	521078ffc9	[embedding] fix a bug in table wise sharding (#1538)	2 years ago
Jiarui Fang	87134524fd	[embedding] tablewise sharding polish (#1535)	2 years ago
CsRic	5156d5b4f8	[embedding] add tablewise sharding for FAW (#1526)	2 years ago
Jiarui Fang	4537d39df9	[doc] docstring for FreqAwareEmbeddingBag (#1525)	2 years ago
Jiarui Fang	9a9ef65313	[FAW] cpu caching operations (#1520)	2 years ago
Jiarui Fang	af5438caa2	[FAW] refactor reorder() for CachedParamMgr (#1514)	2 years ago
Jiarui Fang	9feee6d06b	[FAW] LFU initialize with dataset freq (#1513)	2 years ago
CsRic	1b8fee8e9c	[FAW] shrink freq_cnter size (#1509)	2 years ago
Jiarui Fang	ba61109b6c	[FAW] remove code related to chunk (#1501)	2 years ago
Jiarui Fang	d5085bb317	[FAW] add more docs and fix a warning (#1500)	2 years ago
CsRic	0ed2f46131	[FAW] FAW embedding use LRU as eviction strategy intialized with dataset stats (#1494)	2 years ago
CsRic	b8d0e39eaf	[FAW] LFU cache for the FAW	2 years ago
Jiarui Fang	cde7b8a5b8	[FAW] init an LFU implementation for FAW (#1488)	2 years ago
Geng Zhang	0aad53c62b	[FCE] update interface for frequency statistics in FreqCacheEmbedding (#1462)	2 years ago
Jiarui Fang	a1476ea882	[NFC] polish doc style for ColoTensor (#1457)	2 years ago