github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
ColossalAI/applications/Colossal-LLaMA/dataset
History
Tong Li 4a68efb7da
[Colossal-LLaMA] Refactor latest APIs (#6030) 
* refactor latest code

* update api

* add dummy dataset

* update Readme

* add setup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update files

* add PP support

* update arguments

* update argument

* reorg folder

* update version

* remove IB infor

* update utils

* update readme

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update save for zero

* update save

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add apex

* update

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 		2024-08-28 17:01:58 +08:00
..
prepare_pretrain_dataset.py [Colossal-LLaMA] Refactor latest APIs (#6030) 2024-08-28 17:01:58 +08:00
prepare_sft_dataset.py [Colossal-LLaMA] Refactor latest APIs (#6030) 2024-08-28 17:01:58 +08:00