github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
ColossalAI/applications/ColossalChat/benchmarks
History
YeAnbang 544b7a38a1 fix style, add kto data sample 2024-07-18 08:38:56 +00:00
..
ray [ColossalChat] Update RLHF V2 (#5286) 2024-03-29 14:12:29 +08:00
Opt.json [ColossalChat] Update RLHF V2 (#5286) 2024-03-29 14:12:29 +08:00
README.md [ColossalChat] Update RLHF V2 (#5286) 2024-03-29 14:12:29 +08:00
benchmark_dpo.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-07-10 10:44:32 +00:00
benchmark_dpo.sh fix style 2024-07-10 10:37:17 +00:00
benchmark_kto.py add kto 2024-07-18 07:54:11 +00:00
benchmark_kto.sh fix style, add kto data sample 2024-07-18 08:38:56 +00:00
benchmark_memory_consumption.txt [ColossalChat] Update RLHF V2 (#5286) 2024-03-29 14:12:29 +08:00
benchmark_orpo.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-07-10 10:44:32 +00:00
benchmark_orpo.sh fix style 2024-07-10 10:37:17 +00:00
benchmark_performance_summarization.txt [ColossalChat] Update RLHF V2 (#5286) 2024-03-29 14:12:29 +08:00
benchmark_ppo.py [misc] refactor launch API and tensor constructor (#5666) 2024-04-29 10:40:11 +08:00
benchmark_ppo.sh [ColossalChat] Update RLHF V2 (#5286) 2024-03-29 14:12:29 +08:00
benchmark_sft.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-07-10 10:44:32 +00:00
benchmark_sft.sh fix style, add kto data sample 2024-07-18 08:38:56 +00:00
data_preparation.sh [ColossalChat] Update RLHF V2 (#5286) 2024-03-29 14:12:29 +08:00
dummy_dataset.py add kto 2024-07-18 07:54:11 +00:00

README.md

Benchmarks

Benchmark OPT with LoRA on dummy prompt data

We provide various OPT models (string in parentheses is the corresponding model name used in this script):

  • OPT-125M (125m)
  • OPT-350M (350m)
  • OPT-700M (700m)
  • OPT-1.3B (1.3b)
  • OPT-2.7B (2.7b)
  • OPT-3.5B (3.5b)
  • OPT-5.5B (5.5b)
  • OPT-6.7B (6.7b)
  • OPT-10B (10b)
  • OPT-13B (13b)

We also provide various training strategies:

  • gemini: ColossalAI GeminiPlugin with placement_policy="cuda", like zero3
  • gemini_auto: ColossalAI GeminiPlugin with placement_policy="cpu", like zero3-offload
  • zero2: ColossalAI zero2
  • zero2_cpu: ColossalAI zero2-offload
  • 3d: ColossalAI HybridParallelPlugin with TP, DP support

How to Run

cd ../tests
# Prepare data for benchmark
SFT_DATASET=/path/to/sft/data/ \
PROMPT_DATASET=/path/to/prompt/data/ \
PRETRAIN_DATASET=/path/to/ptx/data/ \
PREFERENCE_DATASET=/path/to/preference/data \
./test_data_preparation.sh
# Start benchmark
./benchmark_ppo.sh