ColossalAI/examples/language/llama2/scripts/benchmark_70B/3d.sh

#!/bin/bash

# TODO: fix this
echo "3D parallel for LLaMA-2 is not ready yet"
exit 1

################
#Load your environments and modules here
################

HOSTFILE=$(realpath hosts.txt)

cd ../..

export OMP_NUM_THREADS=8

colossalai run --nproc_per_node 8 --hostfile $HOSTFILE benchmark.py -c 70b -p 3d -g -x -b 8 --tp 4 --pp 2 --mbs 1
[example] add llama2 example (#4527) * [example] transfer llama-1 example * [example] fit llama-2 * [example] refactor scripts folder * [example] fit new gemini plugin * [cli] fix multinode runner * [example] fit gemini optim checkpoint * [example] refactor scripts * [example] update requirements * [example] update requirements * [example] rename llama to llama2 * [example] update readme and pretrain script * [example] refactor scripts 2023-08-28 09:59:11 +00:00			`#!/bin/bash`

			`# TODO: fix this`
			`echo "3D parallel for LLaMA-2 is not ready yet"`
			`exit 1`

			`################`
			`#Load your environments and modules here`
			`################`

			`HOSTFILE=$(realpath hosts.txt)`

			`cd ../..`

			`export OMP_NUM_THREADS=8`

[hotfix]: add pp sanity check and fix mbs arg (#5268) * fix: fix misleading mbs arg * feat: add pp sanity check * fix: fix 1f1b sanity check 2024-01-15 07:57:40 +00:00			`colossalai run --nproc_per_node 8 --hostfile $HOSTFILE benchmark.py -c 70b -p 3d -g -x -b 8 --tp 4 --pp 2 --mbs 1`