Hotfix/tutorial readme index (#1922)

* [tutorial] removed tutorial index in readme

* [tutorial] removed tutorial index in readme
pull/1924/head
Frank Lee 2 years ago committed by GitHub
parent 24cbee0ebe
commit d43a671ad6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -1,4 +1,4 @@
# Handson 3: Auto-Parallelism with ResNet
# Auto-Parallelism with ResNet
## Prepare Dataset

@ -1,4 +1,4 @@
# Handson 1: Multi-dimensional Parallelism with Colossal-AI
# Multi-dimensional Parallelism with Colossal-AI
## Install Titans Model Zoo

@ -1,4 +1,4 @@
# Handson 4: Comparison of Large Batch Training Optimization
# Comparison of Large Batch Training Optimization
## Prepare Dataset

@ -1 +1 @@
# Handson 5: Fine-tuning and Serving for OPT from Hugging Face
# Fine-tuning and Serving for OPT from Hugging Face

@ -1,6 +1,6 @@
# Handson 2: Sequence Parallelism with BERT
# Sequence Parallelism with BERT
In this example, we implemented BERT with sequence parallelism. Sequence parallelism splits the input tensor and intermediate
In this example, we implemented BERT with sequence parallelism. Sequence parallelism splits the input tensor and intermediate
activation along the sequence dimension. This method can achieve better memory efficiency and allows us to train with larger batch size and longer sequence length.
Paper: [Sequence Parallelism: Long Sequence Training from System Perspective](https://arxiv.org/abs/2105.13120)
@ -16,7 +16,7 @@ First, let's prepare the WikiPedia dataset from scratch. To generate a preproces
For the preprocessing script, we thank Megatron-LM for providing a preprocessing script to generate the corpus file.
```python
# download raw data
# download raw data
mkdir data && cd ./data
wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
@ -24,7 +24,7 @@ wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.
git clone https://github.com/FrankLeeeee/wikiextractor.git
pip install ./wikiextractor
# extractmodule
# extractmodule
wikiextractor --json enwiki-latest-pages-articles.xml.bz2
cat text/*/* > ./corpus.json
cd ..
@ -34,7 +34,7 @@ mkdir vocab && cd ./vocab
wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt
cd ..
# preprocess some data
# preprocess some data
git clone https://github.com/NVIDIA/Megatron-LM.git
cd ./Megatron-LM
python tools/preprocess_data.py \
@ -86,12 +86,12 @@ class Encoder(object):
## How to Train with Sequence Parallelism
We provided `train.py` for you to execute training. Before invoking the script, there are several
We provided `train.py` for you to execute training. Before invoking the script, there are several
steps to perform.
### Step 1. Set data path and vocab path
At the top of `config.py`, you can see two global variables `DATA_PATH` and `VOCAB_FILE_PATH`.
At the top of `config.py`, you can see two global variables `DATA_PATH` and `VOCAB_FILE_PATH`.
```python
DATA_PATH = <data-path>
@ -106,7 +106,7 @@ For example, if your my-bert_text_sentence.bin is /home/Megatron-LM/my-bert_text
DATA_PATH = '/home/Megatron-LM/my-bert_text_sentence'
```
The `VOCAB_FILE_PATH` refers to the path to the vocabulary downloaded when you prepare the dataset
The `VOCAB_FILE_PATH` refers to the path to the vocabulary downloaded when you prepare the dataset
(e.g. bert-large-uncased-vocab.txt).
### Step 3. Make Dataset Helper
@ -121,12 +121,12 @@ make
### Step 3. Configure your parameters
In the `config.py` provided, a set of parameters are defined including training scheme, model, etc.
You can also modify the ColossalAI setting. For example, if you wish to parallelize over the
You can also modify the ColossalAI setting. For example, if you wish to parallelize over the
sequence dimension on 8 GPUs. You can change `size=4` to `size=8`. If you wish to use pipeline parallelism, you can set `pipeline=<num_of_pipeline_stages>`.
### Step 4. Invoke parallel training
Lastly, you can start training with sequence parallelism. How you invoke `train.py` depends on your
Lastly, you can start training with sequence parallelism. How you invoke `train.py` depends on your
machine setting.
- If you are using a single machine with multiple GPUs, PyTorch launch utility can easily let you
@ -137,7 +137,6 @@ machine setting.
```
- If you are using multiple machines with multiple GPUs, we suggest that you refer to `colossalai
launch_from_slurm` or `colossalai.launch_from_openmpi` as it is easier to use SLURM and OpenMPI
to start multiple processes over multiple nodes. If you have your own launcher, you can fall back
launch_from_slurm` or `colossalai.launch_from_openmpi` as it is easier to use SLURM and OpenMPI
to start multiple processes over multiple nodes. If you have your own launcher, you can fall back
to the default `colossalai.launch` function.

Loading…
Cancel
Save