[example] improved the clarity yof the example readme (#2427)

* [example] improved the clarity yof the example readme

* polish workflow

* polish workflow

* polish workflow

* polish workflow

* polish workflow

* polish workflow
pull/2434/head
Frank Lee 2023-01-11 10:46:32 +08:00 committed by GitHub
parent 21256674e9
commit a3e5496156
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 33 additions and 20 deletions

View File

@ -39,7 +39,7 @@ jobs:
res=`python .github/workflows/scripts/example_checks/detect_changed_example.py --fileNameList $changedFileName` res=`python .github/workflows/scripts/example_checks/detect_changed_example.py --fileNameList $changedFileName`
echo "All changed examples are $res" echo "All changed examples are $res"
if [ "$x" = "[]" ]; then if [ "$res" = "[]" ]; then
echo "anyChanged=false" >> $GITHUB_OUTPUT echo "anyChanged=false" >> $GITHUB_OUTPUT
echo "matrix=null" >> $GITHUB_OUTPUT echo "matrix=null" >> $GITHUB_OUTPUT
else else
@ -54,7 +54,8 @@ jobs:
if: | if: |
github.event.pull_request.draft == false && github.event.pull_request.draft == false &&
github.base_ref == 'main' && github.base_ref == 'main' &&
github.event.pull_request.base.repo.full_name == 'hpcaitech/ColossalAI' && github.event_name == 'pull_request' github.event.pull_request.base.repo.full_name == 'hpcaitech/ColossalAI' && github.event_name == 'pull_request' &&
needs.detect-changed-example.outputs.anyChanged == 'true'
name: Test the changed example name: Test the changed example
needs: detect-changed-example needs: detect-changed-example
runs-on: [self-hosted, gpu] runs-on: [self-hosted, gpu]

View File

@ -1,28 +1,40 @@
## Examples folder document # Colossal-AI Examples
## Table of Contents ## Table of Contents
<ul>
<li><a href="#Example-folder-description">Example folder description</a> </li>
<li><a href="#Integrate-Your-Example-With-System-Testing">Integrate Your Example With System Testing</a> </li>
</ul>
## Example folder description - [Colossal-AI Examples](#colossal-ai-examples)
- [Table of Contents](#table-of-contents)
- [Overview](#overview)
- [Folder Structure](#folder-structure)
- [Integrate Your Example With Testing](#integrate-your-example-with-testing)
This folder provides several examples using colossalai. The images folder includes model like diffusion, dreambooth and vit. The language folder includes gpt, opt, palm and roberta. The tutorial folder is for concept illustration, such as auto-parallel, hybrid-parallel and so on. ## Overview
This folder provides several examples accelerated by Colossal-AI. The `tutorial` folder is for everyone to quickly try out the different features in Colossal-AI. Other folders such as `images` and `language` include a wide range of deep learning tasks and applications.
## Integrate Your Example With System Testing ## Folder Structure
For example code contributor, to meet the expectation and test your code automatically using github workflow function, here are several steps: ```text
└─ examples
└─ images
└─ vit
└─ test_ci.sh
└─ train.py
└─ README.md
└─ ...
└─ ...
```
## Integrate Your Example With Testing
- (must) Have a test_ci.sh file in the folder like shown below in 'File Structure Chart' Regular checks are important to ensure that all examples run without apparent bugs and stay compatible with the latest API.
- The dataset should be located in the company's machine and can be announced using environment variable and thus no need for a separate terminal command. Colossal-AI runs workflows to check for examples on a on-pull-request and weekly basis.
- The model parameters should be small to allow fast testing. When a new example is added or changed, the workflow will run the example to test whether it can run.
- File Structure Chart Moreover, Colossal-AI will run testing for examples every week.
└─examples Therefore, it is essential for the example contributors to know how to integrate your example with the testing workflow. Simply, you can follow the steps below.
└─images
└─vit 1. Create a script called `test_ci.sh` in your example folder
└─requirements.txt 2. Configure your testing parameters such as number steps, batch size in `test_ci.sh`, e.t.c. Keep these parameters small such that each example only takes several minutes.
└─test_ci.sh 3. Export your dataset path with the prefix `/data` and make sure you have a copy of the dataset in the `/data/scratch/examples-data` directory on the CI machine. Community contributors can contact us via slack to request for downloading the dataset on the CI machine.
4. Implement the logic such as dependency setup and example execution