github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

[doc]Update doc for colossal-inference (#4989) 

* update doc

* Update README.md

---------

Co-authored-by: cuiqing.li <lixx336@gmail.com>
pull/4996/head
Cuiqing Li (李崔卿) 2023-10-31 10:48:07 +08:00 committed by GitHub
parent abe071b663
commit 4f0234f236
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
colossalai/inference/README.md
View File

@ -59,16 +59,14 @@ dependencies
pytorch= 1.13.1 (gpu) pytorch= 1.13.1 (gpu)
cuda>= 11.6 cuda>= 11.6
transformers= 4.30.2 transformers= 4.30.2
triton==2.0.0.dev20221202 triton
# for install vllm, please use this branch to install https://github.com/tiandiao123/vllm/tree/setup_branch # for install flash-attention
vllm
# for install flash-attention, please use commit hash: 67ae6fd74b4bc99c36b2ce524cf139c35663793c
flash-attention flash-attention
# install lightllm since we depend on lightllm triton kernels # install lightllm since we depend on lightllm triton kernels
git clone https://github.com/ModelTC/lightllm git clone https://github.com/ModelTC/lightllm
git checkout 28c1267cfca536b7b4f28e921e03de735b003039
cd lightllm cd lightllm
git checkout 28c1267cfca536b7b4f28e921e03de735b003039
pip3 install -e . pip3 install -e .
# also, install xformers from source: # also, install xformers from source:
@ -93,8 +91,8 @@ pip install -e .
# install lightllm # install lightllm
git clone https://github.com/ModelTC/lightllm git clone https://github.com/ModelTC/lightllm
git checkout 28c1267cfca536b7b4f28e921e03de735b003039
cd lightllm cd lightllm
git checkout 28c1267cfca536b7b4f28e921e03de735b003039
pip3 install -e . pip3 install -e .
# install xformers from source # install xformers from source