mirror of https://github.com/hpcaitech/ColossalAI
19 lines
426 B
Markdown
19 lines
426 B
Markdown
# Colossal-Infer
|
|
## Introduction
|
|
Colossal-Infer is a library for inference of LLMs and MLMs. It is built on top of Colossal AI.
|
|
|
|
## Structures
|
|
### Overview
|
|
The main design will be released later on.
|
|
## Roadmap
|
|
- [] design of structures
|
|
- [] Core components
|
|
- [] engine
|
|
- [] request handler
|
|
- [] kv cache manager
|
|
- [] modeling
|
|
- [] custom layers
|
|
- [] online server
|
|
- [] supported models
|
|
- [] llama2
|