# Colossal-Infer
## Introduction
Colossal-Infer is a library for inference of LLMs and MLMs. It is built on top of Colossal AI.

## Structures
### Overview
The main design will be released later on.
## Roadmap
- [] design of structures
- [] Core components
    - [] engine
    - [] request handler
    - [] kv cache manager
    - [] modeling
    - [] custom layers
    - [] online server
- [] supported models
    - [] llama2