mirror of https://github.com/hpcaitech/ColossalAI
![]() |
||
---|---|---|
.. | ||
core | ||
kv_cache | ||
modeling | ||
__init__.py | ||
config.py | ||
logit_processors.py | ||
readme.md | ||
sampler.py | ||
struct.py |
readme.md
Colossal-Infer
Introduction
Colossal-Infer is a library for inference of LLMs and MLMs. It is built on top of Colossal AI.
Structures
Overview
The main design will be released later on.
Roadmap
- [] design of structures
- [] Core components
- [] engine
- [] request handler
- [] kv cache manager
- [] modeling
- [] custom layers
- [] online server
- [] supported models
- [] llama2