ColossalAI

Commit Graph

Author	SHA1	Message	Date
Frank Lee	027aa1043f	[doc] updated inference readme (#5343 )	10 months ago
Yuanheng Zhao	3de2e62299	[Inference] Add CacheBlock and KV-Cache Manager (#5156 ) * [Inference] Add KVCache Manager * function refactored * add test for KVCache Manager * add attr beam width * Revise alloc func in CacheManager * Fix docs and pytests * add tp slicing for head number * optimize shapes of tensors used as physical cache * Apply using InferenceConfig on KVCacheManager * rm duplicate config file * Optimize cache allocation: use contiguous cache * Fix config in pytest (and config)	11 months ago

Author

SHA1

Message

Date

Frank Lee

027aa1043f

[doc] updated inference readme (#5343 )

Yuanheng Zhao

3de2e62299

[Inference] Add CacheBlock and KV-Cache Manager (#5156 )

* [Inference] Add KVCache Manager

* function refactored

* add test for KVCache Manager

* add attr beam width

* Revise alloc func in CacheManager

* Fix docs and pytests

* add tp slicing for head number

* optimize shapes of tensors used as physical cache

* Apply using InferenceConfig on KVCacheManager

* rm duplicate config file

* Optimize cache allocation: use contiguous cache

* Fix config in pytest (and config)

2 Commits (5d9a49483d98ccd4bebebbfd039162caceefe6bd)