ColossalAI

Commit Graph

Author	SHA1	Message	Date
Frank Lee	027aa1043f	[doc] updated inference readme (#5343 )	2024-02-02 14:31:10 +08:00
Yuanheng Zhao	3de2e62299	[Inference] Add CacheBlock and KV-Cache Manager (#5156 ) * [Inference] Add KVCache Manager * function refactored * add test for KVCache Manager * add attr beam width * Revise alloc func in CacheManager * Fix docs and pytests * add tp slicing for head number * optimize shapes of tensors used as physical cache * Apply using InferenceConfig on KVCacheManager * rm duplicate config file * Optimize cache allocation: use contiguous cache * Fix config in pytest (and config)	2024-01-11 13:39:29 +00:00

Author

SHA1

Message

Date

Frank Lee

027aa1043f

[doc] updated inference readme (#5343 )

2024-02-02 14:31:10 +08:00

Yuanheng Zhao

3de2e62299

[Inference] Add CacheBlock and KV-Cache Manager (#5156 )

* [Inference] Add KVCache Manager

* function refactored

* add test for KVCache Manager

* add attr beam width

* Revise alloc func in CacheManager

* Fix docs and pytests

* add tp slicing for head number

* optimize shapes of tensors used as physical cache

* Apply using InferenceConfig on KVCacheManager

* rm duplicate config file

* Optimize cache allocation: use contiguous cache

* Fix config in pytest (and config)

2024-01-11 13:39:29 +00:00

2 Commits (2b28b54ac6d19d33079d9117b9717fd2779f2b08)