2 Commits (b1172740743998ca08808e2ad4f93a8fc6cf3035)

Author	SHA1	Message	Date
Frank Lee	027aa1043f	[doc] updated inference readme (#5343)	2024-02-02 14:31:10 +08:00
Yuanheng Zhao	3de2e62299	[Inference] Add CacheBlock and KV-Cache Manager (#5156) ... * [Inference] Add KVCache Manager * function refactored * add test for KVCache Manager * add attr beam width * Revise alloc func in CacheManager * Fix docs and pytests * add tp slicing for head number * optimize shapes of tensors used as physical cache * Apply using InferenceConfig on KVCacheManager * rm duplicate config file * Optimize cache allocation: use contiguous cache * Fix config in pytest (and config)	2024-01-11 13:39:29 +00:00