Yuanheng Zhao 2226c6836c [feature] add KV cache manager for llama & bloom inference (#4495) ... * add kv cache memory manager * add stateinfo during inference * format * format * rename file * add kv cache test * revise on BatchInferState * file dir change		2023-08-24 16:44:14 +08:00
..
_utils.py	[infer] Infer/llama demo (#4503)	2023-08-24 15:42:41 +08:00
test_kvcache_manager.py	[feature] add KV cache manager for llama & bloom inference (#4495)	2023-08-24 16:44:14 +08:00
test_llama_infer.py	[infer] Infer/llama demo (#4503)	2023-08-24 15:42:41 +08:00