ColossalAI/colossalai/inference/kv_cache/__init__.py

from .block_cache import CacheBlock
from .kvcache_manager import KVCacheManager, RPCKVCacheManager

__all__ = ["CacheBlock", "KVCacheManager", "RPCKVCacheManager"]
[Inference] Add CacheBlock and KV-Cache Manager (#5156) * [Inference] Add KVCache Manager * function refactored * add test for KVCache Manager * add attr beam width * Revise alloc func in CacheManager * Fix docs and pytests * add tp slicing for head number * optimize shapes of tensors used as physical cache * Apply using InferenceConfig on KVCacheManager * rm duplicate config file * Optimize cache allocation: use contiguous cache * Fix config in pytest (and config) 2023-12-11 02:56:18 +00:00			`from .block_cache import CacheBlock`
[Feat]Inference RPC Server Support (#5705) * rpc support source * kv cache logical/physical disaggregation * sampler refactor * colossalai launch built in * Unitest * Rpyc support --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> 2024-05-14 02:00:55 +00:00			`from .kvcache_manager import KVCacheManager, RPCKVCacheManager`
[Inference] Add CacheBlock and KV-Cache Manager (#5156) * [Inference] Add KVCache Manager * function refactored * add test for KVCache Manager * add attr beam width * Revise alloc func in CacheManager * Fix docs and pytests * add tp slicing for head number * optimize shapes of tensors used as physical cache * Apply using InferenceConfig on KVCacheManager * rm duplicate config file * Optimize cache allocation: use contiguous cache * Fix config in pytest (and config) 2023-12-11 02:56:18 +00:00
[Feat]Inference RPC Server Support (#5705) * rpc support source * kv cache logical/physical disaggregation * sampler refactor * colossalai launch built in * Unitest * Rpyc support --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> 2024-05-14 02:00:55 +00:00			`__all__ = ["CacheBlock", "KVCacheManager", "RPCKVCacheManager"]`