You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Yuanheng Zhao b21aac5bae [Inference] Optimize and Refactor Inference Batching/Scheduling (#5367) ... * add kvcache manager funcs for batching * add batch bucket for batching * revise RunningList struct in handler * add kvcache/batch funcs for compatibility * use new batching methods * fix indexing bugs * revise abort logic * use cpu seq lengths/block tables * rm unused attr in Sequence * fix type conversion/default arg * add and revise pytests * revise pytests, rm unused tests * rm unused statements * fix pop finished indexing issue * fix: use index in batch when retrieving inputs/update seqs * use dict instead of odict in batch struct * arg type hinting * fix make compress * refine comments * fix: pop_n_seqs to pop the first n seqs * add check in request handler * remove redundant conversion * fix test for request handler * fix pop method in batch bucket * fix prefill adding		9 months ago
..
__init__.py	[doc] updated inference readme (#5343)	10 months ago
engine.py	[Inference] Optimize and Refactor Inference Batching/Scheduling (#5367)	9 months ago
request_handler.py	[Inference] Optimize and Refactor Inference Batching/Scheduling (#5367)	9 months ago