You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Yuanheng Zhao b21aac5bae [Inference] Optimize and Refactor Inference Batching/Scheduling (#5367) ... * add kvcache manager funcs for batching * add batch bucket for batching * revise RunningList struct in handler * add kvcache/batch funcs for compatibility * use new batching methods * fix indexing bugs * revise abort logic * use cpu seq lengths/block tables * rm unused attr in Sequence * fix type conversion/default arg * add and revise pytests * revise pytests, rm unused tests * rm unused statements * fix pop finished indexing issue * fix: use index in batch when retrieving inputs/update seqs * use dict instead of odict in batch struct * arg type hinting * fix make compress * refine comments * fix: pop_n_seqs to pop the first n seqs * add check in request handler * remove redundant conversion * fix test for request handler * fix pop method in batch bucket * fix prefill adding		9 months ago
..
_C	…
_analyzer	[misc] update pre-commit and run all files (#4752)	1 year ago
accelerator	[accelerator] fixed npu api	10 months ago
amp	[npu] change device to accelerator api (#5239)	11 months ago
auto_parallel	[npu] change device to accelerator api (#5239)	11 months ago
autochunk	[misc] update pre-commit and run all files (#4752)	1 year ago
booster	Merge branch 'main' into sync/npu	10 months ago
checkpoint_io	[pipeline,shardformer] Fix p2p efficiency in pipeline, allow skipping loading weight not in weight_map when `strict=False`, fix llama flash attention forward, add flop estimation by megatron in llama benchmark (#5017)	1 year ago
cli	[bug] Fix the version check bug in colossalai run when generating the cmd. (#4713)	1 year ago
cluster	fix-test (#5210)	11 months ago
context	[moe] merge moe into main (#4978)	1 year ago
device	[npu] add npu support for hybrid plugin and llama (#5090)	1 year ago
fx	[misc] update pre-commit and run all files (#4752)	1 year ago
inference	[Inference] Optimize and Refactor Inference Batching/Scheduling (#5367)	9 months ago
interface	[lazy] support from_pretrained (#4801)	1 year ago
kernel	[Inference/opt] Fused KVCahce Memcopy (#5374)	10 months ago
lazy	[doc] add lazy init docs (#4808)	1 year ago
legacy	merge commit	10 months ago
logging	[misc] update pre-commit and run all files (#4752)	1 year ago
moe	Merge pull request #5310 from hpcaitech/feature/npu	10 months ago
nn	[feat] refactored extension module (#5298)	10 months ago
pipeline	[feat] refactored extension module (#5298)	10 months ago
shardformer	fix typo change dosen't to doesn't (#5308)	10 months ago
tensor	fix some typo (#5307)	10 months ago
testing	[npu] change device to accelerator api (#5239)	11 months ago
utils	Merge pull request #5310 from hpcaitech/feature/npu	10 months ago
zero	Merge pull request #5310 from hpcaitech/feature/npu	10 months ago
__init__.py	[accelerator] init the accelerator module (#5129)	1 year ago
initialize.py	[npu] change device to accelerator api (#5239)	11 months ago