mirror of https://github.com/hpcaitech/ColossalAI
b21aac5bae
* add kvcache manager funcs for batching * add batch bucket for batching * revise RunningList struct in handler * add kvcache/batch funcs for compatibility * use new batching methods * fix indexing bugs * revise abort logic * use cpu seq lengths/block tables * rm unused attr in Sequence * fix type conversion/default arg * add and revise pytests * revise pytests, rm unused tests * rm unused statements * fix pop finished indexing issue * fix: use index in batch when retrieving inputs/update seqs * use dict instead of odict in batch struct * arg type hinting * fix make compress * refine comments * fix: pop_n_seqs to pop the first n seqs * add check in request handler * remove redundant conversion * fix test for request handler * fix pop method in batch bucket * fix prefill adding |
||
|---|---|---|
| .. | ||
| kit | ||
| test_analyzer | ||
| test_auto_parallel | ||
| test_autochunk | ||
| test_booster | ||
| test_checkpoint_io | ||
| test_cluster | ||
| test_config | ||
| test_device | ||
| test_fx | ||
| test_gptq | ||
| test_infer | ||
| test_lazy | ||
| test_legacy | ||
| test_moe | ||
| test_optimizer | ||
| test_pipeline | ||
| test_shardformer | ||
| test_smoothquant | ||
| test_tensor | ||
| test_utils | ||
| test_zero | ||
| __init__.py | ||