Jianghai cf579ff46d [Inference] Dynamic Batching Inference, online and offline (#4953) ... * [inference] Dynamic Batching for Single and Multiple GPUs (#4831) * finish batch manager * 1 * first * fix * fix dynamic batching * llama infer * finish test * support different lengths generating * del prints * del prints * fix * fix bug --------- Co-authored-by: CjhHa1 <cjh18671720497outlook.com> * [inference] Async dynamic batching (#4894) * finish input and output logic * add generate * test forward * 1 * [inference]Re push async dynamic batching (#4901) * adapt to ray server * finish async * finish test * del test --------- Co-authored-by: yuehuayingxueluo <867460659@qq.com> * Revert "[inference]Re push async dynamic batching (#4901)" (#4905) This reverts commit fbf3c09e67. * Revert "[inference] Async dynamic batching (#4894)" This reverts commit fced140250. * Revert "[inference] Async dynamic batching (#4894)" (#4909) This reverts commit fced140250. * Add Ray Distributed Environment Init Scripts * support DynamicBatchManager base function * revert _set_tokenizer version * add driver async generate * add async test * fix bugs in test_ray_dist.py * add get_tokenizer.py * fix code style * fix bugs about No module named 'pydantic' in ci test * fix bugs in ci test * fix bugs in ci test * fix bugs in ci test * [infer]Add Ray Distributed Environment Init Scripts (#4911) * Revert "[inference] Async dynamic batching (#4894)" This reverts commit fced140250. * Add Ray Distributed Environment Init Scripts * support DynamicBatchManager base function * revert _set_tokenizer version * add driver async generate * add async test * fix bugs in test_ray_dist.py * add get_tokenizer.py * fix code style * fix bugs about No module named 'pydantic' in ci test * fix bugs in ci test * fix bugs in ci test * fix bugs in ci test * support dynamic batch for bloom model and is_running function * [Inference]Test for new Async engine (#4935) * infer engine * infer engine * test engine * test engine * new manager * change step * add * test * fix * fix * finish test * finish test * finish test * finish test * add license --------- Co-authored-by: yuehuayingxueluo <867460659@qq.com> * add assertion for config (#4947) * [Inference] Finish dynamic batching offline test (#4948) * test * fix test * fix quant * add default * fix * fix some bugs * fix some bugs * fix * fix bug * fix bugs * reset param --------- Co-authored-by: yuehuayingxueluo <867460659@qq.com> Co-authored-by: Cuiqing Li <lixx3527@gmail.com> Co-authored-by: CjhHa1 <cjh18671720497outlook.com>		2023-10-30 10:52:19 +08:00
..
__init__.py	rename chatglm to chatglm2 (#4484)	2023-08-22 14:13:31 +08:00
albert.py	[misc] update pre-commit and run all files (#4752)	2023-09-19 14:20:26 +08:00
bert.py	[test] merge old components to test to model zoo (#4945)	2023-10-20 10:35:08 +08:00
blip2.py	[test] merge old components to test to model zoo (#4945)	2023-10-20 10:35:08 +08:00
bloom.py	[test] merge old components to test to model zoo (#4945)	2023-10-20 10:35:08 +08:00
chatglm2.py	[test] merge old components to test to model zoo (#4945)	2023-10-20 10:35:08 +08:00
gpt.py	[test] merge old components to test to model zoo (#4945)	2023-10-20 10:35:08 +08:00
llama.py	[Inference] Dynamic Batching Inference, online and offline (#4953)	2023-10-30 10:52:19 +08:00
opt.py	[test] merge old components to test to model zoo (#4945)	2023-10-20 10:35:08 +08:00
sam.py	[test] merge old components to test to model zoo (#4945)	2023-10-20 10:35:08 +08:00
t5.py	[test] merge old components to test to model zoo (#4945)	2023-10-20 10:35:08 +08:00
vit.py	[test] merge old components to test to model zoo (#4945)	2023-10-20 10:35:08 +08:00
whisper.py	[test] merge old components to test to model zoo (#4945)	2023-10-20 10:35:08 +08:00