Jianghai
|
f47f2fbb24
|
[Inference] Fix API server, test and example (#5712)
* fix api server
* fix generation config
* fix api server
* fix comments
* fix infer hanging bug
* resolve comments, change backend to free port
|
2024-05-15 15:47:31 +08:00 |
|
Jianghai
|
de378cd2ab
|
[Inference] Finish Online Serving Test, add streaming output api, continuous batching test and example (#5432)
* finish online test and add examples
* fix test_contionus_batching
* fix some bugs
* fix bash
* fix
* fix inference
* finish revision
* fix typos
* revision
|
2024-05-08 15:20:52 +00:00 |
|
Jianghai
|
69cd7e069d
|
[Inference] ADD async and sync Api server using FastAPI (#5396)
* add api server
* fix
* add
* add completion service and fix bug
* add generation config
* revise shardformer
* fix bugs
* add docstrings and fix some bugs
* fix bugs and add choices for prompt template
|
2024-05-08 15:18:28 +00:00 |