3257 Commits (feat/online-serving)

 

All Branches   

Search

Author	SHA1	Message	Date
Hongxin Liu	956b561b54	[moe] fix mixtral forward default value (#5329)	10 months ago
Hongxin Liu	b60be18dcc	[moe] fix mixtral checkpoint io (#5314)	10 months ago
Hongxin Liu	da39d21b71	[moe] support mixtral (#5309) ... * [moe] add mixtral block for single expert * [moe] mixtral block fwd support uneven ep * [moe] mixtral block bwd support uneven ep * [moe] add mixtral moe layer * [moe] simplify replace * [meo] support save sharded mixtral * [meo] support load sharded mixtral * [meo] support save sharded optim * [meo] integrate moe manager into plug * [meo] fix optimizer load * [meo] fix mixtral layer	10 months ago
Hongxin Liu	c904d2ae99	[moe] update capacity computing (#5253) ... * [moe] top2 allow uneven input * [moe] update capacity computing * [moe] remove debug info * [moe] update capacity computing * [moe] update capacity computing	10 months ago
Xuanlei Zhao	7d8e0338a4	[moe] init mixtral impl	10 months ago
Jianghai	1f8c7e7046	[Inference] User Experience: update the logic of default tokenizer and generation config. (#5337) ... * add * fix * fix * pause * fix * fix pytest * align * fix * license * fix * fix * fix readme * fix some bugs * remove tokenizer config	10 months ago
yuehuayingxueluo	6fb4bcbb24	[Inference/opt] Fused KVCahce Memcopy (#5374) ... * fused kv memcopy * add TODO in test_kvcache_copy.py	10 months ago
Frank Lee	58740b5f68	[inference] added inference template (#5375)	10 months ago
Frank Lee	8106ede07f	Revert "[Inference] Adapt to Fused rotary (#5348)" (#5373) ... This reverts commit 9f4ab2eb92.	10 months ago
Jianghai	9f4ab2eb92	[Inference] Adapt to Fused rotary (#5348) ... * revise rotary embedding * remove useless print * adapt * fix * add * fix * modeling * fix * fix * fix	10 months ago
yuehuayingxueluo	35382a7fbf	[Inference]Fused the gate and up proj in mlp，and optimized the autograd process. (#5365) ... * fused the gate and up proj in mlp * fix code styles * opt auto_grad * rollback test_inference_engine.py * modifications based on the review feedback. * fix bugs in flash attn * Change reshape to view * fix test_rmsnorm_triton.py	10 months ago
Hongxin Liu	084c91246c	[llama] fix memory issue (#5371) ... * [llama] fix memory issue * [llama] add comment	10 months ago
Yuanheng Zhao	1dedb57747	[Fix/Infer] Remove unused deps and revise requirements (#5341) ... * remove flash-attn dep * rm padding llama * revise infer requirements * move requirements out of module	10 months ago
Hongxin Liu	c53ddda88f	[lr-scheduler] fix load state dict and add test (#5369)	10 months ago
Hongxin Liu	eb4f2d90f9	[llama] polish training script and fix optim ckpt (#5368)	10 months ago
Camille Zhong	a5756a8720	[eval] update llama npu eval (#5366)	10 months ago
Camille Zhong	44ca61a22b	[llama] fix neftune & pbar with start_step (#5364)	10 months ago
Hongxin Liu	a4cec1715b	[llama] add flash attn patch for npu (#5362)	10 months ago
Hongxin Liu	73f9f23fc6	[llama] update training script (#5360) ... * [llama] update training script * [doc] polish docstr	10 months ago
Hongxin Liu	6c0fa7b9a8	[llama] fix dataloader for hybrid parallel (#5358) ... * [plugin] refactor prepare dataloader * [plugin] update train script	10 months ago
Hongxin Liu	2dd01e3a14	[gemini] fix param op hook when output is tuple (#5355) ... * [gemini] fix param op hook when output is tuple * [gemini] fix param op hook	10 months ago
yuehuayingxueluo	631862f339	[Inference]Optimize generation process of inference engine (#5356) ... * opt inference engine * fix run_benchmark.sh * fix generate in engine.py * rollback tesh_inference_engine.py	10 months ago
yuehuayingxueluo	21ad4a27f9	[Inference/opt]Optimize the mid tensor of RMS Norm (#5350) ... * opt rms_norm * fix bugs in rms_layernorm	10 months ago
Wenhao Chen	1c790c0877	[fix] remove unnecessary dp_size assert (#5351) ... * fix: remove unnecessary assert * test: add more 3d plugin tests * fix: add warning	10 months ago
Frank Lee	027aa1043f	[doc] updated inference readme (#5343)	10 months ago
Frank Lee	e76acbb076	[inference] moved ops tests to test_infer (#5354)	10 months ago
Frank Lee	db1a763307	[inference] removed redundancy init_batch (#5353)	10 months ago
Hongxin Liu	ffffc32dc7	[checkpointio] fix gemini and hybrid parallel optim checkpoint (#5347) ... * [checkpointio] fix hybrid parallel optim checkpoint * [extension] fix cuda extension * [checkpointio] fix gemini optimizer checkpoint * polish code	10 months ago
yuehuayingxueluo	249644c23b	[Inference]Repalce Attention layer and MLP layer by shardformer to optimize the weight transpose operation，add fused_qkv and fused linear_add (#5340) ... * add fused qkv * replace attn and mlp by shardformer * fix bugs in mlp * add docstrings * fix test_inference_engine.py * add optimize unbind * add fused_addmm * rm squeeze(1) * refactor codes * fix ci bugs * rename ShardFormerLlamaMLP and ShardFormerLlamaAttention * Removed the dependency on LlamaFlashAttention2 * rollback test_inference_engine.py	10 months ago
Frank Lee	f8e456d202	[inference] simplified config verification (#5346) ... * [inference] simplified config verification * polish * polish	10 months ago
YeAnbang	c5239840e6	[Chat] fix sft loss nan (#5345) ... * fix script * fix script * fix chat nan * fix chat nan	10 months ago
Frank Lee	abd8e77ad8	[extension] fixed exception catch (#5342)	10 months ago
Jianghai	df0aa49585	[Inference] Kernel Fusion, fused copy kv cache into rotary embedding (#5336) ... * revise rotary embedding * remove useless print * adapt	10 months ago
Frank Lee	1336838a91	Merge pull request #5339 from FrankLeeeee/sync/merge-main ... Sync/merge main	10 months ago
FrankLeeeee	c565519913	merge commit	10 months ago
Yuanheng Zhao	5f98a9d68a	[Infer] Optimize Blocked KVCache And Kernels Using It (#5325) ... * revise shape of kvcache (context attn kernel) * revise shape of kvcache (flash decoding kernel) * revise shape of kvcache (kvcache copy) and attn func * init of kvcache in kvcache manager * revise llama modeling * revise block size retrieval * use torch for rms_norm benchmarking * revise block size retrieval	10 months ago
yuehuayingxueluo	e8f0642f28	[Inference]Add Nopadding Llama Modeling (#5327) ... * add nopadding llama modeling * add nopadding_llama.py * rm unused codes * fix bugs in test_xine_copy.py * fix code style	10 months ago
digger yu	71321a07cf	fix typo change dosen't to doesn't (#5308)	10 months ago
digger yu	6a3086a505	fix typo under extensions/ (#5330)	10 months ago
Frank Lee	febed23288	[doc] added docs for extensions (#5324) ... * [doc] added docs for extensions * polish * polish	10 months ago
flybird11111	388179f966	[tests] fix t5 test. (#5322) ... * [ci] fix shardformer tests. (#5255) * fix ci fix * revert: revert p2p * feat: add enable_metadata_cache option * revert: enable t5 tests --------- Co-authored-by: Wenhao Chen <cwher@outlook.com> * fix t5 test --------- Co-authored-by: Wenhao Chen <cwher@outlook.com>	10 months ago
Jianghai	c7c104cb7c	[DOC] Update inference readme (#5280) ... * add readme * add readme * 1 * update engine * finish readme * add readme	10 months ago
Frank Lee	a6709afe66	Merge pull request #5321 from FrankLeeeee/hotfix/accelerator-api ... [accelerator] fixed npu api	10 months ago
FrankLeeeee	087d0cb1fc	[accelerator] fixed npu api	10 months ago
Frank Lee	8823cc4831	Merge pull request #5310 from hpcaitech/feature/npu ... Feature/npu	10 months ago
Frank Lee	73f4dc578e	[workflow] updated CI image (#5318)	10 months ago
Jianghai	1f8a75d470	[Inference] Update rms norm kernel, benchmark with vLLM (#5315) ... * add * xi * del * del * fix	10 months ago
Jianghai	7ddd8b37f0	fix (#5311)	10 months ago
yuehuayingxueluo	4f28cb43c0	[inference]Optimize the usage of the mid tensors space in flash attn (#5304) ... * opt flash attn * opt tmp tensor * fix benchmark_llama * fix code style * fix None logic for output tensor * fix adapted to get_xine_cache * add comment * fix ci bugs * fix some codes * rm duplicated codes * rm duplicated codes * fix code style * add _get_dtype in config.py	10 months ago
Frank Lee	7cfed5f076	[feat] refactored extension module (#5298) ... * [feat] refactored extension module * polish * polish * polish * polish * polish * polish * polish * polish * polish * polish	10 months ago