3189 Commits (feat/speculative-decoding)

 

All Branches   

Search

Author	SHA1	Message	Date
Yaozheng Fang	5ae9099f92	[kernel] Add RMSLayerNorm triton kernel (#5262) ... * add layerrmsnorm triton kernel * add layerrmsnorm kernel * modify the atol and rtol in test file * Remove the logics of mean computations, and update the name of ther kernel functions and files * add benchmark of rms norm	10 months ago
Zhongkai Zhao	5d9a0ae75b	[hotfix] Fix ShardFormer test execution path when using sequence parallelism (#5230)	10 months ago
yuehuayingxueluo	86b63f720c	[Inference]Adapted to the triton attn kernels (#5264) ... * adapted to the triton attn kernels * fix pad input * adapted to copy_kv_to_blocked_cache * fix ci test * update kv memcpy * remove print	10 months ago
flybird11111	46e091651b	[shardformer] hybridparallelplugin support gradients accumulation. (#5246) ... * support gradients acc fix fix fix fix fix fix fix fix fix fix fix fix fix * fix fix * fix fix fix	10 months ago
flybird11111	2a0558d8ec	[ci] fix test_hybrid_parallel_plugin_checkpoint_io.py (#5276) ... * fix ci fix * fix test * revert: revert p2p * feat: add enable_metadata_cache option * revert: enable t5 tests * fix --------- Co-authored-by: Wenhao Chen <cwher@outlook.com>	10 months ago
Frank Lee	d69cd2eb89	[workflow] fixed oom tests (#5275) ... * [workflow] fixed oom tests * polish * polish * polish	10 months ago
Yuanheng Zhao	0f2b46a41c	[kernel] Revise KVCache copy triton kernel API (#5273) ... * [kernel/fix] revise kvcache copy kernel api * fix benchmark	10 months ago
Frank Lee	04244aaaf1	[workflow] fixed incomplete bash command (#5272)	10 months ago
Jianghai	d8db500efc	[Inference] Fix request handler and add recycle logic (#5260) ... * fix request handler * fix comment	11 months ago
Frank Lee	c597678da4	[doc] updated inference readme (#5269)	11 months ago
Yuanheng Zhao	fa85e02b3b	[kernel] Add KV cache copy kernel during decoding (#5261) ... * add kv copy triton kernel during decoding stage * add pytest and fix kernel * fix test utilities * revise kernel config * add benchmark for kvcache copy	11 months ago
Wenhao Chen	ef4f0ee854	[hotfix]: add pp sanity check and fix mbs arg (#5268) ... * fix: fix misleading mbs arg * feat: add pp sanity check * fix: fix 1f1b sanity check	11 months ago
FrankLeeeee	1ded7e81ef	[git] fixed rebased files	11 months ago
Yuanheng Zhao	1513f20f4d	[kernel] Add flash decoding triton kernel for blocked kv cache (#5249) ... * add flash decoding unpad triton kernel * rename flash decoding kernel * add kernel testing (draft) * revise pytest * support kv group (GQA) * (trivial) fix api and pytest * (trivial) func renaming * (trivial) func/file renaming * refactor pytest for attention * (trivial) format and consistent vars of context/decode attn * (trivial) remove test redundancy	11 months ago
Jianghai	fded91d049	[Inference] Kernel: no pad rotary embedding (#5252) ... * fix bugs * comment * use more accurate atol * fix	11 months ago
yuehuayingxueluo	d40eb26029	fix bugs in request_handler.py and engine.py	11 months ago
yuehuayingxueluo	10e3c9f923	rm torch.cuda.synchronize	11 months ago
yuehuayingxueluo	fab294c7f4	fix CI bugs	11 months ago
yuehuayingxueluo	2a73e828eb	fix bugs related to processing padding mask	11 months ago
Jianghai	e545a871b8	[Hotfix] Fix accuracy and align attention method api with Triton kernel (#5229) ... * fix accuracy * alignment in attention * fix attention * fix * fix bugs * fix bugs * fix bugs	11 months ago
yuehuayingxueluo	fa4fbdbffb	adapted to pad_context_forward	11 months ago
yuehuayingxueluo	47e53eaa1c	fix bugs in attention.py and request_handler.py	11 months ago
Jianghai	bfd9b1b494	[Inference] Pytorch Attention func, pad&nopad input support (#5219) ... * add attn * add attention test * fix attn forward * fix decoding	11 months ago
yuehuayingxueluo	3ad1f3b78b	fix beam_width	11 months ago
yuehuayingxueluo	b2eb9cd186	Fixed a typo	11 months ago
yuehuayingxueluo	bbfebfb9fc	fix bugs in sampler	11 months ago
yuehuayingxueluo	02c1bf8b2a	add context_attention_unpadded	11 months ago
Yuanheng Zhao	07b5283b6a	[kernel] Add triton kernel for context attention (FAv2) without padding (#5192) ... * add context attn unpadded triton kernel * test compatibility * kv cache copy (testing) * fix k/v cache copy * fix kv cache copy and test * fix boundary of block ptrs * add support for GQA/MQA and testing * fix import statement --------- Co-authored-by: Round Heng <yuanhengzhao@Rounds-MacBook-Pro.local>	11 months ago
yuehuayingxueluo	4df8876fca	Fixed a writing error	11 months ago
yuehuayingxueluo	9489dc64d8	precision alignment	11 months ago
yuehuayingxueluo	62968588d1	fix bugs in request_handler	11 months ago
yuehuayingxueluo	62fd08ee44	Fixed a bug in the inference frame	11 months ago
yuehuayingxueluo	86853a37d5	Add padding llama model	11 months ago
Jianghai	0e616462a7	[Inference] add logit processor and request handler (#5166) ... * add logit processor and request handler * add * add * add * fix * add search tokens and update func * finish request handler * add running list test * fix test * fix some bug * add * add * fix bugs * fix some bugs * fix bug * fix * fix * add copy fun * del useless attn * fix request status --------- Co-authored-by: CjhHa1 <cjh18671720497outlook.com>	11 months ago
yuehuayingxueluo	8daee26989	[Inference] Add the logic of the inference engine (#5173) ... * add infer_struct and infer_config * update codes * change InferConfig * Add hf_model_config to the engine * rm _get_hf_model_config * update codes * made adjustments according to the feedback from the reviewer. * update codes * add ci test for config and struct * Add the logic of the inference engine * update engine and test * Recover cache_manager.py * add logger * fix conflict * update codes * update codes * update model and tokenizer * fix add the logic about shardformer * change kvcache_manager docstring * add policy * fix ci bug in test_kvcache_manager.py * remove codes related o tokenizer and move model_policy * fix code style * add ordered_set to requirements-infer.txt * Delete extra empty lines * add ordered_set to requirements-test.txt	11 months ago
Jianghai	93aeacca34	[Inference]Update inference config and fix test (#5178) ... * unify the config setting * fix test * fix import * fix test * fix * fix * add logger * revise log info --------- Co-authored-by: CjhHa1 <cjh18671720497outlook.com>	11 months ago
Yuanheng Zhao	3de2e62299	[Inference] Add CacheBlock and KV-Cache Manager (#5156) ... * [Inference] Add KVCache Manager * function refactored * add test for KVCache Manager * add attr beam width * Revise alloc func in CacheManager * Fix docs and pytests * add tp slicing for head number * optimize shapes of tensors used as physical cache * Apply using InferenceConfig on KVCacheManager * rm duplicate config file * Optimize cache allocation: use contiguous cache * Fix config in pytest (and config)	11 months ago
yuehuayingxueluo	fab9b931d9	[Inference]Add BatchInferState, Sequence and InferConfig (#5149) ... * add infer_struct and infer_config * update codes * change InferConfig * Add hf_model_config to the engine * rm _get_hf_model_config * update codes * made adjustments according to the feedback from the reviewer. * update codes * add ci test for config and struct	11 months ago
Yuanheng Zhao	2bb92243d4	[Inference/NFC] Clean outdated inference tests and deprecated kernels (#5159) ... * [inference/nfc] remove outdated inference tests * remove outdated kernel tests * remove deprecated triton kernels * remove imports from deprecated kernels	11 months ago
Jianghai	56e75eeb06	[Inference] Add readme (roadmap) and fulfill request handler (#5147) ... * request handler * add readme --------- Co-authored-by: CjhHa1 <cjh18671720497outlook.com>	11 months ago
Jianghai	4cf4682e70	[Inference] First PR for rebuild colossal-infer (#5143) ... * add engine and scheduler * add dirs --------- Co-authored-by: CjhHa1 <cjh18671720497outlook.com>	11 months ago
binmakeswell	c174c4fc5f	[doc] fix doc typo (#5256) ... * [doc] fix annotation display * [doc] fix llama2 doc	11 months ago
flybird11111	e830ef917d	[ci] fix shardformer tests. (#5255) ... * fix ci fix * revert: revert p2p * feat: add enable_metadata_cache option * revert: enable t5 tests --------- Co-authored-by: Wenhao Chen <cwher@outlook.com>	11 months ago
digger yu	756c400ad2	fix typo in applications/ColossalEval/README.md (#5250)	11 months ago
Frank Lee	2b83418719	[ci] fixed ddp test (#5254) ... * [ci] fixed ddp test * polish	11 months ago
Frank Lee	d5eeeb1416	[ci] fixed booster test (#5251) ... * [ci] fixed booster test * [ci] fixed booster test * [ci] fixed booster test	11 months ago
Frank Lee	edf94a35c3	[workflow] fixed build CI (#5240) ... * [workflow] fixed build CI * polish * polish * polish * polish * polish	11 months ago
digger yu	41e52c1c6e	[doc] fix typo in Colossal-LLaMA-2/README.md (#5247)	11 months ago
Frank Lee	9102d655ab	[hotfix] removed unused flag (#5242)	11 months ago
Hongxin Liu	d202cc28c0	[npu] change device to accelerator api (#5239) ... * update accelerator * fix timer * fix amp * update * fix * update bug * add error raise * fix autocast * fix set device * remove doc accelerator * update doc * update doc * update doc * use nullcontext * update cpu * update null context * change time limit for example * udpate * update * update * update * [npu] polish accelerator code --------- Co-authored-by: Xuanlei Zhao <xuanlei.zhao@gmail.com> Co-authored-by: zxl <43881818+oahzxl@users.noreply.github.com>	11 months ago