Open access
Nov 2026
Chat-Fusion: An Efficient Batch Inference System for Cloud LLMs Serving
Chat-Fusion is built, an efficient redundancy-free batch inference system that improves the throughput of popular LLMs by 1.4 times compared to other state-of-the-art inference systems, such as ORCA and vLLM.
Jin-Xin Yu, Lei Wang, Meng-Di Wang et al.
· IEEE Transactions on Paralle... · 0 citations