mzCache: On-Device LLM Memory Management under Multitasking
On-device mobile Large Language Model (LLM) inference is gaining significant attention. However, mobile devices operate in highly dynamic multitasking environments where users frequently switch between applications. This creates memory pressure, forcing LLM memory (model weights and KV cache) to be evicted by the opera...
Hong Yu, Minsung Kim, Jongseok Park et al.
· 0 citations