Lightweight Model Inference Techniques in Edge Computing Environments: Review

Abstract

Computation methods of artificial intelligence are gradually shifting from cloud computing to edge computing and on-device machine learning (ODML). How to contribute an effective machine learning model in the resource-limited environment, has become a significant and rapidly evolving research field. The training and inference of deep learning model used to be performed on the cloud high-performance computing clusters. There are many problems with uploading data to cloud, for example, high latency, round-trip latency, security issues, and a lack of privacy guarantees, and in this case, people cannot make real-time decisions. So, using edge devices to process tasks can significantly decrease the cost of transmission. The need of low latency, quick response, privacy protection and high adaptability has become the drive force of this change. This report aims to provide a comprehensive overview of lightweight model inference technologies in edge computing environments, mainly targeting low performance devices, such as mobile phones, intelligent equipment in vehicle, VR/AR headsets and Internet of Things (IoT). This paper introduces efficient learning and inference on edge devices from four aspects: 1) the definition of core terminology and concrete application environment; 2) the core technology of model compression, neural networks and knowledge distillation, which is used to deal with the tasks in the resource-limited environment; 3) the standards for evaluation of time/space complexity; 4) the challenge and opportunity which people face currently and in future.