Janus: Realizing Practical Operator Parallelism for Latency-Sensitive DNN Inference on GPUs
Janus is proposed, a novel resource- and latency-aware operator scheduling framework to enable efficient and practical operator parallelism for DNN inference on GPUs and introduces an effective stream allocation mechanism that fully incorporates operator resource constraints and latency heterogeneity.