A Comparative Study of Neural Network Performance on the NVIDIA Jetson Orin Nano
This paper presents a comparative evaluation of the runtime characteristics of three deep learning model architectures for both image classification and object detection, executed across three programming languages and three inference frameworks. In addition to standard FP32 execution, the study also examined reduced-precision inference using FP16 across all frameworks and INT8 quantization for TensorRT on the embedded platform. The evaluation analyzes how the choice of software stack and hardware platform affects end-to-end runtime, including preprocessing, inference, and postprocessing under varying input resolutions and batch sizes. Experiments were conducted on a desktop workstation and an embedded edge device to reflect realistic deployment scenarios. The results show consistent performance differences between frameworks and languages: TensorRT achieves the lowest inference latency across all tested models and precisions, whereas PyTorch and ONNX (Open Neural Network Exchange) Runtime exhibit higher runtime. The programming language also influences runtime, with C++ generally outperforming Python and Rust. Increasing the model size, batch size, and input resolution leads to higher latency, memory usage, and power consumption, although the relative ordering between frameworks remains unchanged. Reducing precision leads to reduced latency and GPU memory usage. Preprocessing and postprocessing contribute only a small fraction of the total runtime for classification but represent a substantial portion for object detection, especially at higher resolutions or reduced precision. Overall, the findings offer indicative guidance for choosing combinations of model architectures, frameworks, languages, and precision modes for deployment in resource-constrained hardware.