A comparative study of four widely used approaches for machine vision inference in industrial settings: plain PyTorch, ONNX Runtime, OpenVINO, OpenVINO, and TensorRT, which shows that OpenVINO achieves the lowest inference time on CPUs, while TensorRT achieves the lowest inference time on GPUs.
Abstract
Edge deployment is often the preferred solution for industrial machine vision systems when low latency, data security, or limited connectivity are critical requirements. Several frameworks are available to optimise inference on edge devices; however, relatively few studies have systematically compared their inference-time performance under industrial deployment conditions. In this work, we present a comparative study of four widely used approaches for machine vision inference in industrial settings: plain PyTorch, ONNX Runtime, OpenVINO, and TensorRT. The evaluation focuses on inference time, covers several CPU- and GPU-based hardware platforms, and includes both conventional convolutional neural networks and a transformer-based vision model. For the hardware platforms and models evaluated, the results show that OpenVINO achieves the lowest inference time on CPUs, while TensorRT achieves the lowest inference time on GPUs. However, TensorRT does not outperform plain PyTorch for the transformer-based model considered in this study.
This paper presents a comparative evaluation of the runtime characteristics of three deep learning model architectures for both image classification and object detection, executed across three programming languages and three inference frameworks. In addition to standard FP32 execution, the study also examined reduced-precision inference using FP16 across all frameworks and INT8 quantization for TensorRT on the embedded platform. The evaluation analyzes how the choice of software stack and hardware platform affects end-to-end runtime, including preprocessing, inference, and postprocessing under varying input resolutions and batch sizes. Experiments were conducted on a desktop workstation and an embedded edge device to reflect realistic deployment scenarios. The results show consistent performance differences between frameworks and languages: TensorRT achieves the lowest inference latency across all tested models and precisions, whereas PyTorch and ONNX (Open Neural Network Exchange) Runtime exhibit higher runtime. The programming language also influences runtime, with C++ generally outperforming Python and Rust. Increasing the model size, batch size, and input resolution leads to higher latency, memory usage, and power consumption, although the relative ordering between frameworks remains unchanged. Reducing precision leads to reduced latency and GPU memory usage. Preprocessing and postprocessing contribute only a small fraction of the total runtime for classification but represent a substantial portion for object detection, especially at higher resolutions or reduced precision. Overall, the findings offer indicative guidance for choosing combinations of model architectures, frameworks, languages, and precision modes for deployment in resource-constrained hardware.
Richard Szegedi, Yannick Bukschat, Marcus Vetter· IEEE Access· 0 citations
Image classification is a major area in computer vision, driven by rapid advances in deep learning. Over the last decade, convolutional neural networks (CNNs) and their variants have achieved high performance in applications such as medical diagnosis, autonomous driving, industrial inspection, remote sensing, and biometrics. However, choosing the right model remains challenging due to trade-offs between accuracy, computational cost, efficiency, and robustness. This paper presents a comparative study of different deep learning architectures, including classical CNNs, deep hierarchical models, residual and dense networks, and compound-scaled architectures. Using a common evaluation framework and standard datasets, the study analyzes performance based on key design factors such as depth, width, receptive field, skip connections, and normalization. Theoretical concepts like convolution operations, residual learning, and optimization are also discussed.The results show that deeper networks provide better representation, while residual connections and compound scaling improve training stability and efficiency. Lightweight models perform well in resource-limited and real-time environments. Overall, the study offers practical guidance for selecting suitable architectures and highlights future research areas such as neural architecture search, self-supervised learning, and efficient model deployment.
Riyaz Mohammed· International Journal of App...· 0 citations
The findings demonstrate that introducing prior structure, through strategic sparse supervision, pre-trained multimodal knowledge, or transferable representations, consistently compensates for limited labeled data across diverse vision tasks.
Edge-deployed vision systems in target recognition, surveillance, autonomous vehicles, and drone domains require hierarchical inference pipelines where a detection model identifies objects of interest and downstream classifiers provide fine-grained attribute analysis. Running all models on the GPU creates a serial bottleneck that limits real-time throughput as pipeline stages grow. Modern edge SoCs pair GPUs with dedicated neural accelerators (NPUs, DLAs) capable of concurrent execution, yet deploying custom models on these accelerators remains impractical due to strict operator constraints, quantization incompatibilities, and an undocumented end-to-end pipeline. We target NVIDIA Jetson DLA cores as the representative platform. We present a five-step methodology for zero GPU fallback DLA INT8 deployment of classification backbones, comprising architecture adaptation, manual dynamic range workaround to rescue TensorRT's implicit quantization (recovering 94.0% accuracy from implicit quantization's 75%) for rapid pipeline validation before explicit quantization, quantization-aware training, ONNX graph surgery for DLA compilation, and a concurrent GPU-detection/DLA-classification inference pipeline. We document nine engineering constraints with root-cause analysis and generalizable solutions. Validation on a dual-head person attribute classifier running on DLA alongside a GPU object detector on a Jetson Orin NX demonstrates near-zero pipeline overhead (12.5 vs. 13.3~FPS detector-only at 1080p), with dual-DLA scaling at no additional cost. The methodology is backbone-agnostic and generalizes to any detection-classification edge pipeline.
A comparative analysis of dynamic stochastic computing using a convolutional neural network for MNIST digit classification indicates that dynamic stochastic computing can serve as an efficient alternative for the design and implementation of neural-network accelerators.
G. Rajashekhar, P. Muralidhar· International Journal of Wav...· 0 citations
The increasing energy cost of training deep vision models has made computational efficiency an important objective in modern deep learning. Although tensor caching, automatic mixed precision (AMP), gradient accumulation, $\ell _{1}$ sparsity regularization, learning-rate scheduling, pruning, and quantization are established techniques, their individual and cumulative accuracy–energy effects are difficult to interpret when several factors are modified simultaneously. This paper presents E2AM, a controlled, hardware-measured framework for evaluating five training-time interventions—tensor caching, AMP, gradient accumulation, $\ell _{1}$ sparsity regularization, and cosine learning-rate scheduling—together with an energy-aware early-stopping rule. The interventions are evaluated individually and cumulatively across ResNet-50, EfficientNetV2-S, MobileViTv2, and ConvNeXtV2-Tiny using CIFAR-10, CIFAR-100, and Tiny-ImageNet. All models are trained from random initialization under a common maximum budget of 50 epochs, and GPU-board energy is integrated from hardware power telemetry over 180 logged runs. Relative to the all-off reference configuration, the complete E2AM configuration reduces measured training energy by 30–63% across the evaluated architecture–dataset pairs and achieves higher observed best validation accuracy for 11 of the 12 pairs within the fixed training budget. ConvNeXtV2-Tiny on CIFAR-10 provides a lower-energy near-iso-accuracy operating point, with an accuracy difference of −0.08 percentage points. Because each configuration is evaluated using a single seed, the reported accuracy differences are interpreted as descriptive operating-point comparisons rather than inferential estimates. The observed responses also differ between the SGD-trained architectures and the AdamW-trained ConvNeXtV2-Tiny configuration. Because optimizer and architecture are not independently varied, this behavior is interpreted as architecture–optimizer-configuration dependence rather than as an isolated causal optimizer effect. The deployment analysis further reports post-optimization accuracy, serialized artifact size, latency, and throughput. INT8 quantization provides an approximately fourfold reduction in serialized artifact size for most graph-traceable configurations, whereas one configuration exhibits substantial accuracy degradation. Therefore, E2AM contributes a controlled evaluation methodology for characterizing training and deployment efficiency effects rather than proposing novel constituent optimization techniques.