Searching for Accuracy vs. Efficiency Trade-off Solution in Deep Differentiable Logic Gate Networks

Abstract

Recent research on Deep Differentiable Logic Gate Networks (DLNs) aim to shift machine learning from traditional weight based models to weightless networks that learn the best logical operators using gradient-based optimization. While DLNs can provide much faster inference and require less memory together with an interpretability of the results, they still face issues such as vanishing gradients in deeper networks and high computational cost during training. We compare several DLN configurations to find the best trade-off between accuracy and efficiency evaluating baseline DLNs against versions that use Residual Initialization (RI) at identity gates (denoted as $\mathbf{D L N}_{\alpha * *}$ where $\alpha \in\{\mathbf{Z}, \mathbf{M}, \mathbf{L}\}$, representing Zadeh, Menger, and Łukasiewicz T-norm respectively) and a mirror gate architecture (denoted as DLN ${ }_{\alpha * *}^{m}$). We focus on the T-norm choices and on two architecture modifications. Results on benchmark datasets show that $\mathbf{D L N}_{\alpha}^{* *}$ models greatly reduce vanishing gradients and improve accuracy in deeper networks.