Gecko is presented, designed to limit this additional risk while retaining a compact encrypted predictor, and formalizes ideal independence and information-preservation conditions as design guidance, then separately evaluate component-reuse extraction attacks.
Abstract
Private inference protects both user inputs and server models during neural network inference, but existing solutions remain too slow for practical deployment. This motivates recent efforts to run a public encoder, such as a pretrained backbone, outside the protection boundary and evaluate only a small private predictor cryptographically. While appealing for efficiency, this design is not inherently secure: naively offloading a public encoder may create a feature-space shortcut: an extraction adversary may learn the remaining private predictor's feature-to-output mapping more easily than the original model's input-to-output behavior. We present Gecko, designed to limit this additional risk while retaining a compact encrypted predictor. We leverage a frozen backbone that contributes hierarchical features, fixed Fastfood projections that compress them, and private feature gating that prepares them for prediction. We formalize ideal independence and information-preservation conditions as design guidance, then separately evaluate component-reuse extraction attacks. Across image and audio tasks, Gecko achieves 0.4-2.2 second inference with at most 10.8 MB communication and accuracy comparable to transfer-learning baselines. Under the evaluated attacks, reusing the offloaded public encoder provides no significant advantage to model-extraction adversaries. Source code and a demo are available at https://github.com/CassiniHuy/gecko-infer.
GradLock is introduced, a novel training-time injection attack that stealthily injects sensitive training data directly into the model parameters and employs dynamic gradient locking to prevent payload degradation during the optimization process.
Jin-Seong Kim, Han-Ju Lee, Seok-Won Hong et al.· 0 citations
Processing long, sensitive documents with machine-learning models requires efficient, privacy-preserving long-context inference. Prior private inference systems optimize or distribute encrypted Transformer attention, but its quadratic token-pair work remains the bottleneck as sequence length grows. Selective state-space models (SSMs) offer linear-time recurrence, yet direct encrypted implementation incurs linear multiplicative depth, sequence-wide state residency, or dense FHE-MPC conversion. We present Factorized Encrypted Scan-Contract (FESC), a hybrid FHE-MPC system for private long-context selective SSM inference. Its factorized scan-contract keeps input-dependent transitions compact across conversion boundaries, composes them without dense expansion, streams state chunks on demand, and contracts outputs before conversion. We demonstrate interface compatibility of the scan-contract implementation across invariant and selective SSM architectures. For our Mamba-2 instantiation, we design GPU-optimized CKKS kernels for linear computations, MPC protocols for SiLU, softplus, exponential, and RMSNorm, with approximation-aware fine-tuning. To our knowledge, FESC is the first private long-document inference system to complete native end-to-end execution at $L \geq 1{,}024$ on a single GPU. At $L = 2{,}048$, a 12-layer Mamba-base model completes inference in 77.3 minutes on one A100 GPU with a peak memory footprint of 32.7 GB, while maintaining near-plaintext accuracy on the evaluated long-document tasks.
Yufan Zhu, Chao Jin, Khin Mi Mi Aung et al.· 0 citations
SVIiT is described as communication-reduced relative to the evaluated MPC baselines rather than universally lightweight, and its present practical scope is primarily high-bandwidth LAN or provider-edge deployments.
Tingting Chen· ICST Transactions on Scalabl...· 0 citations
Applications such as personalized assistance and proprietary document analysis require large language models (LLMs) to generate outputs from private data. Yet powerful LLMs typically cannot be deployed on the resource-constrained devices where private data resides, and uploading private data to cloud-hosted LLMs exposes sensitive information. Recent work addresses this tension with a cloud-edge collaborative decoding paradigm, where private data are kept on the edge with a small language model (SLM) producing next-token distributions, which are fused with predictions from a cloud LLM operating solely on public data. In this paper, we systematically analyze the privacy risks of such a paradigm with a novel evaluation framework using constructed QA datasets, which show that such collaboration can expose substantial private-context information. To address such privacy leakage, we propose CoVeil, a defense mechanism which dynamically optimizes transmitted signals to suppress leakage during decoding time while preserving the collaborative quality. Extensive evaluations demonstrate that CoVeil consistently improves the privacy-utility trade-off over existing baselines by reducing data leakage by up to 87.2%, with minimal accuracy loss.
Ke-Jia Zhang, Tianyuan Zou, Zi-Xuan Gu et al.· 0 citations
We address the challenge of securely and efficiently outsourcing AI computations from a trusted but computationally weak client to an untrusted but powerful server, in the setting where the client holds both the input and the model, and the server must learn neither. We present MOSAIC, whose core is a novel matrix-multiplication masking protocol that scales to far larger matrices than prior work, enabling the safe outsourcing of modern workloads such as large transformer inference. By introducing small amounts of noise to the multiplication result and thereby relaxing correctness, MOSAIC achieves optimal asymptotic client overhead and concrete runtimes orders of magnitude faster than prior work. Its security reduces to the decisional LWE and LPN assumptions. Because this noise accumulates across the many layers of a transformer, a key technical challenge is bounding error growth; MOSAIC addresses this with an error-scaling mechanism based on random Hadamard rotations. On large 70B transformer models, MOSAIC's perplexity is comparable to popular quantization approaches and even matches full-precision BF16 inference on HumanEval. Finally, we present an end-to-end implementation showing how ideas like MOSAIC can promise a path towards large-scale confidential AI in modern data centers. Non-confidential inference is already distributed across phase (prefill/decode), layer, and time to maximize utilization of heterogeneous hardware, using RDMA-like networking to move activations, cached KV values, and weights across nodes. MOSAIC enables scaling of confidential compute by keeping the trusted computing base (TCB) small and outsourcing the bulk of the AI computation to untrusted accelerators.
J. Chiang, Sheila Zingg, Kari Kostiainen et al.· 2 citations
Encrypted inference under homomorphic encryption (HE) imposes stringent constraints on computation, making the direct deployment of deep neural networks infeasible on resource-constrained edge devices. Existing approaches, such as FHEEDGE and UniHENN, primarily focus on improving the efficiency of encrypted computation through algorithmic and systems-level optimizations. In contrast, we adopt a complementary perspective by redesigning neural architectures to align with the inherent cost model of HE operations. In this work, we present a framework that reformulates HE-aware model design as a constrained neural architecture search problem, where the objective is to identify architectures that are both cryptographically feasible and computationally efficient while preserving task performance. Starting from a pretrained network, we perform a structured, iterative transformation process that includes HE-compatible operator substitutions, depth–width trade-offs, and rotation-efficient design strategies such as successive average pooling. These transformations are guided by constraints on ciphertext slots, multiplicative depth, and rotation complexity, and are evaluated using short-horizon training to efficiently explore the architecture space. We provide a theoretical analysis showing that the transformed network approximates the original model with bounded error under layer-wise Lipschitz propagation, ensuring controlled performance degradation. As a case study, we apply the proposed framework to encrypted crack detection for structural health monitoring. The optimized model achieves an execution time of 86.5 seconds for single-threaded CPU inference in the encrypted domain, compared to an estimated 3.5 hours required by the original encrypted convolutional pipeline. Furthermore, deployment on a Raspberry Pi demonstrates end-to-end encrypted inference within approximately 7.5 minutes. These results highlight that architecture-level optimization, when framed as a constrained search problem, complements existing algorithmic acceleration approaches and provides a practical pathway for enabling efficient and privacy-preserving inference on edge devices across a broad class of sensitive AI workloads.
Reeshav Chowdhury, Anoop Mishra, Deepak Khazanchi et al.· ACM Transactions on Internet...· 0 citations