Skip to content

Fiona: Accelerating FHE Inference with Packing-Aware Ternary Weights

Sep 2026 · 0 citations · 40 references
Computer Science

TL;DR

FIONA is presented, an offline optimizer that selectively ternarizes weights within a given packing layout based on the estimated effect of ternary conversion on the model's performance, and fits lower-degree replacements under a cumulative accuracy budget, reducing multiplicative depth and bootstrapping.

Abstract

Fully homomorphic encryption (FHE) enables neural network inference directly on encrypted inputs, but it remains orders of magnitude slower than plaintext in- ference. Applying the server's plaintext weights to encrypted activations involves plaintext-ciphertext multiplications (PMult) and accounts for more than half of inference time in recent systems. Ternary quantization can replace these multipli- cations with additions and subtractions, but the savings rarely materialize under packed execution. A single PMult applies a weight group fixed by the packing layout and can be avoided only when all its weights share the same ternary value. Ternarizing all groups, however, largely degrades accuracy. We present FIONA, an offline optimizer that selectively ternarizes weights within a given packing layout based on the estimated effect of ternary conversion on the model's performance. FIONA encourages a shared ternary value within each weight group and retains full-precision weights for sensitive groups, so ternar- ized and full-precision paths coexist within a layer. It then compiles these hybrid operators exactly, applying common scaling factors once to accumulated inputs and reusing sums across outputs. Weight ternarization can also narrow the input ranges of downstream polynomials. FIONA fits lower-degree replacements under a cumulative accuracy budget, reducing multiplicative depth and bootstrapping. On VGG11, ViT, and BERT, FIONA reduces PMult operations by 53.4-79.5% and accelerates end-to-end encrypted inference by 2.38x, 1.68x, and 1.84x, re- spectively, with less than 1% accuracy loss across all three models.

View source

Similar papers

Preprint Sep 2026

Memory-Efficient Designs for Word-Wise Universal Fully Homomorphic Encryption

Fully Homomorphic Encryption (FHE) enables computation on encrypted data, preserving privacy throughout analysis. While its privacy is very strong, FHE is much slower to execute than the original computation. In particular, due to the recent success in accelerating its compute, the performance bottleneck shifts to the...

A. W. B. Yudha, Erwin Eko Wahyudi, R. Rajagede et al. · 0 citations
Sep 2026

An Open-Source End-to-End FHE Implementation for Privacy-Preserving Llama 3 8B Inference

Odin is the first open-source end-to-end GPU CKKS implementation of Llama-3, an FHE inference system that co-designs ciphertext packing and model execution for Llama and uses a feature-major cross-layer layout to unify residual connections and layer interfaces.

Yu-Hang Fan, Yu-Si Chen, Kan-Yu Ye et al. · 1 citation
Open access Aug 2026

EHEIR: Efficient Homomorphic Encrypted Inference via Architectural Redesign

This work presents a framework that reformulates HE-aware model design as a constrained neural architecture search problem, where the objective is to identify architectures that are both cryptographically feasible and computationally efficient while preserving task performance.

Reeshav Chowdhury, Anoop Mishra, Deepak Khazanchi et al. · 0 citations
Open access Oct 2026

Dynamic Parameter Selection for CKKS Fully Homomorphic Encryption and Its RNS Variant

In recent years, the Cheon–Kim–Kim–Song (CKKS) scheme has gained widespread popularity among practitioners due to its support for approximate arithmetic on encrypted data. However, the decrypted outputs of CKKS inherently contain approximation errors arising from homomorphic operations. If the error term in the ciphert...

Yu Zhan, Zhen Zhao, Yan-Bo Wang et al. · 0 citations
Sep 2026

Shuffling is Not Enough: Breaking Permutation-Based Model Confidentiality in Hybrid FHE Inference

It is shown that input DP is orthogonal to model confidentiality and that the local-DP premise required for shuffle amplification cannot hold under correctness-bounded noise, and that the leaked spectra enable fingerprinting, lineage attribution, and improved logit-based extraction, while suppressing them destroys infe...

Jiseung Kim, Hyung Tae Lee · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.