TL;DR

A survey of FHE-based privacy-preserving deep learning that analyzes how researchers adapt neural networks to HE constraints (layers, activations, encoding, packing, bootstrapping, client-side help) and proposes hardware-independent metrics for fair cross-paper comparison [Abstract, p. 117477; §IX, p. 117494-117496].

Problem and motivation

Machine Learning as a Service (MLaaS) forces a client to share possibly sensitive data with a server, while the server has its own model-privacy concerns; FHE lets a third party compute on encrypted data without seeing it, but using FHE with NNs introduces multiplicative-depth limits, restricted operations, and large computational overhead [§I, p. 117477-117478]. The survey scopes itself to FHE for neural networks only, excluding broader cryptography surveys, PPML library benchmarks, and FHE compiler SoKs [§I, p. 117478]. Threat model is the standard MLaaS setup: an honest-but-curious server runs encrypted inference (or rarely training) on the client’s ciphertexts; the server learns neither input nor output [§II-A, p. 117478-117479].

Key contributions

FHE setup

ML setup

Datasets

Dataset Task Size (train/test) Modality Notes
MNIST Hand-written digit classification 60,000 / 10,000, 28×28 grey Images Most widely used; survey warns “strategies that work on MNIST do not necessarily work on other datasets” [§VIII-A, p. 117492]
CIFAR-10 Image classification 50,000 / 10,000, 32×32×3 = 3072 px Color images Closer to real-world; used by [63], [65], [66], [81], [87] [§VIII-A, p. 117493]
ImageNet Image classification ~1M images, 224×224×3 = 150,528 px, 1001 classes Color images Only SHE [65] tackles it [§VIII-A, p. 117493]
IMDB movie reviews Text classification 50,000 reviews, positive/negative Text Used by [78], [82], [83] [§VIII-A, p. 117493]
YouTube Spam Collection Spam classification ~2000 instances; encrypted training set ≈ 550 GB Text PrivFT training benchmark [78] [§VIII-B1, p. 117493]
iDASH-style / speech / medical sets Various Varies Speech, medical, tabular Mentioned in passing via [64] (encrypted speech recognition) and Faster CryptoNets [66] medical eval [§III-B, §VIII-B1, p. 117482, 117493]

Pipeline diagram

Generic HE-PPML server-side flow, with the optional interactive noise-removal loop that several surveyed works require [§VI-D, p. 117489].

flowchart LR
    subgraph Client
        A[Encode input] --> B[Encrypt with pk]
    end
    subgraph Server
        C[HE NN layers]
        C --> D{Noise OK?}
        D -- yes --> C
        D -- no --> E[Bootstrap or client refresh]
        E --> C
    end
    subgraph Client_dec [Client]
        F[Decrypt with sk] --> G[Decode / softmax]
    end
    B -. ciphertext .-> C
    D -. ciphertext .-> F
    E <-. interactive .-> F

Pipeline steps (text)

  1. Client encodes data (fixed-point, integer-scaled, binary, or ±1) and encrypts with the public key [§VI-A, p. 117488].
  2. Server runs HE-compatible NN layers (FC, conv, pooling, batch-norm, recurrent, embedding) over the ciphertext [§IV, p. 117483-117486].
  3. As noise accumulates, server either bootstraps locally (Bourse, Lee, Nandakumar, Jang [60], [70], [85], [76]) or interactively asks the client to decrypt-re-encrypt for noise refresh (CryptoRNN, Podschwadt and Takabi [84], [82]) [§VI-D, §VI-E, p. 117489].
  4. Server returns the encrypted output (often logits, leaving softmax / beam search to the client) [§VII-B, p. 117491].
  5. Client decrypts and post-processes (softmax, beam-search decoding, argmax) [§VIII-C, p. 117494].

Architecture diagram

The survey is taxonomic rather than a single model; the diagram depicts the survey’s classification of design choices (Sections IV-VI) and the major works mapped onto each branch.

flowchart LR
    R([FHE PPML for NNs])

    R --> L[NN layers IV]
    L --> Lfc[FC and conv: matrix mult and dot product]
    L --> Lpool[Pooling: avg or scaled-avg, max only via TFHE]
    L --> Lbn[Batch norm: stabilize poly activations]
    L --> Lrnn[Recurrent: RNN/LSTM/GRU, high depth]
    L --> Lemb[Embedding: client-side lookup]

    R --> A[Activation functions V]
    A --> Astep[Step/sign via TFHE bootstrap]
    A --> Asq[Square x^2 as ReLU]
    A --> Apoly[Low-deg poly Chebyshev/polyfit]
    A --> Ahigh[Composite high-deg poly with boot]
    A --> Alut[Lookup table]
    A --> Acli[Client-side activation]

    R --> E[Adaptations VI]
    E --> Eenc[Encoding: int / fixed-point / binary / ±1]
    E --> Eq[Weight quant: log2 / sparse-poly / Lloyd-max]
    E --> Ecf[Constant folding]
    E --> Ecli[Client-side compute and noise refresh]
    E --> Eboot[Bootstrapping]
    E --> Ebat[SIMD batching]

    R --> O[Optimization goals III]
    O --> Olow[Low overhead - Bourse]
    O --> Othp[High throughput - CryptoNets/CryptoDL/Chabanne]
    O --> Oimp[Improved compute - SHE/Faster CryptoNets/E2DM/LoLa/Lee/Mihara/MatHEAAN]
    O --> Ohw[Hardware accel - PrivFT/AlBadawi GPU]
    O --> Odepth[High mult-depth - Podschwadt/RNN Blocks/CryptoRNN/Nandakumar]
    O --> Ouse[Usability - nGraph-HE/CHET]

    R --> S[Security VII]
    S --> Sce[CKKS CPA attack Li-Micciancio]
    S --> Sleak[Logit/parameter/interactive leakage]

    R --> M[Proposed metrics IX]
    M --> Mres[HE op counts and memory and ciphertext sizes]
    M --> Mperf[State-of-the-art / plain / HE-friendly / encrypted four-way]
    M --> Mds[Eval beyond MNIST]

Results

The survey aggregates numbers from the surveyed papers in Table 7. Hardware is rarely normalized; the survey itself flags this as a comparability problem [§VIII-B3, IX-A, p. 117494-117495]. Headline numbers it cites:

Metric This paper (cited work) Baseline Hardware
CryptoNets [61] MNIST batched 250 s for a batch; ~0.06 s amortized per instance First HE-NN baseline Not reported
LoLa / Brutzkus et al. [69] MNIST 2.2 s for one instance via ciphertext packing (same model as CryptoNets) CryptoNets 250 s Not reported
nGraph-HE [86] MNIST 16.7 s batched total, ~0.004 s per instance CryptoNets Not reported
E2DM [68] MNIST 17.4 MB ciphertext transfer 330-410 MB for batched MNIST works [61], [63], [66] Not reported
CHET [87] CIFAR-10 164.7 s Brutzkus et al. 711 s on CIFAR-10 Not reported
Al Badawi et al. [81] CIFAR-10 GPU 553 s single GPU, 304 s on 4 GPUs CHET 164.7 s on CPU NVIDIA GPUs, ≤16 GB each
Hesamifard / Faster CryptoNets / SHE on CIFAR-10 (no packing) 3,300-12,000 s; SHE with bootstrapping projected 42.5M s (492 days) CHET 164.7 s Not reported
Lee et al. [70] ResNet-20 on CIFAR-10 ~3 h per image, 92.43% accuracy (best in survey); 31.6% time in inter-layer bootstrap — Not reported
Jang et al. [76] GRU on MNIST 90 s per sequence element (28 elements); 94.2% accuracy — Not reported
Nandakumar et al. [85] MNIST training 40 min per minibatch of 60; projected 666.7 h (~28 days) per epoch — Not reported
PrivFT [78] YouTube Spam training 120 h on 8 GPUs in parallel; 550 GB encrypted training set — 8 GPUs
Mihara et al. [75] training 29 h for 400 epochs on a “tiny dataset” — Not reported
Faster CryptoNets [66] CIFAR-10 memory 1,433.6 GB Al Badawi GPU 187 GB total Commodity vs. GPU
SHE [65] ImageNet ciphertext 7.7 GB per image — Not reported
RNN Blocks [83] client transfer 8.75-100 GB depending on input split Podschwadt and Takabi [82] 15.2 GB incl. interactive Not reported

Limitations and assumptions

Positions itself against: Azraoui et al. “SoK: Cryptography for NNs” [8], Papernot et al. “SoK: Security and privacy in ML” [9], Pulido-Gaytan et al. on PPML libraries [10], Mouris et al. on standardized FHE benchmarks [11], and Viand et al. “SoK: FHE compilers” [12]. The survey’s contribution relative to those is to focus specifically on NN + FHE design choices and to propose hardware-independent reporting metrics [§I, p. 117478; §IX, p. 117494-117496]. Internally, the survey discusses (among others): CryptoNets [61], CryptoDL [63], Chabanne et al. [62], SHE [65], Faster CryptoNets [66], E2DM [68], LoLa / Brutzkus et al. [69], Lee et al. ResNet-20 [70], Gazelle [71], MatHEAAN / Jang et al. [76], PrivFT [78], Al Badawi et al. [81], Podschwadt and Takabi [82], RNN Blocks [83], CryptoRNN [84], Nandakumar et al. [85], nGraph-HE [86], CHET [87], Bourse et al. [60], Mihara et al. [75], Zhang et al. (encrypted speech) [64].

Code and artifacts

Not released (this is a survey). The survey notes only about half of the surveyed PPML papers released source code, and refers readers to Table 3 of the paper for per-work implementation links [§IX-A, p. 117495]. Encryption libraries discussed: PALISADE, HElib, SEAL, HEAAN, TFHE library [§II-E, p. 117481].

Extra diagrams (optional)

Threat model

flowchart LR
    C[Client] -- enc(x) and pk --> S[Server: HE NN eval]
    S -- enc(y) --> C
    S <-. optional noise-refresh round .-> C

Standard MLaaS: honest-but-curious server gets only ciphertexts and the public key; some surveyed works (CryptoRNN [84], Podschwadt and Takabi [82]) add an interactive noise-refresh loop that risks leaking intermediate state to the client [§II-A, §VI-D, §VII-B, p. 117478-117479, 117489, 117491].

Activation approximation

The survey catalogues, rather than introduces, activation approximations. See Section V (p. 117486-117488) for: step (Bourse et al. [60]), square x^2 (CryptoNets and many others [61], [63], [64], [68], [69], [75], [81], [84], [86]), Chabanne et al.’s even-degree polyfit on the standard normal [62], Chebyshev integration of the derivative (CryptoDL [63], Podschwadt and Takabi [82], RNN Blocks [83]), composite poly sign of degrees 7/15/27 (Lee et al. [70]), minimax softmax 1/8 x^2 + 1/2 x + 1/4 (PrivFT [78]), Goldschmidt-division-based exp/softmax (Lee et al. [70]), learned poly coefficients (CHET [87], nGraph-HE [86], Jang et al. [76]), and sigmoid lookup tables (Nandakumar et al. [85]).

Open questions