TL;DR

A 2021 survey of privacy-preserving neural networks with homomorphic encryption (NN-HE), covering FHE scheme families, ML-as-a-Service (MLaaS) approaches, libraries, application areas, and open challenges, complemented by a small empirical comparison of CryptoNets and LoLa implemented in SEAL, PyTorch, and a SEAL+PyTorch combination on MNIST and Fashion-MNIST [§1, §7].

Problem and motivation

Cloud-hosted machine learning exposes confidential inputs to cloud service providers (CSPs), and conventional encryption protects data at rest and in transit but not “data in use” once decrypted for processing [§1]. FHE promises blind, non-interactive third-party computation over encrypted data without disclosing inputs or outputs [§1]. Threat model is broadly outsourcing to a non-trustworthy third-party / honest-but-curious CSP that sees only ciphertexts and never the secret key [§1, §4.2]. The paper notes the literature so far is dominated by practitioners and lacks comprehensive review of NN-HE specifically [Abstract].

Key contributions

FHE setup

ML setup

Datasets

Dataset Task Size (train/test) Modality Notes
MNIST Handwritten digit classification Test set 10,000 examples; training split not reported Grayscale 28×28 images Used to benchmark CryptoNets and LoLa under SEAL/PyTorch [§7]
Fashion-MNIST Fashion image classification Test set 10,000 examples; training split not reported Grayscale 28×28 images MNIST-like, used as a more challenging benchmark [§7]

Pipeline diagram

flowchart LR
    subgraph Client
        A[Preprocess image] --> B[Encrypt with pk]
    end
    subgraph Server
        C[Encrypted CNN inference] --> D[Square activations]
        D --> E[Encrypted logits]
    end
    subgraph Client_dec [Client]
        F[Decrypt with sk] --> G[Argmax label]
    end
    B -. ciphertext .-> C
    E -. ciphertext .-> F

Pipeline steps (text)

  1. Client preprocesses a 28×28 MNIST or F-MNIST image.
  2. Client encrypts the input using the SEAL public key (BFV or CKKS depending on the variant).
  3. Server runs the pre-trained CNN homomorphically: conv layers with plaintext weights, square activations, pooling, and a final FC layer.
  4. Server returns encrypted class logits to the client.
  5. Client decrypts and applies argmax to obtain the predicted digit / class.

Architecture diagram

Large (L) six-layer CNN used in the benchmark [§7]:

flowchart LR
    I([Input 28x28]) --> C1[Conv input layer]
    C1 --> A1[Square + Pool]
    A1 --> C2[Conv hidden]
    C2 --> P1[Pool hidden]
    P1 --> FC1[FC + Square]
    FC1 --> FC2[FC 10 outputs]
    FC2 --> O([Argmax])

CryptoNets / LoLa (Native)

Re-implementations of the original CryptoNets [52] and LoLa [57] networks; exact layer widths not restated in this paper — see [52, 57]. Both use square activations and run on MNIST 28×28 inputs [§4.2, §7].

Results

Performance comparison of three implementations of Convolutional Neural Networks, measured on a single prediction (latency, accuracy, memory) on the test set of 10,000 examples [§7, Table 5]:

Tool NN-HE variant Instance MNIST Lat. (s) MNIST Acc. (%) MNIST Mem (MB) F-MNIST Lat. (s) F-MNIST Acc. (%) F-MNIST Mem (MB)
PyTorch (plaintext) – L 0.011 99.51 465.20 0.010 89.20 448.42
SEAL CryptoNets N 283.56 98.95 4750.21 290.01 83.74 5781.14
SEAL LoLa S 0.34 96.92 593.70 0.34 82.02 576.32
SEAL LoLa N 3.50 98.95 2076.40 3.45 83.74 2010.90
SEAL LoLa L 14.43 99.20 2781.60 14.21 84.13 2430.10
SEAL+PyTorch – S 1.2 96.90 1543.34 1.16 82.01 1453.21
SEAL+PyTorch – N 12.67 98.88 7301.92 12.52 83.71 7135.67
SEAL+PyTorch – L 46.92 99.16 9871.81 46.11 84.08 8869.10

Hardware: Windows 10 64-bit, Intel(R) Core(TM) i7-8565U @ 1.8 GHz, 16 GB RAM, 256 GB SSD [§7]. The paper notes LoLa is roughly 80× faster than CryptoNets at the same accuracy (e.g. 3.50 s vs 283.56 s on MNIST native) [§7]. SEAL+PyTorch trades latency/memory for ease of implementation [§7]. Survey-level numbers also include the CRT/SIMD insight that CryptoNets has the same cost for one or 4096 predictions (amortized) [§7].

Limitations and assumptions

CryptoNets [Dowlin et al., 52]; CryptoDL [Hesamifard et al., 54]; HCNN / AlexNet-on-FHE [Badawi et al., 55]; Low-Latency CryptoNets — LoLa [Brutzkus et al., 57]; Privacy-preserving multi-party ML with HE [Takabi et al., 58]; additively-HE SGD on DNNs [Phong et al., 59]; SecureNN [Wagh et al., 60]; Babenko et al. RNS number comparison [61]; nGraph-HE2 [Boemer et al., 83]; Faster CryptoNets [Chou et al., 78]; CryptoRNN [Bakshi & Last, 80]; Fast Homomorphic evaluation of discretized NNs [Bourse et al., 81]; surveys by Acar et al. [40], Armknecht et al. [37], Naehrig et al. [38], Archer et al. [39], Martins et al. [41], Vaikuntanathan [2], among others [Table 1].

Code and artifacts

Not released. No public repository for the empirical SEAL/PyTorch/SEAL+PyTorch comparison is mentioned in the paper.

Extra diagrams (optional)

Threat model

flowchart LR
    C[Client / data owner] -- enc(x) --> S[Cloud / CSP MLaaS]
    S -- enc(y) --> C
    note[Server is honest-but-curious; sees only ciphertexts, never sk]

Activation approximation

The paper does not introduce a new polynomial activation. It surveys square (used by CryptoNets and the empirical CNNs), low-degree Taylor and Chebyshev approximations of ReLU/sigmoid/tanh, and binary activations (BNN, returning 1 if y>0 else −1) [§5.4, §8]. Square is favoured because it has multiplicative depth 1, but its unbounded derivative destabilizes training beyond two non-linear layers [§5.4]. See [§5.4] in the paper.

Open questions