TL;DR

The paper designs and benchmarks three FHE-friendly activation strategies — square, degree-50 Chebyshev polynomial approximation of ReLU, and a novel CKKS<->TFHE scheme-switching ReLU — across LeNet-5/MNIST and ResNet-20/CIFAR-10, showing that scheme-switching recovers 6 percentage points of accuracy over poly-approx on ResNet-20 at roughly 1.5x the inference latency [Abstract, §5.2].

Problem and motivation

FHE natively supports only linear operations, which forces practitioners to either replace nonlinear activations (ReLU, sigmoid, tanh) with low-degree polynomial proxies (e.g. x^2) or approximate them with higher-degree polynomials, both of which trade accuracy for cost [§1, §4]. There has been “little systematic investigation regarding the contexts and model architectures where different activation functions perform best under FHE constraints” [§1]. Threat model: standard privacy-preserving outsourced inference — a user encrypts inputs under their public key, the cloud runs FHE inference on a (model-provider) model, and only the user can decrypt; an honest-but-curious server is assumed and “even in the event of a cloud breach, the user data and model stay protected” [§3.1, Fig. 1].

Key contributions

FHE setup

ML setup

Datasets

Dataset Task Size (train/test) Modality Notes
MNIST Handwritten digit classification Trained in PyTorch on full set; encrypted eval on 750 images from the MNIST validation set 28x28 grayscale images Two LeNet-5 variants trained — one with ReLU, one with square; weights exported as CSV [§5.1]
CIFAR-10 10-class image classification Trained on full set; encrypted eval on 250 images from the CIFAR-10 validation set 3x32x32 RGB images Only ReLU training was feasible; square training “infeasible due to instability during backpropagation” [§5.1]

Pipeline diagram

flowchart LR
    subgraph Client
        A[Image input] --> B[Encrypt CKKS]
    end
    subgraph Server
        C[CKKS linear layer] --> D{Activation?}
        D -- square --> E[CKKS x*x]
        D -- poly-approx --> F[Chebyshev deg-50]
        D -- scheme-switch --> G[CKKS to TFHE]
        G --> H[TFHE compare]
        H --> I[TFHE to CKKS]
        I --> J[Mask multiply]
        E --> K[Next layer or bootstrap]
        F --> K
        J --> K
        K --> C
    end
    subgraph Client_dec [Client]
        L[Decrypt CKKS] --> M[argmax label]
    end
    B -. ct .-> C
    K -. ct .-> L

Pipeline steps (text)

  1. Client encrypts the image under CKKS with their public key [§3.1, Fig. 1].
  2. Server runs the convolution / FC layer on the CKKS ciphertext using SIMD slot packing [§3.2].
  3. Server picks the activation strategy: (a) one CKKS multiplication for square, (b) degree-50 Chebyshev poly-approx of ReLU (after 1/beta scaling), or (c) scheme-switch CKKS->TFHE for sign, switch back, build mask, multiply [§4.1, §4.2, §4.3].
  4. Server bootstraps the CKKS ciphertext at planned points to refresh the noise budget (counts in Table 1) [§5.1, Table 1].
  5. Server iterates layers (LeNet-5 conv-pool-FC stack, or 9 ResNet-20 BasicBlocks plus shortcut convs in Layer 2 and Layer 3) [Fig. 2, Fig. 3].
  6. Server returns the encrypted logits to the client.
  7. Client decrypts with their secret key and takes argmax [§3.1].

Architecture diagram

LeNet-5 (MNIST)

flowchart LR
    I([Input 1x28x28]) --> C1[Conv 6, 5x5]
    C1 --> A1[Activation]
    A1 --> P1[AvgPool]
    P1 --> C2[Conv 16, 5x5]
    C2 --> A2[Activation]
    A2 --> P2[AvgPool]
    P2 --> F[Flatten 256]
    F --> FC1[FC 120]
    FC1 --> A3[Activation]
    A3 --> FC2[FC 84]
    FC2 --> A4[Activation]
    A4 --> FC3[FC 10]
    FC3 --> O([argmax])

ResNet-20 (CIFAR-10)

flowchart LR
    I([Input 3x32x32]) --> C0[Conv 16, 3x3]
    C0 --> L1[Layer 1: 3 blocks 16->16]
    L1 --> L2[Layer 2: 3 blocks 16->32]
    L2 --> L3[Layer 3: 3 blocks 32->64]
    L3 --> AP[AvgPool]
    AP --> FC[FC 10]
    FC --> O([argmax])
    subgraph Block [BasicBlock]
        B1[Conv 3x3] --> B2[Activation]
        B2 --> B3[Conv 3x3]
        B3 --> B4[Add shortcut]
        B4 --> B5[Activation]
    end

Results

Reported on 750 MNIST validation images for LeNet-5 and 250 CIFAR-10 validation images for ResNet-20; latency was measured on the local AMD Ryzen 9 5900X (12-core, 64 GB RAM); the SOL supercomputer at ASU was used to speed up accuracy sweeps but not the reported latency [§5.1, §5.2].

Metric This paper Baseline Hardware
LeNet-5 / Square accuracy 99.4% encrypted (-0.2 vs 99.6% plaintext) – AMD Ryzen 9 5900X [§5.1, Table 3]
LeNet-5 / Square latency 128 s / image – AMD Ryzen 9 5900X [Table 3]
LeNet-5 / Poly-approx ReLU accuracy 98.9% encrypted (-0.3 vs 99.2% plaintext) – AMD Ryzen 9 5900X [Table 3]
LeNet-5 / Poly-approx ReLU latency 95 s / image (best of the three) – AMD Ryzen 9 5900X [Table 3]
LeNet-5 / Scheme-switch ReLU accuracy 98.9% encrypted (-0.3 vs 99.2% plaintext) – AMD Ryzen 9 5900X [Table 3]
LeNet-5 / Scheme-switch ReLU latency 168 s / image (~1.7x poly-approx) – AMD Ryzen 9 5900X [§5.2, Table 3]
ResNet-20 / Poly-approx ReLU accuracy 83.8% encrypted (-8.4 vs 92.2% plaintext) Lee et al. 2022 91.31% in 2,271 s; Kim & Guyot 2023 92.04% in 255 s; Rovida & Leporati 2024 91.53% in 260 s [§2] AMD Ryzen 9 5900X [Table 3]
ResNet-20 / Poly-approx ReLU latency 1,145 s / image (see prior row) AMD Ryzen 9 5900X [Table 3]
ResNet-20 / Scheme-switch ReLU accuracy 89.8% encrypted (-2.4 vs 99.2% plaintext reported; ResNet-20 plaintext is listed as 99.2% in Table 3 row “ResNet Scheme Switch”, likely a typo for the same ResNet-20 reference) – AMD Ryzen 9 5900X [Table 3, §5.2]
ResNet-20 / Scheme-switch ReLU latency 1,697 s / image (~1.5x poly-approx) – AMD Ryzen 9 5900X [Table 3, §5.2]

Headline: scheme-switching recovers ~6 percentage points of accuracy over Chebyshev poly-approx on ResNet-20 (89.8% vs 83.8%) at the cost of ~1.5x more inference latency [§5.2, Table 3].

Limitations and assumptions

CryptoNets [GBDL+16], CryptoDL [HTG17], HCNN/AlexNet-on-FHE [ABJL+20], E2DM [JKLS18], DiNN [BMMP17], Benamira et al. TFHE-CNN [BP23], Lee et al. 2022 [LKL+22], Kim & Guyot 2023 [KG23], Rovida & Leporati 2024 [RL24]; hybrid FHE+MPC systems Gazelle, MiniONN, XONN are mentioned but considered out of scope [§2].

Code and artifacts

Not released.

Extra diagrams

Activation approximation

flowchart LR
    X[Input x] -->|square| S[y = x*x, 1 CKKS mult, 0 bootstrap]
    X -->|Chebyshev deg-50| C1[Scale by 1/beta] --> C2[Chebyshev T_n recursion]
    C2 --> C3[Rescale by beta]
    C3 --> C4[~8 mult depth, deg 50 chosen from sweep 10..100]
    X -->|scheme-switch| K1[CKKS to TFHE]
    K1 --> K2[TFHE compare to 0]
    K2 --> K3[TFHE to CKKS as bit]
    K3 --> K4[mask = 1 - bit]
    K4 --> K5[CKKS multiply, 1 mult depth]

Key approximation details from the paper:

See Figures 2 and 3 in the paper for the LeNet-5 and ResNet-20 layer-level diagrams.

Open questions