TL;DR

ConvFHE packs each input image and kernel into the coefficients of a single CKKS plaintext polynomial so that a convolution becomes one ciphertext multiplication whose cost is independent of kernel width [§IV-B]. Combined with a modified bootstrapping that switches between coefficient and slot domains, the method evaluates a 20-layer CNN in 398 s for 94.0% on CIFAR-10 and reduces a part of an 18-layer ImageNet network by 48.1% [§I, §V-B].

Problem and motivation

FHE-based secure inference offers the strongest privacy properties (one round, no extra interaction) but prior work was confined to shallow, narrow networks with square activations because deep CNNs require bootstrapping at every layer and per-convolution cost scales with k^2 rotations under standard vector encoding [§I, §II-B]. The threat model assumes a client outsourcing CNN inference to an honest-but-curious service provider; semantic security of CKKS hides everything except input size; the model structure (layers, batches) is assumed known to the evaluator, but the kernel size need not be [§IV-D].

Key contributions

FHE setup

ML setup

Datasets

Dataset Task Size (train/test) Modality Notes
CIFAR-10 [48] 10-class classification 50K / 10K 32x32x3 RGB images Used to evaluate full Plain-20 CNN and variants under FHE [§V-B, Appendix E]
CIFAR-100 [48] 100-class classification 50K / 10K 32x32x3 RGB images Same architecture as CIFAR-10 [§V-B, Appendix E]
ImageNet (ILSVRC 2012) [49] 1000-class classification 1.28M / 50K val / 100K test 224x224x3 (after cropping) Only the final 8 layers of Plain-18 evaluated under FHE (Deep Representation method [19]); some results over 2000 samples only (marked dagger in Table IV) [§V-B, Appendix E]

Pipeline diagram

flowchart LR
    subgraph Client
        A[Preprocess image] --> B[Coeff encode + encrypt]
    end
    subgraph Server
        C[ConvFHE batch conv] --> D[PackLWEs into 1 ct]
        D --> E[CtoS + EvalSine]
        E --> F[Poly ReLU in slots]
        F --> G[Ext valid / strided]
        G --> H[StoC back to coeffs]
        H -. next layer .-> C
        H --> I[Final FC]
    end
    subgraph Client_dec [Client]
        J[Decrypt with sk] --> K[Argmax label]
    end
    B -. ciphertext .-> C
    I -. ciphertext .-> J

Pipeline steps (text)

  1. Client coefficient-encodes the input image into polynomial coefficients of R and CKKS-encrypts under public key [§IV-A].
  2. Server runs ConvFHE BatchConv: each convolution becomes one ciphertext mult with no rotations, and outputs are packed into a single ciphertext via generalized PackLWEs [§IV-B, Alg. 1-2].
  3. Bootstrapping’s CtoS + EvalSine step re-encodes the coefficient-domain output into CKKS slot ciphertexts [§IV-C].
  4. Composite-polynomial ReLU (deg-13 via f_3 o f_2 o f_1) is evaluated component-wise in slots [§IV-C, Appendix B].
  5. Ext step extracts valid (and strided) output positions in the slot domain using one multiply + at most w/2 rotations [§IV-C, Appendix C, Table V].
  6. StoC re-encodes the activated tensor back to the coefficient domain so the next ConvFHE conv can consume it; this is the only bootstrapping per layer [§IV-C, Fig. 4].
  7. Repeat for each convolutional layer (every layer is bootstrapped) [§V-B].
  8. Final fully-connected layer produces an encrypted logit vector returned to the client [Appendix E, Table VII].
  9. Client decrypts and applies argmax.

Architecture diagram

Plain-20 with wideness omega and kernel k; widths shown for omega=1, k=3 (the original CIFAR-10 net) [Appendix E, Table VII].

flowchart LR
    I([Input 3x32x32]) --> C0[Conv0 k x k x 16]
    C0 --> B1[BN + polyReLU]
    B1 --> CB1["Conv1 block (k x k x 16) x 6"]
    CB1 --> S2[Strided conv k x k x 32]
    S2 --> CB2["Conv2 block (k x k x 32) x 5"]
    CB2 --> S3[Strided conv k x k x 64]
    S3 --> CB3["Conv3 block (k x k x 64) x 5"]
    CB3 --> GAP[Global avg pool]
    GAP --> FC[FC -> 10 or 100]
    FC --> O([Argmax])

For ImageNet, Plain-18 has the same shape with different block sizes; only the final 8 layers (Conv3/Conv4 + FC, >=91% of params) run under FHE [§V-B, Appendix E].

Results

Metric This paper Baseline Hardware
CIFAR-10 accuracy, k3-d20-w1 (Plain-20) 91.31% 91.31% plain CNN (no FHE) [44] AMD EPYC 7402P, 1 thread [§V]
CIFAR-10 best accuracy (k7-d20-w3) 94.00% 91.5% prior FHE [29] at 2271 s same
CIFAR-10 latency at 92.0% 255 s 2271 s @ 91.3% [29], 10602 s [28] (64 threads) same vs. prior-work hw [§II-B, §V-B]
CIFAR-10 latency at 94.0% (k7-d20-w3) 398 s n/a (state-of-art) same
CIFAR-100 best accuracy 73.5% n/a same
Plain-20 CIFAR-10/100 timing reduction over Vector-Encoding baseline -18.9% Vector Encoding ([15], [29] estimated favourably) same [§V-B, Table III]
Part of Plain-18 ImageNet (final 8 layers) timing reduction -48.1% Vector-Encoding baseline same [§V-B, Table III]
Single conv speedup (k=3 / 5 / 7) ~12x / ~25x / ~46x faster Vector-Encoding baseline same [§V-A, Fig. 5]
Conv-layer-with-bootstrap reduction 13% - 83% Vector Encoding same [§V-A, Fig. 6]
Communication cost (input ciphertext for CIFAR-10 net) ~2 MB ~110 MB best hybrid [34] for 32-layer CNN; 123 MB SHE [27] [§II-B, §II-C]
ReLU approximation precision 10-bit on [-1, 1] square activation in prior CKKS-only deep nets [§IV-C, Appendix B]

Single-inference time used in the comparison table: 398 s (Plain-20 variant k7-d20-w3 reaching 94.0% on CIFAR-10) [§I, §V-B, Table IV].

Limitations and assumptions

CryptoNets [9], CryptoDL [11], Faster CryptoNets [12], nGraph-HE / nGraph-HE2 [13], [14] (high-throughput); Gazelle [15], CHET [16], EVA / EVA improved [17], [18], LoLa [19], Falcon [20] (low-latency); TFHE-based Bourse et al. [23], Chillotti et al. [24], SHE [27]; deep-CNN CKKS works Lee et al. [28], [29]; hybrid PPML CrypTFlow2 [31], Delphi [32], APAS [33], Cheetah [34]; ENSEI [21], Fourier-FALCON [22]; secure human action recognition [30].

Code and artifacts

Source available: https://anonymous.4open.science/r/optimal_conv-BD07 [§V footnote 5]. License: not reported.

Extra diagrams (optional)

Threat model

flowchart LR
    C[Client] -- "Coeff-encoded enc(x), pk" --> S[Service provider]
    S -- "enc(CNN(x))" --> C
    note["S knows topology (layers, batches);<br/>does NOT need to know kernel size"]

Single-round, non-interactive secure inference under semantic security of CKKS; honest-but-curious server [§IV-D].

Activation approximation

Composite polynomial approximation of sign(x) from Lee et al. [42]:

Then ReLU(x) ~= (x + x * f_3(f_2(f_1(x)))) / 2, max error 2^-10 on [-1, 1]; multiplicative-depth budget = 3 + 3 + 4 = 10 [Appendix B].

Open questions