TL;DR

SAFHIRE is a hybrid FHE inference framework that keeps linear layers (convolutions, fully connected) encrypted on the server under a TFHE-based RLWE scheme and offloads non-linearities (ReLU) to the client in plaintext, eliminating bootstrapping and polynomial approximations [Abstract, §1]. It achieves 1.5x-10.5x lower end-to-end latency than ORION at comparable accuracy, and uses randomized shuffling plus differential-privacy amplification to preserve model confidentiality despite client visibility into intermediate outputs [Abstract, §1, §5].

Problem and motivation

End-to-end FHE-based ML inference is impractical at scale due to (1) costly bootstrapping (62%-85% of total inference time in ORION on ResNet-20/18/34) and (2) inefficient polynomial approximations of non-linearities such as ReLU [§1, Fig. 1]. The work targets ML-as-a-Service (MLaaS) deployments where clients submit sensitive inputs (e.g., medical, financial) and the server holds a proprietary model [§1]. Threat model: semi-honest (honest-but-curious) server that follows the protocol but tries to infer client inputs; adversarial clients that may send arbitrary inputs to reverse-engineer model weights; TLS-authenticated channel; side channels out of scope; explicit leakage limited to model metadata exchanged at setup and message sizes [§3].

Key contributions

FHE setup

ML setup

Datasets

Dataset Task Size (train/test) Modality Notes
CIFAR-10 image classification Not reported (standard splits) RGB images, 32x32x3 10 classes; used with ResNet-20 [§6.2]
Tiny (Tiny ImageNet) image classification Not reported RGB images, 64x64x3 Used with ResNet-18 [§6.2]
ImageNet image classification Not reported RGB images, 224x224x3 Used with ResNet-34; max pool replaced by avg pool [§6.2]

Pipeline diagram

flowchart LR
    subgraph Client_pre [Client: round r start]
        A[Flatten and chunk input] --> B[Power computations gamma]
        B --> C[Encrypt RLWE]
    end
    subgraph Server [Server: linear round r]
        D[Partial trace extract] --> E[Unshuffle prior perm]
        E --> F[Encrypted convolution]
        F --> G[Shuffle rows sigma_r]
        G --> H[Fast packing]
    end
    subgraph Client_post [Client: round r end]
        I[Decrypt RLWE] --> J[Plaintext ReLU]
        J --> K[Requantize for next layer]
    end
    C -. ciphertexts .-> D
    H -. packed ciphertexts .-> I
    K -. next round input .-> A
    K --> L([Softmax / argmax on final layer])

Pipeline steps (text)

  1. Client flattens the input image into a 1-D vector V and splits into k chunks of length N, zero-padding the last one [§4.3].
  2. Client computes powers of each chunk polynomial m(X) at evaluations specific to the chosen partial-trace level gamma (gamma in {0,1,2}) [§4.3].
  3. Client encrypts each chunk (and its powers if gamma > 0) under RLWE key s(X) and sends ciphertexts to the server [§4.3].
  4. Server performs partial trace extraction on each ciphertext (no-op if gamma = 0) using a homomorphic factorization of the trace operator [§4.4].
  5. Server applies the inverse permutation sigma_{r-1}^{-1} to undo the previous round’s shuffle (identity for r = 1) [§4.4].
  6. Server runs the linear layer (convolution or fully connected) homomorphically over the unpacked, unshuffled encrypted matrix [§4.4].
  7. Server applies a fresh per-round random permutation sigma_r (derived from a secret per-session seed plus round counter) to shuffle rows of the output matrix [§4.4].
  8. Server applies fast packing to compress t^{beta-1}(t-1) LWE ciphertexts into a single RLWE ciphertext and sends to the client [§4.4].
  9. Client decrypts each packed ciphertext, extracts the designated packed positions, and concatenates them (dropping padding) to recover layer output V’ [§4.5].
  10. Client applies the exact activation (ReLU) element-wise in plaintext [§4.5].
  11. Client requantizes outputs using per-layer scale eta: y -> clip(round(y/eta)) to the input precision of the next layer [§4.5].
  12. Loop steps 1-11 for L rounds (one round per linear block bounded by a non-linearity) [§4.1].
  13. After the last linear layer (no shuffling applied), client computes softmax over the logits to obtain final probabilities [§4.5, §5].

Architecture diagram

The paper does not enumerate the per-layer ResNet structure beyond noting standard ResNet-20/18/34 (with max pool -> avg pool substitution for ResNet-34/ImageNet). The diagram below mirrors what the paper actually runs: the generic per-round hybrid block of which ResNet-20 has 20 stacked instances [§4.1, §6.2].

flowchart LR
    I([Input image 3x32x32]) --> R1[ResNet block 1 conv linear]
    R1 -. enc/dec .-> RA1[ReLU plaintext on client]
    RA1 --> R2[ResNet block 2 conv linear]
    R2 -. enc/dec .-> RA2[ReLU plaintext on client]
    RA2 --> RDOT[... up to L blocks ...]
    RDOT --> RL[Final FC layer encrypted no shuffle]
    RL --> SM([Softmax on client])

Per-layer convention: each “round” in SAFHIRE corresponds to one linear block bounded by a non-linearity; ResNet-20 has L = 20 such blocks following standard He et al. architecture [§4.1, §6.2]. For ResNet-34 on ImageNet the max-pool is replaced by avg-pool to keep operations linear [§6.2].

Results

Headline latency comparison vs ORION (state-of-the-art CKKS baseline). Network speed assumed at 1.25 MB/s (10 Mbit/s) conservatively when computing end-to-end times [Table 3]. Best speedup per (model, precision) shown.

Metric This paper (SAFHIRE) Baseline (ORION) Hardware
E2E latency ResNet-20/CIFAR-10, b=8, gamma=2 (CPU 1-thread) 116.2 s (9.0x speedup) 1040.4 s Intel Xeon E5-2690 v4 @ 2.6GHz, 56 logical cores, 925 GB RAM, 1 thread [§6.1, Table 3]
E2E latency ResNet-18/Tiny, b=8, gamma=2 (CPU 1-thread) 457.2 s (10.5x speedup) 4794.4 s Same Xeon E5-2690 v4 server [Table 3]
E2E latency ResNet-34/ImageNet, b=8, gamma=2 (CPU 1-thread) 2526.3 s (4.3x speedup) 10819.6 s Intel Xeon Platinum 8272L @ 2.6GHz, 104 logical cores, 2975 GB RAM [§6.1, Table 3]
E2E latency ResNet-20/CIFAR-10 (GPU) 13.65 s (b=8, gamma=2) / 11.29 s compute only (b=8, gamma=0) Not reported NVIDIA A100-SXM4 80GB + AMD EPYC 7543, 24 cores @ 2.80GHz, 1.25 MB/s network [§6.1, §6.6]
Accuracy CIFAR-10 (ResNet-20) 90.00% (16-bit) / 89.63% (12-bit) 91.70% (ORION, SiLU) A2Q+ quantization [§6.3, Table 4]
Accuracy Tiny (ResNet-18) 53.19% (16-bit) / 53.16% (12-bit) 57.00% (ORION, SiLU) [Table 4]
Server compute speedup (multithreaded) 1.53x-86.12x vs ORION - CPU multithreading 1-32 threads [§1, §6.5]
Communication (single inference) Up to 499 MB (ResNet-18), up to 170 MB (ResNet-20) Not reported [§1]
Client-side share of compute (Tiny/ResNet-18) 0.5-5.2 s vs 361.5-1141.6 s server-side - Server time 76x-1903x greater than client time [§6.4, Table 5]

Additional observations: gamma = 1 emerges as a robust trade-off across network regimes; gamma = 2 helps at low bandwidth but raises upload size (e.g., from 4.2 MB to 83.9 MB on CIFAR-10 at b=16) [§6.4, Fig. 6]. Multithreading saturates around 16 threads; GPU acceleration delivers a further 14.4x over single-thread CPU for ResNet-20/CIFAR-10 at b=8, gamma=0 (162.57 s -> 11.29 s compute time) [§6.5, §6.6]. Packing dominates server time, e.g., 96.1% of compute on GPU at gamma=0, b=16 [§6.6].

Limitations and assumptions

Code and artifacts

Not released. The paper describes the implementation in Julia (NNlib.jl, FFTW.jl, Permutations.jl) but does not provide a repository URL [§6.1]. License: Not reported.

Extra diagrams (optional)

Threat model

flowchart LR
    C[Client adversarial] -- enc input x --> S[Server semi-honest]
    S -- enc shuffled output sigma_r Ax+b --> C
    C -- enc ReLU output --> S
    S -.- M[Plaintext model weights theta]

The semi-honest server holds plaintext model weights theta and faithfully runs the protocol but tries to learn x from messages [§3]. The client may submit arbitrary crafted inputs (e.g., basis vectors e_i, scaled probes 2*e_i) to attempt model extraction; shuffling sigma_r on rows (with secret per-session seed + round counter) prevents column reconstruction even up to permutation when both input and output shuffles are present [§5].

Activation approximation

Not applicable: SAFHIRE evaluates exact ReLU (and softmax on the last layer) in plaintext on the client, deliberately avoiding any polynomial approximation [§1, §4.5, §8].

Open questions