Skip to content

SoK: Wildest dreams: Reproducible research in privacy-preserving neural network training

Authors: Tanveer Khan; Mindaugas Budzys; Khoa Nguyen; Antonis Michalas
Year: 2024
Venue: Proceedings on Privacy Enhancing Technologies

type: survey collection: sok-references status: auto

Links: DOI · link

Summary

Auto-extracted summary (unreviewed)

This SoK surveys privacy-preserving machine learning based on homomorphic encryption and secure multi-party computation with a focus on the neural-network training phase, providing preliminaries, a comparison of recent frameworks on standard benchmarks, and an assessment of open-science support. The authors reproduce results of some surveyed SMPC frameworks on MNIST and find that none of the surveyed HE training works released open-source code, that HE-based training takes hours to days (or far longer) while reaching plaintext-level accuracy, and that reproducibility and usability remain major gaps.

Abstract

Machine Learning (ML), addresses a multitude of complex issues in multiple disciplines, including social sciences, finance, and medical research. ML models require substantial computing power and are only as powerful as the data utilized. Due to the high computational cost of ML methods, data scientists frequently use Machine Learning-as-a-Service (MLaaS) to outsource computation to external servers. However, when working with private information, like financial data or health records, outsourcing the computation might result in privacy issues. Recent advances in Privacy-Preserving Techniques (PPTs) have enabled ML training and inference over protected data through the use of Privacy-Preserving Machine Learning (PPML). However, these techniques are still at a preliminary stage and their application in real-world situations is demanding. In order to comprehend the discrepancy between theoretical research suggestions and actual applications, this work examines the past and present of PPML, focusing on Homomorphic Encryption (HE) and Secure Multi-party Computation (SMPC) applied to ML. This work primarily focuses on the ML model's training phase, where maintaining user data privacy is of utmost importance. We provide a solid theoretical background that eases the understanding of current approaches and their limitations. We also provide some preliminaries of SMPC, HE, and ML. In addition, we present a systemization of knowledge of the most recent PPML frameworks for model training and provide a comprehensive comparison in terms of the unique properties and performances on standard benchmarks. Also, we reproduce the results for some of the surveyed papers and examine at what level existing works in the field provide support for open science. We believe our work serves as a valuable contribution by raising awareness about the current gap between theoretical advancements and real-world applications in PPML, specifically regarding open-source availability, reproducibility, and usability.
Source: crossref

Extracted details

Field Value Source Confidence
contribution Theoretical background and preliminaries on SMPC, HE and ML for PPML llm 0.80
contribution Systematization of recent PPML frameworks for model training with comparison of properties and benchmark performance llm 0.80
contribution Reproduction of results for some surveyed SMPC frameworks on MNIST llm 0.80
contribution Assessment of open-source availability, reproducibility and usability of surveyed works llm 0.80
schemes homomorphic encryption llm 0.80
schemes secure multi-party computation llm 0.80
schemes BGV llm 0.80
libraries HElib llm 0.80
datasets MNIST llm 0.80
datasets Breast Cancer Wisconsin llm 0.80
datasets Epileptic Seizure Recognition llm 0.80
datasets credit card default (CREDIT) llm 0.80
datasets iDASH genomic llm 0.80
tasks privacy-preserving neural network training llm 0.80
tasks reproducibility assessment llm 0.80
code url https://github.com/attractivechaos/kann (URL found in the full text) regex 0.50
code url https://github.com/CryptoExperts/FV-NFLlib (URL found in the full text) regex 0.50
code url https://github.com/homenc/HElib (URL found in the full text) regex 0.50
limitations Lack of open-source implementations in surveyed works limits reproducibility llm 0.80
limitations Re-implemented code may deviate substantially from original results llm 0.80
llm relevance low llm 0.80
key result open-source availability: none of the surveyed HE training works provided openly available code (reproducibility assessment of HE frameworks) llm 0.80
key result estimated HE training time: ~60K-846K hours (Nandakumar et al.), ~1560 hours (Sphinx) (MNIST, surveyed HE training frameworks) llm 0.80
key result accuracy of surveyed HE frameworks: >90% for almost all (CryptoDL up to 99%; POSEIDON 89.9% on MNIST) (surveyed HE training frameworks) llm 0.80

SoK reference [34]

T. Khan, M. Budzys, K. Nguyen, and A. Michalas, “SoK: Wildest dreams: Reproducible research in privacy-preserving neural network training,” Proceedings on Privacy Enhancing Technologies, vol. 2024, no. 3, pp. 144–164, Jul. 2024.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: MPC 27 [p.1], CKKS 21 [p.4], TFHE 20 [p.3], BGV 18 [p.4], Secret sharing 10 [p.11], Leveled / somewhat HE 10 [p.2], BFV 8 [p.4], FHEW 5 [p.10], Multiparty HE 5 [p.5], Function secret sharing 5 [p.13], Differential privacy 4 [p.1], RSA 3 [p.2], Garbled circuits 3 [p.6], Partially homomorphic encryption 3 [p.17], Paillier 2 [p.4], Functional encryption 2 [p.1]
  • library: Microsoft SEAL 12 [p.4], HElib 12 [p.4], Zama 6 [p.10], ABY / ABY3 5 [p.6], OpenFHE 3 [p.10], Concrete (Zama) 3 [p.10], Lattigo 2 [p.10], TFHE-rs 2 [p.10], PyTorch 2 [p.7], TensorFlow 1 [p.11]
  • model: LeNet 14 [p.7], CNN 13 [p.6], VGG 10 [p.7], AlexNet 7 [p.7], Linear regression 5 [p.14], Logistic regression 3 [p.4], BERT 1 [p.5], GAN 1 [p.15]
  • dataset: MNIST 22 [p.4], ImageNet 10 [p.7], Yelp 6 [p.5], CIFAR-10 5 [p.5], UCI repository 5 [p.5], IMDB 4 [p.5], AG News 4 [p.5], Iris 3 [p.15], CIFAR-100 1 [p.10], Breast cancer (Wisconsin) 1 [p.8], Credit card / German credit 1 [p.8]
  • hardware: GPU 32 [p.5], Intel Xeon 7 [p.8], NVIDIA V100 2 [p.9], CUDA 2 [p.5]
  • technique: Convolution 20 [p.6], Two-party computation 19 [p.2], Three-party computation 13 [p.2], Bootstrapping 11 [p.5], Semi-honest model 11 [p.3], Split learning / split inference 7 [p.1], ReLU 6 [p.4], Federated learning 6 [p.1], Malicious model 6 [p.6], Matrix multiplication 3 [p.6], Encrypted training 3 [p.4], Polynomial approximation 2 [p.3], Sigmoid 2 [p.4], SIMD packing 2 [p.17], Membership inference 2 [p.1]
  • assumption: LWE 7 [p.15], Lattice-based 6 [p.14]
  • task: Medical / healthcare 6 [p.1], Genomics 6 [p.2], Finance 6 [p.1], Text classification 4 [p.14], Machine translation 2 [p.1]
  • topic: Homomorphic encryption 202 [p.1], Privacy-preserving machine learning 99 [p.1], Open source 13 [p.1], Benchmark 6 [p.1], Private / secure inference 2 [p.4]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/034_sok-wildest-dreams-reproducible-research-in-privacy-preservi.pdf sok-references 21 780.4 KB 6adbcba0c263

Database key: sok-wildest-dreams-reproducible-research-in-privacy-preserving-neural · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set sok-wildest-dreams-reproducible-research-in-privacy-preserving-neural COLUMN VALUE