Skip to content
Log in
Open access

A Comprehensive Evaluation of Generative Models for Privacy-Preserving Synthetic Student Data

Divine Iloh Grace Oku Shaozhi Jiang Rashmi Choudhary Rohit Kapa Nitesh Chilakala
2026 · IEEE Access · Vol 14, pp. 98101-98116 · 0 citations · 25 references
Computer Science

TL;DR

This work compares three synthesis paradigms, statistical, adversarial, adversarial, and diffusion-based, on two benchmarks and provides task-driven guidance for selecting a synthesizer in learning analytics.

Abstract

Privacy regulations and institutional policies limit the sharing of educational data, constraining reproducibility in learning analytics. Prior evaluations of synthetic data on benchmarks such as OULAD have examined statistical or adversarial synthesizers in isolation, rarely jointly assessing fidelity, utility, privacy, and explainability. We compare three synthesis paradigms, statistical (Gaussian Copula), adversarial (CTGAN), and diffusion-based (TabDDPM), on two benchmarks (OULAD: 32,593 records; ASSISTments: 8,519) across five evaluation axes: distributional fidelity (SDMetrics), downstream utility (Train on Synthetic, Test on Real), discriminative realism (classifier two-sample test), membership-inference privacy, and feature-importance preservation (SHAP). The pipeline is repeated over five random seeds with bootstrap confidence intervals and Bonferroni-corrected permutation tests (<inline-formula> <tex-math notation="LaTeX">$\alpha \prime \approx ~0.0028$ </tex-math></inline-formula>). Four findings emerge: First, TabDDPM delivers the strongest classification utility: on OULAD, a Random Forest achieves TSTR AUC <inline-formula> <tex-math notation="LaTeX">$= 0.962~\pm ~0.001$ </tex-math></inline-formula>, within 0.5 percentage points of the real-data baseline. Second, all synthesizers exhibit near-chance membership-inference risk under the evaluated kNN-based threat model (worst-case effective AUC <inline-formula> <tex-math notation="LaTeX">$\le 0.527$ </tex-math></inline-formula>). Third, distributional fidelity does not predict task utility; CTGAN scores highest on SDMetrics yet does not yield the smallest utility gap. Fourth, TabDDPM best preserves real-data feature-importance rankings on OULAD (Spearman <inline-formula> <tex-math notation="LaTeX">$\rho =0.846$ </tex-math></inline-formula>, p < 0.001). ASSISTments SHAP correlations (<inline-formula> <tex-math notation="LaTeX">$\rho ~ \ge 0.950$ </tex-math></inline-formula>) reflect a low-dimensionality ceiling effect rather than meaningful synthesis quality differences. These results provide task-driven guidance for selecting a synthesizer in learning analytics. Scope is limited to static tabular benchmarks; temporal, sequential, multimodal, and fairness-aware synthesis remain outside the present scope. Runtime results are based on CPU execution, so neural synthesizers may run faster under GPU acceleration.

DOI Semantic Scholar
Read PDF

Similar papers

Conference Jul 2026

Embedding-Space Anonymization for Privacy-Preserving AI Systems

This paper studies embedding-space privacy as a representation-level learning problem. Rather than altering raw records directly, the proposed framework applies embeddingspace transformation to full-record representations through Gaussian perturbation and adversarial representation sanitization. The method is evaluated through ablation across utility metrics, linkage attacks, attribute-inference attacks, and membership-inference tests. The primary empirical evaluation uses a synthetic fusion recommendation benchmark built from MovieLens [1], [2] 32M behavior and Adult-derived demographics [3], while a secondary synthetic medical benchmark is used to examine cross-domain transferability under more constrained conditions. The strongest results appear in the recommendation experiments. Under grouped demographic privacy evaluation, the combined condition preserves recommendation utility with $N D C G {@} K=0.6312$ while reducing exact and entity linkage from 0.7090/0.7204 to 0.0001/0.0000. Sensitive-target attacker performance remains near the majority baseline, supporting the claim of empirical privacy improvement without visible ranking degradation in that benchmark. The healthcare experiments also demonstrate meaningful embedding transformation and linkage reduction, though the current benchmark remains datalimited and therefore less conclusive for utility-focused evaluation. Overall, the findings support the conclusion that embeddingspace transformation can preserve downstream utility while substantially reducing linkage risk and sensitive-information recoverability under explicit attacker evaluation. The findings support embedding-space transformation as a practical privacypreserving strategy for embedding-driven AI systems under explicit attacker evaluation.

D. Panagoulias, Evangelia-Aikaterini Tsichrintzi, E. Sakkopoulos · 0 citations
Open access Jul 2026

Privacy-Preserving GAN for Synthetic Data against Membership Inference Attack

High-quality data are essential for machine learning and data-driven research, yet data scarcity and privacy concerns remain major obstacles in many domains. Generative models have recently emerged as a promising approach to synthesize data that follow the same statistical distribution as real datasets. However, generative models are vulnerable to membership inference attacks, which threaten data confidentiality by exploiting model outputs to infer whether specific samples were used in training. Existing defense strategies struggle to simultaneously preserve data utility and provide robust privacy protection. To address this challenge, we propose our PPGM-GAN, a Privacy-Preserving GAN for synthetic data against membership inference attack to balance both data utility and data privacy. PPGM-GAN balances privacy and utility through a privacy-utility tradeoff function that quantifies and optimizes both aspects under different adversarial knowledge. To enhance data utility, we incorporate conditional generation and key-attribute screening to ensure sufficient representation of infrequent attribute values. Additionally, differential privacy is employed during training to prevent overfitting and reduce privacy leakage. Experimental results demonstrate that PPGM-GAN outperforms state-of-the-art privacy-preserving generative models, producing high-utility synthetic data under the same privacy constraints.

Guizhang Cui, Guowei Wu, Lin Yao et al. · 0 citations
Review Open access 2026

Synthetic Data Quality Evaluation in Generative AI: Current Trends, Challenges, and Future Directions for Social Science Research

The findings indicate that while modern generative models can produce highly realistic and analytically useful datasets, persistent challenges remain, including the lack of standardized benchmarking protocols, utility–privacy trade-offs, privacy leakage risks, bias amplification, limited explainability, and governance concerns.

N. Emran, Ruhaila Maskat, Abdulrazzak Ali · 0 citations
Conference Open access 2026

Evaluating Retrieval-Augmented Generation on Social Bias Benchmarks across Small Language Models

A fundamental disconnect is suggested between a model's capacity for factual accuracy and its ability to maintain social fairness, highlighting the need for multi-dimensional evaluation frameworks for small-scale systems.

M.J.F. Valdez, Arghir-Nicolae Moldovan · 0 citations