When Can Synthetic Data Be Trusted?

Synthetic data is often judged by how close its distribution looks to the real thing. The question here is whether that closeness carries through to what people actually use the data for: the statistics and inferences computed on it.

With Dr. Chathurika Abeykoon, I built an evaluation suite that benchmarks three classes of generator (Gaussian copula, TVAE, CTGAN) across six metrics at three levels: distributional distance (Wasserstein, MMD, KS), moment agreement, and inferential agreement. It runs on several public datasets, including ones above 100k rows.

The main finding is a hierarchy. Distributional fit predicts moment agreement, which in turn predicts inferential agreement, with a pooled Spearman correlation of about $\rho \approx 0.70$ along the chain.