Generative models, latent spaces and identifiability

Latent spaces are present in many modern generative machine learning models, whether they are explicitly designed with a bottleneck or implicitly rely on a hidden state (e.g., VAEs, diffusion models, flows, GANs).

In some cases, the latent space is not of primary concern (such as in diffusion models or GANs), and we mostly care about sample quality. But in models like VAEs, the representations they learn are crucial for understanding the data-generating process and the underlying mechanisms.

Scientists from various fields, such as biology, physics, chemistry, and the social sciences, often rely on these latent representations to make sense of complex data and make predictions that impact real-world applications. A major challenge in this context is the lack of identifiability of the latent variables, which refers to the impossibility of uniquely determining the latent variables from the observed data. Since this is a proven fact, it poses a major roadblock for the tasks of learning causal and/or disentangled representations, and has led to a vast body of research aiming to characterize and alleviate the issue.

Seen from a geometric point of view, current solutions are essentially trying to make the Euclidean notion of geometric relations (distances, angles, volumes, etc.) identifiable—disregarding the geometry of the models involved. Even though some guarantees have been established by way of extra labeled data or by restricting the flexibility of the model (e.g., through multiple views or factorized structures), these approaches implicitly assume that the data manifold is flat. However, this is not always the case in practice, especially with high-dimensional data.

Instead, we propose to leverage the geometry of the learned model by pulling the ambient metric back to the latent space, and show that this makes the metric structure identifiable. This theoretical result guarantees that the pullback metric is suitable for use in downstream tasks such as clustering, classification, and regression, and may also be beneficial for causal discovery and disentanglement.