@model function pPCA_ARD(X)
# Dimensionality of the problem.
@@ -1938,544 +1938,544 @@
Note that:
@@ -1276,55 +1276,55 @@We can quantitatively check the absolute magnitudes of the column average of the gap between mat_exp and mat_rec:
We see the two groups are well separated in this 2-D space. As an unsupervised learning method, performing PCA on this dataset gives membership for each cell instance. Another way to put it: 2 dimensions is enough to capture the main structure of the data.
A direct question arises from above practice is: how many principal components do we want to keep, in order to sufficiently represent the latent structure in the data? This is a very central question for all latent factor models, i.e. how many dimensions are needed to represent that data in the latent space. In the case of PCA, there exist a lot of heuristics to make that choice. For example, We can tune the number of principal components using empirical methods such as cross-validation based some criteria such as MSE between the posterior predicted (e.g. mean predictions) data matrix and the original data matrix or the percentage of variation explained [3].
-For p-PCA, this can be done in an elegant and principled way, using a technique called Automatic Relevance Determination (ARD). ARD can help pick the correct number of principal directions by regularizing the solution space using a parameterized, data-dependent prior distribution that effectively prunes away redundant or superfluous features [4]. Essentially, we are using a specific prior over the factor loadings \(\mathbf{W}\) that allows us to prune away dimensions in the latent space. The prior is determined by a precision hyperparameter \(\alpha\). Here, smaller values of \(\alpha\) correspond to more important components. You can find more details about this in e.g. [5].
+A direct question arises from above practice is: how many principal components do we want to keep, in order to sufficiently represent the latent structure in the data? This is a very central question for all latent factor models, i.e. how many dimensions are needed to represent that data in the latent space. In the case of PCA, there exist a lot of heuristics to make that choice. For example, We can tune the number of principal components using empirical methods such as cross-validation based some criteria such as MSE between the posterior predicted (e.g. mean predictions) data matrix and the original data matrix or the percentage of variation explained 5.
+For p-PCA, this can be done in an elegant and principled way, using a technique called Automatic Relevance Determination (ARD). ARD can help pick the correct number of principal directions by regularizing the solution space using a parameterized, data-dependent prior distribution that effectively prunes away redundant or superfluous features 6. Essentially, we are using a specific prior over the factor loadings \(\mathbf{W}\) that allows us to prune away dimensions in the latent space. The prior is determined by a precision hyperparameter \(\alpha\). Here, smaller values of \(\alpha\) correspond to more important components. You can find more details about this in, for example, Bishop (2006) 7.
@model function pPCA_ARD(X)
# Dimensionality of the problem.
@@ -1938,544 +1938,544 @@
We can inspect α to see which elements are small (i.e. high relevance). To do this, we first sort α using sortperm() (in ascending order by default), and record the indices of the first two smallest values (among the \(D=9\) \(\alpha\) values). After picking the desired principal directions, we extract the corresponding subset loading vectors from \(\mathbf{W}\), and the corresponding dimensions of \(\mathbf{Z}\). We obtain a posterior predicted matrix \(\mathbf{X} \in \mathbb{R}^{2 \times 60}\) as the product of the two sub-matrices, and compare the recovered info with the original matrix.
We observe that, the data in the original space is recovered with key information, the distinct feature values in the first and last three genes for the two cell groups, are preserved. We can also examine the data in the dimension-reduced space, i.e. the selected components (rows) in \(\mathbf{Z}\).
@@ -3055,102 +3055,102 @@Gareth M. James, Daniela Witten, Trevor Hastie, Robert Tibshirani, An Introduction to Statistical Learning, Springer, 2013.↩︎
Probabilistic PCA by TensorFlow, “https://www.tensorflow.org/probability/examples/Probabilistic_PCA”.↩︎
Probabilistic PCA by TensorFlow, “https://www.tensorflow.org/probability/examples/Probabilistic_PCA”.↩︎
Gareth M. James, Daniela Witten, Trevor Hastie, Robert Tibshirani, An Introduction to Statistical Learning, Springer, 2013.↩︎
David Wipf, Srikantan Nagarajan, A New View of Automatic Relevance Determination, NIPS 2007.↩︎
Christopher Bishop, Pattern Recognition and Machine Learning, Springer, 2006.↩︎