Deploy

cityu/cs4487/cs4487_7/index.html

@@ -223,7 +223,7 @@
 <div class="font-base text-sm"> <time datetime="2024-10-23T00:00:00.000Z"> October 23, 2024 </time> </div> 
 &bull;
 <div class="font-base text-sm">
-Last modified:  <time datetime="2024-10-23T16:06:58.000Z"> October 24, 2024 </time> </div> 
+Last modified:  <time datetime="2024-10-24T10:52:15.000Z"> October 24, 2024 </time> </div> 
 &bull;
 <div class="font-base text-sm"> 7 min read </div> </div> <h1 class="animate text-3xl font-semibold text-black dark:text-white"> Part 7 - The Expectation Maximization Algorithm &amp; Linear Dimensionality Reduction </h1> </div> <details open class="animate rounded-lg border border-black/15 dark:border-white/20" data-astro-cid-xvrfupwn> <summary data-astro-cid-xvrfupwn>Table of Contents</summary> <nav class="" data-astro-cid-xvrfupwn> <ul class="py-3" data-astro-cid-xvrfupwn> <li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#expectation-maximization-em" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Expectation Maximization (EM) </a> <ul class="translate-x-3"> <li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#jensens-inequality" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Jensen’s Inequality </a>  </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#em-derivation" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> EM Derivation </a>  </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#em-convergence" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> EM Convergence </a>  </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#remark" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Remark </a>  </li> </ul> </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#clustering-summary" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Clustering Summary </a>  </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#dimensionality-reduction" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Dimensionality Reduction </a> <ul class="translate-x-3"> <li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#reasons-for-dimensionality-reduction" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Reasons for Dimensionality Reduction </a>  </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#dimensionality-reduction-vs-feature-selection" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Dimensionality Reduction VS. Feature Selection </a>  </li> </ul> </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#linear-dimensionality-reduction" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Linear Dimensionality Reduction </a> <ul class="translate-x-3"> <li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#connection-to-linear-regression" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Connection to Linear Regression </a>  </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#matrix-formulation" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Matrix Formulation </a>  </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#learning-criterion" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Learning Criterion </a>  </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#alternating-least-squares" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Alternating Least Squares </a>  </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#lack-of-uniqueness-for-optimal-parameters" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Lack of Uniqueness for Optimal Parameters </a>  </li> </ul> </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#singular-value-decomposition-svd" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Singular Value Decomposition (SVD) </a> <ul class="translate-x-3"> <li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#reduced-form-svd" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Reduced-Form SVD </a>  </li><li class="list-inside list-disc px-6 py-1.5 text-sm"> <a href="#eckart-young-mirsky-theorem" target="_self" class="inline-block decoration-black/30 dark:decoration-white/30 hover:decoration-black/50 focus-visible:decoration-black/50 dark:hover:decoration-white/50 dark:focus-visible:decoration-white/50 text-current hover:text-black focus-visible:text-black dark:hover:text-white dark:focus-visible:text-white transition-colors duration-300 ease-in-out underline underline-offset-[3px]"> Eckart-Young-Mirsky Theorem </a>  </li> </ul> </li> </ul> </nav> </details> <article class="animate"> <h3 id="expectation-maximization-em">Expectation Maximization (EM)</h3>
 <p>EM solves a maximum likelihood problem of the form,</p>
@@ -236,12 +236,11 @@
 <li>$\{z^{(i)}\}_{i = 1}^M$: Unobserved latent variables. (e.g., in GMM, $z^{(i)}$ indicates which one of the $K$ clusters $x^{(i)}$ belongs to, which is unobserved.)</li>
 </ul>
 <h4 id="jensens-inequality">Jensen’s Inequality</h4>
-<p>Let’s revisit Jensen’s inequality, for concave functions.</p>
 <p>Suppose $f : \mathbb{R} \mapsto \mathbb{R}$ is <strong>concave</strong>, then for all probability distributions $p$, we have,</p>
 <p>$$
-f(\mathbb{E}<em>{\mathbf{x} \sim p}[x]) \geq \mathbb{E}</em>{\mathbf{x} \sim p}[f(x)]
+f(\mathbb{E} [\mathbf{x}]) \geq \mathbb{E} [f(\mathbf{x})].
 $$</p>
-<p>The subscript $\mathbf{x} \sim p$ indicates that the expectation is taken with respect to the random variable $\mathbf{x}$ drawn from the probability distribution $p$.</p>
+<p>Where the expectation is taken with respect to the random variable $\mathbf{x}$ drawn from the probability distribution $p$.</p>
 <p>The equality holds if and only if</p>
 <ol>
 <li>$\mathbf{x}$ is a constant, or,</li>
@@ -397,7 +396,7 @@ <h3 id="linear-dimensionality-reduction">Linear Dimensionality Reduction</h3>
 <p>$$
 \mathbf{x}^{(i)} = \sum_{k = 1}^K z_k^{(i)} \mathbf{b}_k
 $$</p>
-<p>where $\mathbf{b}<em>k = [b</em>{1k}, b_{2k}, \ldots, b_{Nk}]^T$ is a basis vector and $z_k^{(i)} \in \mathbb{R}$ is the corresponding weight.</p>
+<p>where $\mathbf{b}_k$ is a basis vector and $z_k^{(i)} \in \mathbb{R}$ is the corresponding weight.</p>
 <h4 id="connection-to-linear-regression">Connection to Linear Regression</h4>
 <p>If we focus on the $j$-th entry of $\mathbf{x}^{(i)}$, we have,</p>
 <p>$$
@@ -477,11 +476,11 @@ <h4 id="alternating-least-squares">Alternating Least Squares</h4>
 </li>
 </ol>
 <h4 id="lack-of-uniqueness-for-optimal-parameters">Lack of Uniqueness for Optimal Parameters</h4>
-<p>Suppose we run the ALS algorithm to convergence and obtain optimal parameters $\mathbf{Z}^{\star}$ and $\mathbf{B}^{\star}$ such that,
-$$
+<p>Suppose we run the ALS algorithm to convergence and obtain optimal parameters $\mathbf{Z}^{\star}$ and $\mathbf{B}^{\star}$ such that,</p>
+<p>$$
 \ell^{\star} = \Vert \mathbf{X} - \mathbf{Z}^{\star} \mathbf{B}^{\star} \Vert_F^2
 $$</p>
-<p>Let $\mathbf{R} \in \mathbb{R}^{K \times K$ be an arbitrary invertible matrix.</p>
+<p>Let $\mathbf{R} \in \mathbb{R}^{K \times K}$ be an arbitrary invertible matrix.</p>
 <p>A $K \times K$ matrix $\mathbf{R}$ is invertible, if there exists a $K \times K$ matrix $\mathbf{S}$ such that $\mathbf{R} \mathbf{S} = \mathbf{S} \mathbf{R} = \mathbf{I}$.
 Which we also can denote as $\mathbf{R}^{-1} = \mathbf{S}$.</p>
 <p>We obtain a different set of parameters $\mathbf{\tilde{Z}} = \mathbf{Z}^{\star} \mathbf{R}$ and $\mathbf{\tilde{B}} = \mathbf{R}^{-1} \mathbf{B}^{\star}$, with the same optimal value,

cityu/ma3518/ma3518_3/index.html

@@ -569,7 +569,7 @@ <h4 id="two-sample-test">Two-sample test</h4>
 $$
 H_0 : \mu_1 = \mu_2
 $$</p>
-<p>We compute the sample means $\bar{x}_1 and $\bar{x}_2$.</p>
+<p>We compute the sample means $\bar{x}_1$ and $\bar{x}_2$.</p>
 <ul>
 <li>If $\bar{x}_1 \simeq \bar{x}_2$, then H_0 is probably acceptable.</li>
 <li>If $\bar{x}_1$ is considerably different to $\bar{x}_2$, that is evidence that $H_0$ is not true and we are tempted to reject it.</li>
@@ -739,7 +739,7 @@ <h3 id="inferences-for-variance">Inferences for variance</h3>
 $$
 \frac{(n - 1)S^2}{\sigma^2} \sim \chi^2_{n - 1}
 $$</p>
-<p>Let $\chi^2_{\nu; \alpha} be the value such that,
+<p>Let $\chi^2_{\nu; \alpha}$ be the value such that,
 $$
 P(X > \chi^2_{\nu; \alpha}) = \alpha
 $$</p>

pagefind/pagefind-entry.json

@@ -1 +1 @@
-{"version":"1.1.1","languages":{"en":{"hash":"en_1b132e6c4f","wasm":"en","page_count":166}}}
+{"version":"1.1.1","languages":{"en":{"hash":"en_98cf7d6ad2","wasm":"en","page_count":166}}}