1_3_Dimensionality_Reduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning with Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Dimensionality Reduction\n",
    "\n",
    "Dimensionality reduction is a commonly used unsupervised technique to produce low-dimensional projections of data for visualisation, or to reduce the number of features supplied to a machine learning method. It can also be applied to de-noise data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear approach: [PCA](https://scikit-learn.org/stable/modules/decomposition.html#principal-component-analysis-pca)\n",
    "\n",
    "Principal Component Analysis is the most well-known type of dimensionality reduction. It finds a series of orthogonal directions in the high-dimensional data space, ordered by their contribution to the variance. By taking only the first few principal components, we can reduce dimensionality whilst preserving as much of the variation as possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "cancer = load_breast_cancer()\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target,\n",
    "                                                    random_state=1)\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(15, 2, figsize=(10, 20))\n",
    "malignant = X_train[y_train == 0]\n",
    "benign = X_train[y_train == 1]\n",
    "\n",
    "ax = axes.ravel()\n",
    "\n",
    "for i in range(30):\n",
    "    _, bins = np.histogram(X_train[:, i], bins=50)\n",
    "    ax[i].hist(malignant[:, i], bins=bins, alpha=.5)\n",
    "    ax[i].hist(benign[:, i], bins=bins, alpha=.5)\n",
    "    ax[i].set_title(cancer.feature_names[i])\n",
    "    ax[i].set_yticks(())\n",
    "ax[0].set_xlabel(\"Feature magnitude\")\n",
    "ax[0].set_ylabel(\"Frequency\")\n",
    "ax[0].legend([\"malignant\", \"benign\"], loc=\"best\")\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# make a new Scaler object\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# calling fit and transform in sequence (using method chaining)\n",
    "scaler.fit(X_train)\n",
    "X_scaled = scaler.transform(X_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# keep the first two principal components of the data\n",
    "pca = PCA(n_components=2)\n",
    "\n",
    "# fit PCA model to breast cancer data\n",
    "pca.fit(X_scaled)\n",
    "\n",
    "# transform data onto the first two principal components\n",
    "X_pca = pca.transform(X_scaled)\n",
    "print(\"Original shape: {}\".format(str(X_scaled.shape)))\n",
    "print(\"Reduced shape: {}\".format(str(X_pca.shape)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot first vs. second principal component, colored by class\n",
    "plt.figure(figsize=(8, 8))\n",
    "plt.scatter(X_pca[y_train==0, 0], X_pca[y_train==0, 1], label=cancer.target_names[0])\n",
    "plt.scatter(X_pca[y_train==1, 0], X_pca[y_train==1, 1], label=cancer.target_names[1])\n",
    "plt.gca().set_aspect(\"equal\")\n",
    "plt.xlabel(\"First principal component\")\n",
    "plt.ylabel(\"Second principal component\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what happens without the scaling step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# keep the first two principal components of the data\n",
    "pca2 = PCA(n_components=2)\n",
    "\n",
    "# fit PCA model to breast cancer data\n",
    "pca2.fit(X_train)\n",
    "\n",
    "# transform data onto the first two principal components\n",
    "X_pca2 = pca2.transform(X_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot first vs. second principal component, colored by class\n",
    "plt.figure(figsize=(8, 8))\n",
    "plt.scatter(X_pca2[y_train==0, 0], X_pca2[y_train==0, 1], label=cancer.target_names[0])\n",
    "plt.scatter(X_pca2[y_train==1, 0], X_pca2[y_train==1, 1], label=cancer.target_names[1])\n",
    "plt.gca().set_aspect(\"equal\")\n",
    "plt.xlabel(\"First principal component\")\n",
    "plt.ylabel(\"Second principal component\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Non-linear approach: [t-SNE](https://scikit-learn.org/stable/modules/manifold.html#t-distributed-stochastic-neighbor-embedding-t-sne)\n",
    "\n",
    "t-SNE is an example of a [manifold learning](https://scikit-learn.org/stable/modules/manifold.html) algorithm. These are non-linear dimensionality reduction procedures that model the data as a lower-dimensional *manifold* embedded in a higher-dimensional space. They can often provide useful data representations for visualisation.\n",
    "\n",
    "An excellent discussion of t-SNE is given here:\n",
    "https://distill.pub/2016/misread-tsne/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "digits = load_digits()\n",
    "\n",
    "fig, axes = plt.subplots(2, 5, figsize=(10, 5),\n",
    "                         subplot_kw={'xticks':(), 'yticks': ()})\n",
    "for ax, img in zip(axes.ravel(), digits.images):\n",
    "    ax.imshow(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build a PCA model\n",
    "pca = PCA(n_components=2)\n",
    "pca.fit(digits.data)\n",
    "# transform the digits data onto the first two principal components\n",
    "digits_pca = pca.transform(digits.data)\n",
    "colors = [\"#476A2A\", \"#7851B8\", \"#BD3430\", \"#4A2D4E\", \"#875525\",\n",
    "          \"#A83683\", \"#4E655E\", \"#853541\", \"#3A3120\", \"#535D8E\"]\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.xlim(digits_pca[:, 0].min(), digits_pca[:, 0].max())\n",
    "plt.ylim(digits_pca[:, 1].min(), digits_pca[:, 1].max())\n",
    "for i in range(len(digits.data)):\n",
    "    # actually plot the digits as text instead of using scatter\n",
    "    plt.text(digits_pca[i, 0], digits_pca[i, 1], str(digits.target[i]),\n",
    "             color = colors[digits.target[i]],\n",
    "             fontdict={'weight': 'bold', 'size': 9})\n",
    "plt.xlabel(\"First principal component\")\n",
    "plt.ylabel(\"Second principal component\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "tsne = TSNE(random_state=42)\n",
    "# use fit_transform instead of fit, as TSNE has no transform method\n",
    "digits_tsne = tsne.fit_transform(digits.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 10))\n",
    "plt.xlim(digits_tsne[:, 0].min(), digits_tsne[:, 0].max() + 1)\n",
    "plt.ylim(digits_tsne[:, 1].min(), digits_tsne[:, 1].max() + 1)\n",
    "for i in range(len(digits.data)):\n",
    "    # actually plot the digits as text instead of using scatter\n",
    "    plt.text(digits_tsne[i, 0], digits_tsne[i, 1], str(digits.target[i]),\n",
    "             color = colors[digits.target[i]],\n",
    "             fontdict={'weight': 'bold', 'size': 9})\n",
    "plt.xlabel(\"t-SNE feature 0\")\n",
    "plt.ylabel(\"t-SNE feature 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Use PCA to visualise the `wine` dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_wine\n",
    "wine = load_wine()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does standardisation of the data affect the PCA result?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare your PCA result with a t-SNE visualisation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}