01 GaAs Nanowire - Data Inspection - Preprocessing - Unsupervised Machine Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial demonstrates the most important basic steps involved in the analysis of scanning electron diffraction data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data was acquired from a GaAs nanowire adopting the zinc blende structure and exhibiting type I twinning (i.e. on {111}) along its length. The region of interest contains a single nanowire comprising multiple crystals each in one of the two twinned orientations and near to a <1-10> zone axis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This functionaility has been checked to run in pyxem versions 0.12.2 (November 2020) and v0.11.0 (May 2020). Bugs are always possible, do not trust the code blindly, and if you experience any issues please report them here: https://github.com/pyxem/pyxem-demos/issues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. <a href='#loa'> Loading & Inspection</a>\n",
    "2. <a href='#cal'> Alignment & Calibration</a>\n",
    "3. <a href='#vdf'> Virtual Diffraction Imaging</a>\n",
    "4. <a href='#ml'> Machine Learning SPED Data</a>\n",
    "5. <a href='#vec'> Peak Finding & Vector Analysis</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import pyXem and other required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib tk\n",
    "import pyxem as pxm\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='loa'></a> 1. Loading and Inspection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the SPED data acquired from the nanowire"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp = pxm.load('./data/01/twinned_nanowire.hdf5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the dp object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify that the data is electron diffraction data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.set_signal_type('electron_diffraction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the signal type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the data type of the object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.data.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the metadata associated with the object 'dp'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set important experimental parameters using the built in function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.set_experimental_parameters(beam_energy=300.0,\n",
    "                               camera_length=21.0,\n",
    "                               scan_rotation=277.0,\n",
    "                               convergence_angle=0.7,\n",
    "                               exposure_time=10.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how this changed the metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set another metadata item and check it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.metadata.set_item(\"General.title\", 'GaAs Nanowire')\n",
    "dp.metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the data to inspect it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.plot(cmap='inferno', vmax=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='cal'></a> 2. Alignment & Calibration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply distortion corrections to the data due to off-axis acquisition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.apply_affine_transformation(np.array([[0.99,0,0],\n",
    "                                         [0,0.69,0],\n",
    "                                         [0,0,1]]),\n",
    "                               preserve_range=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.data.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Align the dataset based on the direct beam position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.center_direct_beam(method='cross_correlate',\n",
    "                      radius_start=2,\n",
    "                      radius_finish=5,\n",
    "                      half_square_width=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.plot(cmap='inferno', vmax=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Measure known interplanar spacing to obtain calibration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpm = dp.mean((0,1))\n",
    "dpm.plot(cmap='inferno', vmax=50)\n",
    "line = pxm.roi.Line2DROI(x1=25.8525, y1=64.5691, x2=120.907, y2=77.0079, linewidth=5.49734)\n",
    "line.add_widget(dpm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trace = line(dpm)\n",
    "trace.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "recip_d111 = np.sqrt((3/5.6535**2))\n",
    "recip_cal = recip_d111 / 11.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set data calibrations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.set_diffraction_calibration(recip_cal)\n",
    "dp.set_scan_calibration(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the calibrated data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.plot(cmap='inferno', vmax=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='vdf'></a> 3. Virtual Diffraction Imaging & Selecting Regions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Interactive VDF Imaging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot an interactive virtual image integrating intensity within a circular subset of pixels in the diffraction pattern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "roi = pxm.roi.CircleROI(cx=0.,cy=0, r_inner=0, r=0.07)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp.plot_integrated_intensity(roi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the virtual diffraction image associated with the last integration window used interactively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdf = dp.get_integrated_intensity(roi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the virtual dark-field image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdf.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdf.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the virtual dark-field image as a 32bit tif"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdf.change_dtype('float32')\n",
    "vdf.save('vdfeg.tif')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Form multiple images using a VDF Generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the VDFGenerator class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyxem.generators.vdf_generator import VDFGenerator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize the VDFGenerator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdfgen = VDFGenerator(dp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate 10 annular VDF images between 0 and 1 reciprocal angstroms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdfs = vdfgen.get_concentric_vdf_images(k_min=0,\n",
    "                                        k_max=1,\n",
    "                                        k_steps=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the VDF images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdfs.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the stack of VDF image as a 32bit tif stack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdfs.change_dtype('float32')\n",
    "vdfs.save('vdsfeg.tif')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Select a region in the scan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the data with an adjustable marker indicating where to crop the scan region"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg = pxm.roi.RectangularROI(left=50, top=630, right=290, bottom=870)\n",
    "dp.plot(cmap='inferno', vmax=50)\n",
    "reg.add_widget(dp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Crop the dataset based on the region defined above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpc = reg(dp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the mean diffraction pattern from the selected region"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpcm = dpc.mean((0,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the mean diffraction pattern from the selected region"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpcm.plot(cmap='inferno', vmax=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='ml'></a> 4. Unsupervised learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform singular value decomposition (SVD) of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpc.data = dpc.data.astype('float64')\n",
    "dpc.decomposition(True, algorithm='svd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obtain a \"Scree plot\" by plotting the fraction of variance described by each principal component"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpc.plot_explained_variance_ratio()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the decomposition results and have a look at them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpc.plot_decomposition_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform non-negative matrix factorisation (NMF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpc.decomposition(True, algorithm='nmf', output_dimension=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the NMF results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpc.plot_decomposition_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='vec'></a> 5. Peak Finding & Vector Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interactively tune peak finding parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpc.find_peaks_interactive()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform peak finding on all diffraction patterns in data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "peaks = dpc.find_peaks(method='difference_of_gaussians',\n",
    "                       min_sigma=1.,\n",
    "                       max_sigma=6.,\n",
    "                       sigma_ratio=1.6,\n",
    "                       threshold=0.04,\n",
    "                       overlap=0.99)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the peaks object type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "peaks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at what's in the peaks object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "peaks.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the found peak positions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "peaks.plot_diffraction_vectors(xlim=0.8, ylim=0.8, distance_threshold=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot found peak positions as an overlay on the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "peaks.plot_diffraction_vectors_on_signal(dpc, cmap='inferno', vmax=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Form a diffracting pixels map to show where peaks were found and plot it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "crystim = peaks.get_diffracting_pixels_map(binary=False)\n",
    "crystim.plot(cmap='viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Produce a histogram of diffraction vector magnitudes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bins = np.arange(0, 1.5, recip_cal)\n",
    "ghist = peaks.get_magnitude_histogram(bins=bins)\n",
    "ghist.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Imaging using found diffraction conditions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize VDFGenerator with signal and measured diffraction vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdfgen = VDFGenerator(dpc, peaks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get VDF images for all diffraction vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdfs = vdfgen.get_vector_vdf_images(radius=recip_cal*3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the VDF images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vdfs.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "widgets": {
   "state": {
    "129b93d3ba954eb992d51fbf1e9b155e": {
     "views": [
      {
       "cell_index": 57
      }
     ]
    },
    "2ab3842519c3455592db5c834d2e33d8": {
     "views": [
      {
       "cell_index": 57
      }
     ]
    },
    "4033f1e346fd446bb8112a0ab4c1852d": {
     "views": [
      {
       "cell_index": 57
      }
     ]
    },
    "4316c6dbe2df4359be7a8af660609743": {
     "views": [
      {
       "cell_index": 61
      }
     ]
    },
    "449d4a42101d4fc79c6b6203b974695a": {
     "views": [
      {
       "cell_index": 59
      }
     ]
    },
    "48c6dfcccf1840baa54fa734f34c8f51": {
     "views": [
      {
       "cell_index": 57
      }
     ]
    },
    "5bf055ba0910450084cb44f31ff8a2eb": {
     "views": [
      {
       "cell_index": 61
      }
     ]
    },
    "bcb00cf357ee479ba389e9d19aa21bc5": {
     "views": [
      {
       "cell_index": 61
      }
     ]
    },
    "c2b0145b674c42988487114076efc891": {
     "views": [
      {
       "cell_index": 57
      }
     ]
    },
    "c77915d7befa4215a8ca901c6f8a82e5": {
     "views": [
      {
       "cell_index": 57
      }
     ]
    },
    "da3f026a56564b6195cb500d77e5e13a": {
     "views": [
      {
       "cell_index": 63
      }
     ]
    },
    "dfc7d3578ffb4ab79dc72e22be3f6255": {
     "views": [
      {
       "cell_index": 63
      }
     ]
    },
    "fb5ae80c36384f049ed13c49801cd8f6": {
     "views": [
      {
       "cell_index": 57
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}