Awesome Numerics

Resources for learning about numerical methods. This document is a work in progress, but nevertheless contains useful information. Indended audience has little or modest experience with numerical methods. Assumed knowledge is an undergraduate linear algebra course and the calculus/diff. eq. sequence.

Disclaimer: Most linked content is not licensed, owned, or maintained by S. H. Bryngelson, nor can I ensure the correctness of any of the below resources. Use for educational purposes and at your own risk.

Table of contents

• Linear algebra refresher
• Error analysis
• Floating point numbers
• Linear systems
• Least squares
• Eigenvalues
• Solving nonlinear equations
• Optimization
• Interpolation
• Numerical integration and quadrature
• Initial value problems
• Ordinary differential equations
• Partial differential equations

Linear algebra refresher

Numerical analysis and linear algebra are fundamentally entangled. Forget your lin. alg. basics at your own risk. Fortunately, there are many great resources to get you up to speed, so dust off your copy of Gil Strang's book and check out the links below.

• Gilbert Strang (video)
• 3Blue1Brown (video playlist)

Error analysis

• What Is Backward Error? by Nick Higham
• Some examples of error analysis from Doug Hundley
• Lecture notes on conditioning and error analysis from Jim Lambers
• Lecture slides on the same topic from Jan Verschelde
• Some examples with solutions in an IPython Notebook

Floating point numbers

• Computerphile (video)
• IEEE standards (video)
• GeeksForGeeks 
• Other floating point number formats
  • Google bfloat
  • NVIDIA tf32
  • Tesla cfloat
  • Posits, discussed by John Cook
• Avoiding catastrophic cancellation with quadratic formula
• Lecture notes from Richard Krueger of UT 
• What Every Computer Scientist Should Know About Floating-Point Arithmetic by David Goldberg
• Rounding and chopping to a floating point number by Ming Gu
• Lecture slides on floating point rounding by Bei Wang
• Code examples
  • A Primer on Floating-Point Arithmetic and IEEE 754 in C/Cpp
  • Examples of cancellation and floating point numbers from Andreas Klöckner of UIUC (Python)

Linear systems

• Overview of solving linear systems from Chen Greif

• Online text from Fundamentals of Numerical Computation

• Don't invert that matrix!

  • Greg Gundersen
  • John Cook
  • SciComp StackExchange Q/A

• Direct linear solvers

  • John Foster (video) and accompanying notes
  • LU Decomposition
    • thenumericalmethodsguy (videos)
    • Lecture notes from CCNY Math

• Using linear solvers to benchmark supercomputers

  • Vanilla linpack discussion
  • High performance linpack (HPL)

• Code examples

  • Examples from Andreas Klöckner of UIUC (Python)
  • LU Decomp. via Doolittle from Quantstart (Python)
  • LU from John Foster (Python)
  • LU in Matlab and with pivoting
  • Computing Reduced Row Echelon Form from Stelios Sfakianakis (Python)
  • Computing LU and tridiagonal solves in (Matlab) from ashsepra
  • Sherman Morrison formula (Python)

Least squares

• Steve Brunton on linear systems, least squares, and more (video)

• Strang on four ways to solve least squares problems (video)

• QR and its flavors

  • Least Squares via QR with John Lambert
  • Gil Strang on Gram-Schmidt (video)
  • Gram-Schmidt and loss of orthogonality 
  • Blog on QR from TDS, Ben Denis Shaffer
  • Arnold Yim on QR (video)
  • A linear alg. explanation of projectors from Ross MacAusland

• Online text from Fundamentals of Numerical Computation

• Code examples

  • Andreas Klöckner on several methods: QR, Householder, Givens, Gram-Schmidt, and more (Python)
  • QR via Householder (Python)
  • QR and its flavors (Matlab)
  • QR from Trefethen and Bau's book in ipynb (Matlab)

Eigenvalues

• Introduction

  • 3Blue1Brown (video)
  • Lecture notes from Yin Zhang of Rice

• Videos from Prof. Gil Strang (MIT)

  • Similar matrices (video)
  • Symmetric matrices (video)

• Other comprehensive lecture notes

  • From Peter J. Olver of UMinnesota

• Why we don't use characteristic polynomials to compute eigenvalues in practice, at least for large matrices

  • From Michael Heath's book:

    Computing the coefficients of the characteristic polynomial for a given matrix is, in general, a substantial task. The coefficients of the characteristic polynomial can be highly sensitive to perturbations in the matrix, and hence their computation is unstable. Rounding error incurred in forming the characteristic polynomial can destroy the accuracy of the roots subsequently computed. Computing the roots of a polynomial of high degree is another substantial task (indeed, one of the better ways of computing the roots of a polynomial is to compute the eigenvalues of its companion matrix using the methods we use for eigenvalues.

• Krylov Subspace Methods

  • Slides from Sanjeev Kumar of UCSD
  • Slides from Karl Meerbergen of WSC
  • Lecture from Slim Belhaiza of KFUPM (video)

• QR for Eigenvalues

  • Lecture from Martijn Anthonissen of TUe (video)
  • Notes from Tin-Yau Tam of Auburn

• Power method (or iteration)

  • Wikipedia entry
  • Video lecture from Jeffrey Chasnov: Part 1 and Part 2
  • Lecture from Jacob Bishop (video)
  • Lecture notes from of Jeffrey Wong of Duke

Solving nonlinear equations

• Secant method convergence rate is the golden ratio

• Lecture notes on the topic

  • Homer Reid
  • Kendall Atkinson (UIowa)
  • Fabrice Collard (Toulouse)
  • Online text from Fundamentals of Numerical Computation

• Lecture slides on the topic

  • Joseph Mahaffy of SDSU
  • George Kocur (MIT)
  • Alek Donev (NYU Courant)

• Visualizing 2D root solving

  • 2D Newton iteration from Daniel Korenblum
  • Another great example from SamuraiMath (video) on 2D Newton's method
  • 1D and 2D Bisection Method from Oscar Veliz (video)

• Systems of nonlinear equations from Jeff Chasnov (video)

• Code examples

  • Bisection, Newton's method, and more from Andreas Klöckner of UIUC (python)
  • 2D Newton's method from Hub on IT (python)
  • Broyden method from Tim Kelly (Matlab)

Optimization

• Notes

  • Great online notes from Leonid Kovalev of Syracuse including code examples in Matlab.
  • Theoretical and mathematical underpinnings of optimization from Angelia Nedić of ASU
  • The Golden section method from @NumericalGuy
  • A bit math-heavy lecture notes from Georgia Tech's Arkadi Nemirovski

• Video resources

  • Successive parabolic interpolation from Oscar Veliz
  • Comprehensive videos from Christopher Lum, see his other videos as well!

• Code examples

  • Newton's method, steepest descent, and more from Andreas Klöckner of UIUC (python)
  • Nelder-Mead from François Chollet (python)
  • Steepest decent, Newton's method, and more from Eirini Mitsopoulou (matlab)

Interpolation

• Online text from Fundamentals of Numerical Computation

• Splines and all that

  • What's the difference between splines and piecewise interpolants? Check out this SE.math Q/A and this one too
  • There are many types of splines. Check out Quasar Chunawala's comments on the topic!

• Orthogonal polynomials

  • Chebyshev interpolation notes from Nicholas Marshall of Oregon State
  • Runge's phenomenon (yes, that Runge) and the Chebyshev cure from John Cook
  • The zoology of orthogonal polynomials

• Great and rather comprehensive lecture notes from Leonid Kovalev on polynomial interpolation, Chebyshev polynomials, and splines

• Video resources

  • Spline examples and derivations from Jacob Bishop and The Math Guy
  • Bicubic interpolation with application to image scaling by the beloved Mike Pound (computerphile)

• Code examples

  • Monominals, orthogonal polynomials, and more from Andreas Klöckner of UIUC (python)
  • Chebfun! (a classic, in Matlab)

Numerical integration and quadrature

• Lecture notes and slides

  • Comprehensive lecture slides from Joseph Mahaffy (SDSU) on quadrature rules
  • Lecture notes on quadrature from Kevin Kreider of UAkron

• Video resources

  • Jeff Chasnov (youtube)

• Notes on the conditioning and stability of quadrature rules by Miriam Mehl of TUM

• Code examples

  • Newton-Cotes and Gaussian quadrature, finite differences from Andreas Klöckner of UIUC (python)
  • quadpy: A Python package for quadrature rules
  • Embedded notes and Newton-Cotes quadrature examples from Jennifer Klay, Matt Moelter, Jodi Christiansen, and more (python)
  • Trapazoidal rule in Matlab from Christoffer Bredo Lillelund
  • Embedded notes and quadrature examples in Matlab (interactive courseware) from Emma Smith Zbarsky

Initial value problems

• Videos from Prof. Steve Brunton (Washington) 

  • Introduction to the topic 
  • Forward and Backward Euler 
  • Runge-Kutta

• David Shirokoff on Euler's method

Ordinary differential equations

• 3Blue1Brown (video)
• Online text from Fundamentals of Numerical Computation

Partial differential equations

• 3Blue1Brown (video)