02-transfer-learning

Transfer Learning

[ Slides | Sketches ]

Objectives

• Diagram and define the layers of a neural network, introducing the concept of a “convolutional layer”.
• Distinguish between a “feature vector” (aka logits) and the last layer (aka softmax probabilities) of a classification network.
• Understand the process of “transfer learning”.
• Learn about k-nearest neighbors algorithm (k-NN).
• Learn to save a trained model for re-use later.

Lecture Notes

Tools

• Teachable Machine Website
  • Teachable Machine 2.0: Making AI easier for everyone from Google.
  • Image + Teachable Machine Reference from ml5.js Documentation.
  • Teachable Machine Community Repo
• ml5.soundClassifer Reference
  • ml5.soundClassifier Source Code
  • TensorFlow.js Speech Command Recognizer Source Code
• ml5.featureExtractor Archived Reference
  • ml5.featureExtractor Source Code
• ml5.KNNClassifier Archived Reference
  • ml5.KNNClassifier Source Code
  • TensorFlow.js KNN Classifier Source Code

Code Examples

• Image Classification with Teachable Machine and ml5.js
• Sound Classification with Teachable Machine and ml5.js
• Sound Classification - Speech Commands
• Webcam Image Classification with Teachable Machine Demo hosted on GitHub. [ Source Code ]

Related Projects

Accessibility

• Getting Alexa to Respond to Sign Language Using Your Webcam and TensorFlow.js by Abhishek Singh. [ Live Demo ]
• Project Euphonia from Google.
• Objectifier Spatial Programming by Bjørn Karmann.

Games

• Webcam Pacman by Google’s TensorFlow.js Team.
• Pong ML by Alejandro Matamala Ortiz.
• Teachable Snake by Vince MingPu Shao.
• Teachable Arcade by Ryan Flomerfelt Mather.

Playful

• Tiny Sorter by Google Creative Lab.
• Eyeo Festival 2019 - Coding Train with Daniel Shiffman. [ GitHub Repo ]

Visuals

• Regression Curve by Joohyun Park.
• Asemic Writing Teachable Machine by Jillian Zhong.

Supplemental Materials

• Local Development and Hosting Resources Wiki Page by Jack B. Du.
• But what is a Neural Network? by 3Blue1Brown.
• How to Build a Teachable Machine with TensorFlow.js by Nikhil Thorat.

Video Tutorials

Note: ml5.js tutorials below were taught using an older version of ml5.js, refer to the ml5.js Resources Wiki page for more information.

Teachable Machine

• Teachable Machine video tutorials from Experiments with Google.
• Teachable Machine video tutorials by Daniel Shiffman.

Transfer Learning with ml5.js

• ml5.js: Transfer Learning with Feature Extractor - video tutorial by Daniel Shiffman.
• ml5.js: Feature Extractor Classification - video tutorial by Daniel Shiffman.
• ml5.js: Feature Extractor Regression - video tutorial by Daniel Shiffman.
• ml5.js: Save/Load Model - video tutorial by Daniel Shiffman.

KNN Classification with ml5.js

• ml5.js: KNN Classification Part 1 by Daniel Shiffman.
• ml5.js: KNN Classification Part 2 by Daniel Shiffman.
• ml5.js: KNN Classification Part 3 by Daniel Shiffman.

GitHub Pages

• Hosting a p5.js sketch with GitHub Pages - video tutorial by Daniel Shiffman.

Assignment 2

1. Read Excavating AI: The Politics of Images in Machine Learning Training Sets by Kate Crawford and Trevor Paglen. Consider the following excerpt from the conclusion:

The artist René Magritte completed a painting of a pipe and coupled it with the words “Ceci n’est pas une pipe.” Magritte called the painting La trahison des images, “The Treachery of Images.”

Magritte’s assumption was almost diametrically opposed: that images in and of themselves have, at best, a very unstable relationship to the things seem to represent, one that can be sculpted by whoever has the power to say what a particular image means. For Magritte, the meaning of images is relational, open to contestation. At first blush, Magritte’s painting might seem like a simple semiotic stunt, but the underlying dynamic Magritte underlines in the painting points to a much broader politics of representation and self-representation.

Reflect on the relationship between labels and images in a machine learning image classification dataset. Who has the power to label images and how do those labels and machine learning models trained on them impact society?

2. Train your own image classifier using Teachable Machine and apply the model to an interactive p5.js sketch. Feel free to try sound instead of or in addition to images.

3. Document your reading reflections and your coding exercise in a blog post and add a link to the post and your p5 sketch on the Assignment 2 Wiki page. In your blog post, include visual documentation such as a recorded screen capture / video / GIFs of training the model and working in p5. You may include both in one post or write two separate posts if you prefer.