- Learn about TensorFlow Face Landmarks Detection model and how it works.
- Learn about TensorFlow Hand Pose Detection model and how it works.
- Learn to work with ml5.faceMesh and ml5.handPose.
- Learn to work with 3D graphics in p5.js WEBGL mode.
- Keyboard by Use All Five & Google Creative Lab.
- Fingerspelling by Hello Monday.
- Projection Mapping by Nahuel Gerth.
- Customizable AR face masks - Made with TensorFlow.js by Samarth Gulati and Praveen Sinha.
- DeepPrivacy2 - A Toolbox for Realistic Image Anonymization by Hukkelås, Håkon and Lindseth, Frank.
- LipSync by YouTube by Google PI & bit.studio.
- Eye Conductor by Andreas Refsgaard.
- Sampler by Use All Five & Google Creative Lab.
- Mouth-Controlled Synthesizer with FaceMesh by Jack B. Du. [ Live Demo ]
- Melody Painter with HandPose by Jack B. Du. [ Live Demo ]
- Finger Talk by Future Sketches.
- Look At You by Alan Ren.
- Bubbles by Nahuel Gerth.
- Finger Numerals by Nahuel Gerth.
- FaceMesh - Single Image
- FaceMesh - Face Keypoints
- FaceMesh - Face Bounding Box
- FaceMesh - Parts Keypoints
- FaceMesh - Parts Bounding Box
- FaceMesh - Shapes from Parts
- HandPose - Single Image
- HandPose - Keypoints
- HandPose - Keypoints 3D
- HandPose - Parts
- HandPose - Detect Start and Stop
- FaceMesh - Emoji Face
- FaceMesh - Triangulation
- FaceMesh - Face Masks
- HandPose - Particles
- HandPose - Quadrilateral
- HandPose - Quadrilateral with Texture
- Face and hand tracking in the browser with MediaPipe and TensorFlow.js by Ann Yuan and Andrey Vakunov. [ Original Paper ]
- On-Device, Real-Time Hand Tracking with MediaPipe by Valentin Bazarevsky and Fan Zhang.
- 3D Hand Pose with MediaPipe and TensorFlow.js by Valentin Bazarevsky, Ivan Grishchenko, Eduard Gabriel Bazavan, Andrei Zanfir, Mihai Zanfir, Jiuqiang Tang, Jason Mayes, Ahmed Sabie.
- ml5.js FaceMesh Creative Sketch - video tutorial by Daniel Shiffman and Patt Vira.
- Research and find a project (experiments, websites, art installations, games, etc) that utilizes machine learning in a creative way. Consider the following:
- What type of machine learning models did the creator use?
- What data might have been used to train the machine learning model?
- Why did the creator of the project choose to use this machine learning model?
- Pick one of the models above (FaceMesh or HandPose), following the examples above and the ml5.js documentation, experiment with controlling elements of a p5.js sketch (color, geometry, sound, text, etc) with the output of the model. Try to create an interaction that is surprising or one that is inspired by the project you find.
- Document your research, your response, and your p5.js sketch in a blog post. Add a link to the post and your p5.js sketch on the Assignment 4 Wiki page. In your blog post, include visual documentation such as a recorded screen capture / video / GIFs of your sketch.