07-text-generation

Text Generation

[ Slides | Sketches ]

Objectives

• Learn about sequential data and Markov Chains.
• Learn about Recurrent Neural Network (RNN) architectures.
• Learn about transformer architecture.
• Learn to work with large language models (LLMs) in JavaScript.

Lecture Notes

Tools

• Tokenizer
  • OpenAI Tokenizer Demo
• Ollama
  • Ollama Open-Source Large Language Model Library
  • Ollama JavaScript Library
  • Unofficial Ollama JS Library
• OpenAI API
  • OpenAI API Reference
  • OpenAI API Playground

Related Projects

Text

• ITP Course Generator by Allison Parrish.
• Sunspring by Oscar Sharp, Ross Goodwin, et al.
• Writing with the Machine by Robin Sloan.
• Let's Read a Story by Itay Niv.
• 10 things artificial intelligence did in 2018 by Janelle Shane.

Drawing

• Double Agent by Simon Biggs.
• Four Experiments in Handwriting with a Neural Network by Shan Carter, et al.
• Handwriting Generation with RNN and p5.js by @hardmaru.

Music

• Magenta: Make Music and Art Using Machine Learning by Google AI.
• RNN for generating Baroque Music video by @carykh.

Coding

• Using ChatGPT to Implement Sol LeWitt's Wall Drawings by Amy Goodchild.

Code Example

Note: Ollama examples below can only be run locally in conjunction with Ollama.

Markov Chain

• Markov Chain Demo

Ollama

• Ollama - Minimal Example
• Ollama - Click for New Response without Context
• Ollama - Predetermined Prompts with Context
• Ollama - Chat Completion with Context

OpenAI

• OpenAI API - Chat Completion without Context
• OpenAI API - Chat Completion with Context
• OpenAI API - Image Generation

Supplemental Materials

Markov Chains

• Markov Chains by Victor Powell and Lewis Lehe.
• N-Grams and Markov Chains by Allison Parrish.

RNNs and LSTMs

• Understanding LSTM Networks by Christopher Olah.
• The Unreasonable Effectiveness of RNNs by Andrei Karpathy.
• Recurrent Neural Networks & LSTMs by Lenny Khazan and Rohan Kapur.

Transformers

• But what is a GPT? Visual intro to transformers by 3Blue1Brown.
• Attention in transformers, visually explained by 3Blue1Brown.
• The Foundation Model Transparency Index from Center for Research on Foundation Models at Stanford University.

Video Tutorials

Note: ml5.js tutorials below were taught using an older version of ml5.js, refer to the ml5.js Resources Wiki page for more information.

Text

• Markov Chains - Part 1 - video tutorial by Daniel Shiffman.
• Markov Chains - Part 2 - video tutorial by Daniel Shiffman.
• Context-Free Grammar - video tutorial by Daniel Shiffman.
• What is word2vec? - Programming with Text - video tutorial by Daniel Shiffman.

Drawing

• Sketch-RNN Snowflakes with ml5.js - video tutorial by Daniel Shiffman.
• Interactive Drawing with Machine Learning Model (SketchRNN) - video tutorial by Daniel Shiffman.
• RDP Line Simplification Algorithm - video tutorial by Daniel Shiffman.

Assignment 7

1. Read What Can Machine Learning Teach Us About Ourselves?, interview with Emily Martinez, ml5.js Fellow 2020.
2. Read The Subtext of a Black Corpus, in conversation with ITP research fellows Nikita Huggins & Ayodamola Okunseinde by Ashley Lewis.
3. Emily Martinez proposes a set of questions to ask related to working with a corpus of text data. Pick one (or two) of the questions to reflect on as you respond to the above two readings:
   • How can we be more intentional about what we build given the current limitations, problems, and constraints of ML algorithms?
   • How do we prepare datasets and set up guidelines that protect the bodies of knowledge of our communities, that honors lineage, that upholds ethical frameworks rooted in shared, agreed-upon values?
   • How do we work in consensual and respectful ways with texts by marginalized authors that are not as well-represented, and by virtue of that fact alone, much more likely to be misrepresented, misappropriated, or misunderstood if we are not careful?
   • How well can we ensure that the essence of these texts doesn’t dissolve into a word-soup that gets misconstrued?
   • Given that so many of the existing “big data” language models are trained with Western texts and proprietary datasets, what does it even mean to try to decolonize AI?
   • Who do we entrust to do this work?
   • How do we deal with credit and attribution of our new creations?
   • How do we really do ethics with machine learning?
   • How do we get through this whole list of concerns and still build AI that is fun, respectful, tender, pleasurable, kind?
4. Document your response to the readings in a blog post and add a link to the post on the Assignment 7 Wiki page.
5. Review the final project proposal guidelines and post your final project proposal and slides on the Final Proposal Wiki page.