Skip to content

Latest commit

 

History

History
182 lines (107 loc) · 7.57 KB

README.md

File metadata and controls

182 lines (107 loc) · 7.57 KB
Raw

Deep Learning for Natural Language Processing - Lectures 2023

This repository contains slides for the course "20-00-0947: Deep Learning for Natural Language Processing" (Technical University of Darmstadt, Summer term 2023).

This course is jointly lectured by Ivan Habernal and Martin Tutek.

The slides are available as PDF as well as LaTeX source code (we've used Beamer because typesetting mathematics in PowerPoint or similar tools is painful). See the instructions below if you want to compile the slides yourselves.

Logo

The content is licensed under Creative Commons CC BY-SA 4.0 which means that you can re-use, adapt, modify, or publish it further, provided you keep the license and give proper credits.

Note: The following content is continuously updated as the summer term progresses. If you're interested in the full previous 2022 content, checkout the latest 2022 Git commit.

YouTube Playlist

Subscribe the YouTube playlist to get updates on new lectures: https://youtube.com/playlist?list=PL6WLGVNe6ZcA4gUr5MaAKdrGxYzYAETK3

Lecture 1: NLP tasks and evaluation

April 11, 2023

Lecture 2: Mathematical foundations of deep learning

April 18, 2023

Lecture 3: Text classification 1: Log-linear models

April 25, 2023

Lecture 4: Text classification 2: Deep neural networks

May 2, 2023

Lecture 5: Text generation 1: Language models and word embeddings

May 9, 2023

Lecture 6: Text classification 3: Learning word embeddings

May 16, 2023

Lecture 7: Text classification 4: Recurrent neural networks

May 30, 2023

Lecture 8: Text generation 2: Autoregressive encoder-decoder with RNNs and attention

June 6, 2023

Lecture 9: Text generation 3: Transformers

June 13, 2023

Lecture 10: Text classification 4: self-attention and BERT

June 20, 2023

Lecture 11: Text generation 4: Decoder-only Models and GPT

June 27, 2023

Lecture 12: Contemporary LLMs: Prompting and in-context learning

July 4, 2023

Lecture 13: Guest lecture by Dr. Thomas Arnold: Ethics of generative AI

July 11, 2023

Subtitles/Close caption

Thanks to Jan Kühnemund for generating the close caption for YouTube with Open Whisper. We track the subtitles here under subtitles, so if you spot an error there (there are many, such as "tanh" -> "10h"), just open a bug or PR.

FAQ

  • What are some essential pre-requisites?
    • Math: Derivatives and partial derivatives. We cover them in Lecture 2. If you need more, I would recommend these sources:
      • Jeremy Kun: A Programmer's Introduction to Mathematics. Absolutely amazing book. Pay-what-you-want for the PDF book. https://pimbook.org/
      • Deisenroth, A. Aldo Faisal, and Cheng Soon Ong: Mathematics for Machine Learning. Excellent resource, freely available. Might be a bit dense. https://mml-book.github.io/
  • Can I have the slide deck without "unfolding" the content over multiple pages?
    • You can compile the slides with the handout parameter, see below the section Compiling handouts.
  • Where do I find the code for plotting the functions?
    • Most of the plots are generated in Python/Jupyter (in Colab). The links are included as comments in the respective LaTeX sources for the slides.

Compiling slides to PDF

If you run a linux distribution (e.g., Ubuntu 20.04 and newer), all packages are provided as part of texlive. Install the following packages

$ sudo apt-get install texlive-latex-recommended texlive-pictures texlive-latex-extra \
texlive-fonts-extra texlive-bibtex-extra texlive-humanities texlive-science \
texlive-luatex biber wget -y

Install Fira Sans fonts required by the beamer template locally

$ wget https://github.com/mozilla/Fira/archive/refs/tags/4.106.zip -O 4.106.zip \
&& unzip -o 4.106.zip && mkdir -p ~/.fonts/FiraSans && cp Fira-4.106/otf/Fira* \
~/.fonts/FiraSans/ && rm -rf Fira-4.106 && rm 4.106.zip && fc-cache -f -v && mktexlsr

Compile each lecture's slides using lualatex

$ lualatex dl4nlp2023-lecture*.tex && biber dl4nlp2023-lecture*.bcf && \
lualatex dl4nlp2023-lecture*.tex && lualatex dl4nlp2023-lecture*.tex

Compiling slides using Docker

If you don't run a linux system or don't want to mess up your latex packages, I've tested compiling the slides in a Docker.

Install Docker ( https://docs.docker.com/engine/install/ )

Create a folder to which you clone this repository (for example, $ mkdir -p /tmp/slides)

Run Docker with Ubuntu 20.04 interactively; mount your slides directory under /mnt in this Docker container

$ docker run -it --rm --mount type=bind,source=/tmp/slides,target=/mnt \
ubuntu:20.04 /bin/bash

Once the container is running, update, install packages and fonts as above

# apt-get update && apt-get dist-upgrade -y && apt-get install texlive-latex-recommended \
texlive-pictures texlive-latex-extra texlive-fonts-extra texlive-bibtex-extra \
texlive-humanities texlive-science texlive-luatex biber wget -y

Fonts

# wget https://github.com/mozilla/Fira/archive/refs/tags/4.106.zip -O 4.106.zip \
&& unzip -o 4.106.zip && mkdir -p ~/.fonts/FiraSans && cp Fira-4.106/otf/Fira* \
~/.fonts/FiraSans/ && rm -rf Fira-4.106 && rm 4.106.zip && fc-cache -f -v && mktexlsr

And compile

# cd /mnt/dl4nlp/latex/lecture01
# lualatex dl4nlp2023-lecture*.tex && biber dl4nlp2023-lecture*.bcf && \
lualatex dl4nlp2023-lecture*.tex && lualatex dl4nlp2023-lecture*.tex

which generates the PDF in your local folder (e.g, /tmp/slides).

Compiling handouts

We're uploading the PDFs as presented in the lecture. You can compile the slides in a concise way using the handout settings. Just comment/uncomment the respective line at the beginning of the tex file of the lecture slides.