Merge remote-tracking branch 'origin/main' into fix/minor_comments_re…

…view_sarah

episodes/1-introduction.Rmd

@@ -7,24 +7,26 @@ exercises: 30
 :::::::::::::::::::::::::::::::::::::: questions
 
 - What is deep learning?
+- What is a neural network?
+- Which operations are performed by a single neuron?
+- How do neural networks learn?
 - When does it make sense to use and not use deep learning?
-- When is it successful?
-- What are the tools involved?
+- What are tools involved in deep learning?
 - What is the workflow for deep learning?
 - Why did we choose to use Keras in this lesson?
-- How do neural networks learn?
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
 :::::::::::::::::::::::::::::::::::::: objectives
 
+- Define deep learning
+- Describe how a neural network is build up
+- Explain the operations performed by a single neuron
+- Describe what a loss function is
 - Recall the sort of problems for which deep learning is a useful tool
 - List some of the available tools for deep learning
 - Recall the steps of a deep learning workflow
-- Identify the inputs and outputs of a deep neural network.
-- Explain the operations performed in a single neuron
-- Test that you have correctly installed the Keras, Seaborn and Sklearn libraries
-- Describe what a loss function is
+- Test that you have correctly installed the Keras, Seaborn and scikit-learn libraries
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
@@ -38,32 +40,92 @@ Deep learning (DL) is just one of many techniques collectively known as machine
 The image below shows some differences between artificial intelligence, machine learning and deep learning.
 
 
-![
-Image credit: Tukijaaliwa, CC BY-SA 4.0, via Wikimedia Commons, 
-[original source]( https://en.wikipedia.org/wiki/File:AI-ML-DL.svg)
-](fig/01_AI_ML_DL_differences.png){
-alt='An infographic showing the relation of AI, ML, NN and DL. NN are methods in DL which is a subset of ML algorithms that falls within the umbrella of AI'
+![](fig/01_AI_ML_DL_differences.jpg){
+alt='An infographic showing the relation of artificial intelligence, machine learning, and deep learning. Deep learning is a specific subset of machine learning algorithms. Machine learning is one of the approaches to artificial intelligence.'
+width='60%'
 }
 
 
 #### Neural Networks
 
-A neural network is an artificial intelligence technique loosely based on the way neurons in the brain work. 
+A neural network is an artificial intelligence technique loosely based on the way neurons in the brain work.
+A neural network consists of connected computational units called **neurons**.
+Let's look at the operations of a single neuron.
 
 ##### A single neuron
-A neural network consists of connected computational units called **neurons**. Each neuron ...
+ Each neuron ...
 
 - has one or more inputs ($x_1, x_2, ...$), e.g. input data expressed as floating point numbers
 - most of the time, each neuron conducts 3 main operations:
-  + take the weighted sum of the inputs where ($w_1, w_2, ... $) indicate weights
+  + take the weighted sum of the inputs where ($w_1, w_2, ...$) indicate weights
   + add an extra constant weight (i.e. a bias term) to this weighted sum
-  + apply a non-linear function to the output so far (using a predefined activation function such as the [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) function)
+  + apply an **activation function** to the output so far, we will explain activation functions
 - return one output value, again a floating point number.
-- one example equation to calculate the output for a neuron is: $output = ReLU(\sum_{i} (x_i*w_i) + bias)$
+- one example equation to calculate the output for a neuron is: $output = Activation(\sum_{i} (x_i*w_i) + bias)$
 
 
 ![](fig/01_neuron.png){alt='A diagram of a single artificial neuron combining inputs and weights using an activation function.' width='600px'}
 
+##### Activation functions
+The goal of the activation function is to convert the weighted sum of the inputs to the output signal of the neuron.
+This output is then passed on to the next layer of the network.
+There are many different activation functions, 3 of them are introduced in the exercise below.
+
+::: challenge
+## Activation functions
+Look at the following activation functions:
+
+**A. Sigmoid activation function**
+The sigmoid activation function is given by:
+$$ f(x) = \frac{1}{1 + e^{-x}} $$
+
+![](fig/01_sigmoid.svg){alt='Plot of the sigmoid function' width='70%' align='left'}
+<br clear="all" />
+
+**B. ReLU activation function**
+The Rectified Linear Unit (ReLU) activation function is defined as:
+$$ f(x) = \max(0, x) $$
+
+This involves a simple comparison and maximum calculation, which are basic operations that are computationally inexpensive.
+It is also simple to compute the gradient: 1 for positive inputs and 0 for negative inputs.
+
+![](fig/01_relu.svg){alt='Plot of the ReLU function'  width='70%' align='left'}
+<br clear="all" />
+
+**C. Linear (or identity) activation function (output=input)**
+The linear activation function is simply the identity function:
+$$ f(x) = x $$
+
+![](fig/01_identity_function.svg){alt='Plot of the Identity function'  width='70%' align='left'}
+<br clear="all" />
+
+
+Combine the following statements to the correct activation function:
+
+1. This function enforces the activation of a neuron to be between 0 and 1
+2. This function is useful in regression tasks when applied to an output neuron
+3. This function is the most popular activation function in hidden layers, since it introduces non-linearity in a computationally efficient way.
+4. This function is useful in classification tasks when applied to an output neuron
+5. (optional) For positive values this function results in the same activations as the identity function.
+6. (optional) This function is not differentiable at 0
+7. (optional) This function is the default for Dense layers (search the Keras documentation!)
+
+*Activation function plots by Laughsinthestocks - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=44920411,
+https://commons.wikimedia.org/w/index.php?curid=44920600, https://commons.wikimedia.org/w/index.php?curid=44920533*
+
+:::: solution
+## Solution
+1. A
+2. C
+3. B
+4. A
+5. B
+6. B
+7. C
+::::
+:::
+
+
 ##### Combining multiple neurons into a network
 Multiple neurons can be joined together by connecting the output of one to the input of another. These connections are associated with weights that determine the 'strength' of the connection, the weights are adjusted during training. In this way, the combination of neurons and connections describe a computational graph, an example can be seen in the image below.
 
@@ -136,52 +198,6 @@ b. This solves the XOR logical problem, the output is 1 if only one of the two i
 ::::
 :::
 
-::: challenge
-## Activation functions
-Look at the following activation functions:
-
-**A. Sigmoid activation function**
-
-![](fig/01_sigmoid.svg){alt='Plot of the sigmoid function' width='70%' align='left'}
-<br clear="all" />
-
-**B. ReLU activation function**
-
-![](fig/01_relu.svg){alt='Plot of the ReLU function'  width='70%' align='left'}
-<br clear="all" />
-
-**C. Identity (or linear) activation function (output=input)**
-
-![](fig/01_identity_function.svg){alt='Plot of the Identity function'  width='70%' align='left'}
-<br clear="all" />
-
-
-Combine the following statements to the correct activation function:
-
-1. This function enforces the activation of a neuron to be between 0 and 1
-2. This function is useful in regression tasks when applied to an output neuron
-3. This function is the most popular activation function in hidden layers, since it introduces non-linearity in a computationally efficient way.
-4. This function is useful in classification tasks when applied to an output neuron
-5. (optional) For positive values this function results in the same activations as the identity function.
-6. (optional) This function is not differentiable at 0
-7. (optional) This function is the default for Dense layers (search the Keras documentation!)
-
-*Activation function plots by Laughsinthestocks - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=44920411,
-https://commons.wikimedia.org/w/index.php?curid=44920600, https://commons.wikimedia.org/w/index.php?curid=44920533*
-
-:::: solution
-## Solution
-1. A
-2. C
-3. B
-4. A
-5. B
-6. B
-7. C
-::::
-:::
-
-
 ##### What makes deep learning deep learning?
 Neural networks aren't a new technique, they have been around since the late 1940s. But until around 2010 neural networks tended to be quite small, consisting of only 10s or perhaps 100s of neurons. This limited them to only solving quite basic problems. Around 2010, improvements in computing power and the algorithms for training the networks made much larger and more powerful networks practical. These are known as deep neural networks or deep learning.
 
@@ -440,7 +456,7 @@ Keras also benefits from a very good set of [online documentation](https://keras
 
 ### Installing Keras and other dependencies
 
-Follow the instructions in the [setup]({{ page.root }}//setup) document to install Keras, Seaborn and Sklearn.
+Follow the instructions in the [setup]({{ page.root }}//setup) document to install Keras, Seaborn and scikit-learn.
 
 ## Testing Keras Installation
 Keras is available as a module within TensorFlow, as described in the [setup]({{ page.root }}//setup).
@@ -467,8 +483,8 @@ print(seaborn.__version__)
 ```
 You should get a version number reported. At the time of writing 0.13.2 is the latest version.
 
-## Testing Sklearn Installation
-Lets check you have a suitable version of sklearn installed.
+## Testing scikit-learn Installation
+Lets check you have a suitable version of scikit-learn installed.
 In your Jupyter notebook or interactive python console run the following commands:
 ```python
 import sklearn

episodes/2-keras.Rmd

@@ -5,7 +5,6 @@ exercises: 50
 ---
 
 ::: questions
-- What is a neural network?
 - How do I compose a neural network using Keras?
 - How do I train this network on a dataset?
 - How do I get insight into learning process?
@@ -15,6 +14,7 @@ exercises: 50
 ::: objectives
 - Use the deep learning workflow to structure the notebook
 - Explore the dataset using pandas and seaborn
+- Identify the inputs and outputs of a deep neural network.
 - Use one-hot encoding to prepare data for classification in Keras
 - Describe a fully connected layer
 - Implement a fully connected layer with Keras

episodes/3-monitor-the-model.Rmd

@@ -55,6 +55,14 @@ years 2000 to 2010. For all locations the data contains the variables ‘mean te
 ### Import Dataset
 We will now import and explore the weather data-set:
 
+::: callout
+## Load the data
+If you have not downloaded the data yet, you can also load it directly from Zenodo:
+```python
+data = pd.read_csv("https://zenodo.org/record/5071376/files/weather_prediction_dataset_light.csv?download=1")
+```
+:::
+
 ```python
 import pandas as pd
 
@@ -72,15 +80,6 @@ data.head()
 |4| 	20000105 	|1 	|5 	|0.90 	|1.0246|... |
 
 
-::: callout
-## Load the data
-If you have not downloaded the data yet, you can also load it directly from Zenodo:
-```python
-data = pd.read_csv("https://zenodo.org/record/5071376/files/weather_prediction_dataset_light.csv?download=1")
-```
-:::
-
-
 ### Brief exploration of the data
 Let us start with a quick look at the type of features that we find in the data.
 ```python

instructors/instructor-notes.md

@@ -14,11 +14,6 @@ The episodes are quite long, because they cover a full cycle of the deep learnin
 ## Episode 3: Monitor the training process
 When episode 3 is taught on a different day then episode 2, it is very useful to start with a recap of episode 2. The Key Points of episode 2 can be iterated, and you can go through the code of the previous session (without actually running it). This will help learners in the big exercise on creating a neural network.
 
-If learners did not download the data yet, they can also load the data directly from zenodo (instead of first downloading and saving):
-```python
-data = pd.read_csv("https://zenodo.org/record/5071376/files/weather_prediction_dataset_light.csv?download=1")
-```
-
 The following exercises work well to do in groups / break-out rooms:
 - Split data into training, validation, and test set
 - Create the neural network. Note that this is a fairly challenging exercise, but learners should be able to do this based on their experiences in episode 2 (see also remark about recap).