From 7410d409cf712ab048c76e3da94470d24c2761bd Mon Sep 17 00:00:00 2001 From: GitHub Actions Date: Mon, 22 Jul 2024 12:48:16 +0000 Subject: [PATCH] site deploy Auto-generated via {sandpaper} Source : 1496a749e14e3275ab557cce796579997bb0ff61 Branch : md-outputs Author : GitHub Actions Time : 2024-07-22 12:47:57 +0000 Message : markdown source builds Auto-generated via {sandpaper} Source : 48804282ac6e8c1ca683fb68737341790a6ee226 Branch : main Author : Carsten Schnober Time : 2024-07-22 12:46:53 +0000 Message : Merge pull request #498 from carpentries-incubator/resolve-comments-mike-ivs-part-II Properly introduce activation functions --- 1-introduction.html | 160 ++++++++++++----------- 3-monitor-the-model.html | 16 +-- aio.html | 185 ++++++++++++++------------ images.html | 24 ++-- index.html | 10 +- instructor/1-introduction.html | 162 ++++++++++++----------- instructor/3-monitor-the-model.html | 22 ++-- instructor/aio.html | 193 +++++++++++++++------------- instructor/images.html | 24 ++-- instructor/index.html | 14 +- md5sum.txt | 2 +- pkgdown.yml | 2 +- 12 files changed, 440 insertions(+), 374 deletions(-) diff --git a/1-introduction.html b/1-introduction.html index 391652ff..961cf575 100644 --- a/1-introduction.html +++ b/1-introduction.html @@ -392,27 +392,100 @@

Deep Learnin
An infographic showing the relation of artificial intelligence, machine learning, and deep learning. Deep learning is a specific subset of machine learning algorithms. Machine learning is one of the approaches to artificial intelligence.

Neural Networks

A neural network is an artificial intelligence technique loosely -based on the way neurons in the brain work.

+based on the way neurons in the brain work. A neural network consists of +connected computational units called neurons. Let’s +look at the operations of a single neuron.

A single neuron
-

A neural network consists of connected computational units called -neurons. Each neuron …

+

Each neuron …

  • has one or more inputs (\(x_1, x_2, ...\)), e.g. input data expressed as floating point numbers
  • most of the time, each neuron conducts 3 main operations: -
    • take the weighted sum of the inputs where ($w_1, w_2, … $) indicate -weights
      • +
      • take the weighted sum of the inputs where (\(w_1, w_2, ...\)) indicate weights
      • add an extra constant weight (i.e. a bias term) to this weighted sum
        • -
      • apply a non-linear function to the output so far (using a predefined -activation function such as the ReLU function)
        • +
      • apply an activation function to the output so far, +we will explain activation functions
    • return one output value, again a floating point number.
      • -
    • one example equation to calculate the output for a neuron is: \(output = ReLU(\sum_{i} (x_i*w_i) + +
    • one example equation to calculate the output for a neuron is: \(output = Activation(\sum_{i} (x_i*w_i) + bias)\)
    A diagram of a single artificial neuron combining inputs and weights using an activation function.
+
Activation functions
+

The goal of the activation function is to convert the weighted sum of +the inputs to the output signal of the neuron. This output is then +passed on to the next layer of the network. There are many different +activation functions, 3 of them are introduced in the exercise +below.

+
+
+ +
+
+

Activation functions +

+
+

Look at the following activation functions:

+

A. Sigmoid activation function The sigmoid +activation function is given by: \[ f(x) = +\frac{1}{1 + e^{-x}} \]

+

Plot of the sigmoid function

+

B. ReLU activation function The Rectified Linear +Unit (ReLU) activation function is defined as: \[ f(x) = \max(0, x) \]

+

This involves a simple comparison and maximum calculation, which are +basic operations that are computationally inexpensive. It is also simple +to compute the gradient: 1 for positive inputs and 0 for negative +inputs.

+

Plot of the ReLU function

+

C. Linear (or identity) activation function +(output=input) The linear activation function is simply the +identity function: \[ f(x) = x \]

+

Plot of the Identity function

+

Combine the following statements to the correct activation +function:

+
  1. This function enforces the activation of a neuron to be between 0 +and 1
    2. +
  2. This function is useful in regression tasks when applied to an +output neuron
    3. +
  3. This function is the most popular activation function in hidden +layers, since it introduces non-linearity in a computationally efficient +way.
    4. +
  4. This function is useful in classification tasks when applied to an +output neuron
    5. +
  5. (optional) For positive values this function results in the same +activations as the identity function.
    6. +
  6. (optional) This function is not differentiable at 0
    7. +
  7. (optional) This function is the default for Dense layers (search the +Keras documentation!)
    8. +

Activation function plots by Laughsinthestocks - Own work, CC +BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=44920411, +https://commons.wikimedia.org/w/index.php?curid=44920600, +https://commons.wikimedia.org/w/index.php?curid=44920533

+
+
+
+
+
+ +
+
+
  1. A
    2. +
  2. C
    3. +
  3. B
    4. +
  4. A
    5. +
  5. B
    6. +
  6. B
    7. +
  7. C
    8. +
+
+
+
+
+
Combining multiple neurons into a network

Multiple neurons can be joined together by connecting the output of one to the input of another. These connections are associated with @@ -476,12 +549,12 @@

2. (optional) Calculate output

-
+
- -
+

1: calculate the output for one neuron

@@ -520,63 +593,6 @@

2: Calculate outputs for a network

-
-
- -
-
-

Activation functions -

-
-

Look at the following activation functions:

-

A. Sigmoid activation function

-

Plot of the sigmoid function

-

B. ReLU activation function

-

Plot of the ReLU function

-

C. Identity (or linear) activation function -(output=input)

-

Plot of the Identity function

-

Combine the following statements to the correct activation -function:

-
  1. This function enforces the activation of a neuron to be between 0 -and 1
    2. -
  2. This function is useful in regression tasks when applied to an -output neuron
    3. -
  3. This function is the most popular activation function in hidden -layers, since it introduces non-linearity in a computationally efficient -way.
    4. -
  4. This function is useful in classification tasks when applied to an -output neuron
    5. -
  5. (optional) For positive values this function results in the same -activations as the identity function.
    6. -
  6. (optional) This function is not differentiable at 0
    7. -
  7. (optional) This function is the default for Dense layers (search the -Keras documentation!)
    8. -

Activation function plots by Laughsinthestocks - Own work, CC -BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=44920411, -https://commons.wikimedia.org/w/index.php?curid=44920600, -https://commons.wikimedia.org/w/index.php?curid=44920533

-
-
-
-
-
- -
-
-
  1. A
    2. -
  2. C
    3. -
  3. B
    4. -
  4. A
    5. -
  5. B
    6. -
  6. B
    7. -
  7. C
    8. -
-
-
-
What makes deep learning deep learning?
@@ -684,7 +700,7 @@

2. (optional) Huber loss

@@ -811,7 +827,7 @@

Deep Learning Problems Exercise