Introduction
-I have finally finished my undergrad and would like to make a blog post about what I have been working on these past ~6 months.
-The tile of our thesis is:
+ 14 min read
My BSc Thesis, ClaudesLens
Table of Contents
ClaudesLens
+Dubitatio dictat animas nostras
+Introduction
+I finally finished my undergrad and would like to make a blog post about what I have been working on these past ~6 months.
+The title of our thesis is:
@@ -236,12 +239,12 @@ClaudesLens: Uncertainty Quantification in Computer Vision Models
BayesLens
Even more ambitiously, apply this to self-driving cars.Needless to say, this was a bit too ambitious for a BSc thesis, so we had to scale down our project a bit. We didn’t have the prerequisite knowledge or the compute to do such a task within that time frame.
-So about ~1/3 into the project, when our supervisor wanted us to explore the entropy of predictions and got really excited about our results, we got ClaudesLens.
+Enough about what we wanted to do and what we did.
+So about ~1/3 into the project, our supervisor wanted us to explore the entropy of predictions, and we got promising results right from the start.
+This was the start of ClaudesLens.
ClaudesLens
-From the results using entropy as a measure of uncertainty — which in itself is nothing new — we decided to focus on this instead. -I’ll go into more detail and motivate how this approach works.
-Enough about what we wanted to do and what we did. -Let’s start what lies at the heart of this project: Neural Networks.
+From the results of using entropy as a measure of uncertainty — which in itself is nothing new — we decided to explore this further.
+Let’s start what lies at the heart of this project: Neural Networks.
Neural Networks
There are many ways to explain neural networks, in this post I will use a mathematical approach which will let us view the entire network as a single function.
The Neuron
@@ -524,22 +527,28 @@The Neuron
The Network
-Lets build upon what we now have learned and see how we can extend this.
+Let’s write the equation for a single neuron in a more compact form.
We can represent the inputs of a neuron as a vector, $$ +\begin{equation} \mathbf{x} = \left[x_1, x_2, \ldots, x_n\right], +\end{equation} $$
where each element corresponds to an input to the neuron.
Similarly, we can represent the associated weights as a vector, $$ +\begin{equation} \mathbf{w} = \left[w_1, w_2, \ldots, w_n\right], +\end{equation} $$
with this the summation can be simplified to a dot product, $$ +\begin{equation} z = \mathbf{w} \cdot \mathbf{x} + b. +\end{equation} $$
But only using one neuron will only get us so far, if we instead have multiple neurons and try to mimic the structure of the brain, we can get something more powerful.
-A layer is a collection of neurons, stacked on top of each other. +
A layer is a collection of neurons, stacked on top of each other. Very often when we are referring to a layer, are we referring to a fully connected layer, where each neuron in the layer is connected to all the neurons in the previous layer.
In the case of a network, we can now talk about the input layer and the output layer.
@@ -854,24 +863,30 @@
The Network
Figure 2: A simple neural network with one hidden layer.
-As we see in the picture, we now have multiple neurons with numerous inter-neuron connections, along with multiple outputs.
+As we see in Figure 2, we now have multiple neurons with numerous inter-neuron connections, along with multiple outputs.
The matrix-vector equation, $$ +\begin{equation} \mathbf{a} = \mathbf{W_1} \mathbf{x} + \mathbf{b_1} = [a_1, a_2, \ldots, a_m], +\end{equation} $$
yields each output of each neuron in the hidden layer (intermediate layers between the input and output layers).
Note, I’m explicitly leaving out the required transpose operations out of the equations, in reality we need our matrix and vectors to match dimension, but our theory and intuition will still hold.
-$\mathbf{W_1}$ is the weight matrix with rows $\mathbf{w_i} = [w_{i, 1}, w_{i, 2}, \ldots, w_{i, n}]$ corresponding to the weights of the $i$-th neuron in the hidden layer. -The bias values are represented by $\mathbf{b_1} = [b_1, b_2, \ldots, b_m]$.
+$\mathbf{W_1}$ is the weight matrix with rows $\mathbf{w_i} = [w_{i, 1}, w_{i, 2}, \ldots, w_{i, n}]$ corresponding to the weights of the $i$-th neuron in the hidden layer.
+The bias values are represented by $\mathbf{b_1} = [b_1, b_2, \ldots, b_m]$.
In the case of several layers, we work with multiple weight matrices and bias vectors, which we index as $\mathbf{W_j}$ and $\mathbf{b_j}$, respectively.
So, given an input $\mathbf{x}$, the output of the hidden layer (i.e., Figure 2) is given by, $$ +\begin{equation} \mathbf{a} = f.(\mathbf{W_1} \mathbf{x} + \mathbf{b_1}), +\end{equation} $$
where the dot indicates that the activation function $f$ is applied element-wise.
So the final output is therefore, $$ +\begin{equation} \mathbf{y} = f.(\mathbf{W_2} \mathbf{a} + \mathbf{b_2}). = f.(\mathbf{W_2} f.(\mathbf{W_1} \mathbf{x} + \mathbf{b_1}) + \mathbf{b_2}). +\end{equation} $$
Exercise: Is it necessary to apply the same activation function to all layers in a neural network?
@@ -902,7 +917,9 @@Uncertainty in Information Theory
5The entropy of a random variable $X$ is defined as, $$ +\begin{equation} H(X) = -\sum_{x \in \chi} p(x) \log p(x), +\end{equation} $$
where $p(x) = P(X = x)$.
If I were to explain Shannon-Entropy in an intuitive way. @@ -953,8 +970,8 @@
Entropy-based Uncert
So, let us introduce some uncertainty, let’s make our network stochastic.
To make the network stochastic, we need to introduce some randomness somewhere in the network.
I want to emphasize that you can do this in a lot of different ways and “inject” the randomness at different stages in a neural network. -We chose the most straightforward and primitive ways, adding image on the input image and adding noise to the weights.
-So, the perturbed weight matrix is given by,
+We chose the most straightforward and primitive ways, adding random noise to the input image or all weights. +So, the perturbed weight matrix is given by,
$$ \begin{equation} \mathbf{W}_{\sigma} = \mathbf{W} + \sigma \mathbf{N}, @@ -1060,14 +1077,17 @@
Mapping entropy categorically
A very important part of this framework is that the input $\mathbf{x}$ is fixed. Do images (given same perturbation level) with the same entropy yield similar predictions?
$$ +\begin{equation} p_{\sigma} = P(\hat{y_{\sigma}} = Y | H_{\sigma}(\mathbf{x}) = h), \quad \text{ where } h = H(\mathbf{x}). +\end{equation} $$
-The function mapping $\mathbf{x}$ to $h$ can be understood as the probability of making a correct prediciton within all draws from the data, which have the same entropy as $\mathbf{x}$.
-TO BE CONTINUED…
+The function mapping $\mathbf{x} \mapsto h$ can be understood as the probability of making a correct prediction within all draws from the data, which have the same entropy as $\mathbf{x}$. +This means we can categorize images based on their entropy and gain insight into the model’s predictions without seeing the ground truth label.
Conclusion
-This is a very brief overview of what we did in our thesis, and I hope that I have motivated why we did what we did. -I thought it was a very challenging but fun project, and I learned a lot from it.
-Read the paper as well, we basically died writing it.
+I hope you’ve understood this framework of quantifying uncertainty in neural networks.
+I’ve started to see a lot more people talk about uncertainty quantification 6 this year. +Which makes me happy, it is an interesting field.
+My closing statement is that, don’t forget that statistics is an exact science making sense of an uncertain and inexact world 7.
Footnotes
-
@@ -1085,6 +1105,12 @@
Conclusion
-
Claude E. Shannon, “A mathematical theory of communication.” ↩
+ - + + +
- + +