Skip to content

Commit

Permalink
update week 16
Browse files Browse the repository at this point in the history
  • Loading branch information
@mhjensen
mhjensen committed May 1, 2024
1 parent 85439c6 commit bb3d370
Show file tree
Hide file tree
Showing 8 changed files with 477 additions and 27 deletions.
93 changes: 87 additions & 6 deletions doc/pub/week16/html/week16-bs.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,33 @@
2,
None,
'plans-for-the-week-of-may-6-10-2024'),
('Diffusion models', 2, None, 'diffusion-models'),
('Summary of course and topics covered',
('Diffusion models, basics', 2, None, 'diffusion-models-basics'),
('Problems with probabilistic models',
2,
None,
'summary-of-course-and-topics-covered')]}
'problems-with-probabilistic-models'),
('Diffusion models', 2, None, 'diffusion-models'),
('Original idea', 2, None, 'original-idea'),
('Diffusion learning', 2, None, 'diffusion-learning')]}
end of tocinfo -->

<body>



<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>


<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
Expand All @@ -64,8 +83,11 @@
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#plans-for-the-week-of-may-6-10-2024" style="font-size: 80%;">Plans for the week of May 6-10, 2024</a></li>
<!-- navigation toc: --> <li><a href="#diffusion-models-basics" style="font-size: 80%;">Diffusion models, basics</a></li>
<!-- navigation toc: --> <li><a href="#problems-with-probabilistic-models" style="font-size: 80%;">Problems with probabilistic models</a></li>
<!-- navigation toc: --> <li><a href="#diffusion-models" style="font-size: 80%;">Diffusion models</a></li>
<!-- navigation toc: --> <li><a href="#summary-of-course-and-topics-covered" style="font-size: 80%;">Summary of course and topics covered</a></li>
<!-- navigation toc: --> <li><a href="#original-idea" style="font-size: 80%;">Original idea</a></li>
<!-- navigation toc: --> <li><a href="#diffusion-learning" style="font-size: 80%;">Diffusion learning</a></li>

</ul>
</li>
Expand Down Expand Up @@ -116,13 +138,72 @@ <h2 id="plans-for-the-week-of-may-6-10-2024" class="anchor">Plans for the week o
</div>


<!-- !split -->
<h2 id="diffusion-models-basics" class="anchor">Diffusion models, basics </h2>

<p>Diffusion models are inspired by non-equilibrium thermodynamics. They
define a Markov chain of diffusion steps to slowly add random noise to
data and then learn to reverse the diffusion process to construct
desired data samples from the noise. Unlike VAE or flow models,
diffusion models are learned with a fixed procedure and the latent
variable has high dimensionality (same as the original data).
</p>

<!-- !split -->
<h2 id="problems-with-probabilistic-models" class="anchor">Problems with probabilistic models </h2>

<p>Historically, probabilistic models suffer from a tradeoff between two
conflicting objectives: \textit{tractability} and
\textit{flexibility}. Models that are \textit{tractable} can be
analytically evaluated and easily fit to data (e.g. a Gaussian or
Laplace). However, these models are unable to aptly describe structure
in rich datasets. On the other hand, models that are \textit{flexible}
can be molded to fit structure in arbitrary data. For example, we can
define models in terms of any (non-negative) function \( \phi(\boldsymbol{x}) \)
yielding the flexible distribution \( p\left(\boldsymbol{x}\right) =
\frac{\phi\left(\boldsymbol{x} \right)}{Z} \), where \( Z \) is a normalization
constant. However, computing this normalization constant is generally
intractable. Evaluating, training, or drawing samples from such
flexible models typically requires a very expensive Monte Carlo
process.
</p>

<!-- !split -->
<h2 id="diffusion-models" class="anchor">Diffusion models </h2>
<p>Diffusion models have several interesting features</p>
<ul>
<li> extreme flexibility in model structure,</li>
<li> exact sampling,</li>
<li> easy multiplication with other distributions, e.g. in order to compute a posterior, and</li>
<li> the model log likelihood, and the probability of individual states, to be cheaply evaluated.</li>
</ul>
<!-- !split -->
<h2 id="original-idea" class="anchor">Original idea </h2>

<p>Text to come here.</p>
<p>In the original formulation, one uses a Markov chain to gradually
convert one distribution into another, an idea used in non-equilibrium
statistical physics and sequential Monte Carlo. Diffusion models build
a generative Markov chain which converts a simple known distribution
(e.g. a Gaussian) into a target (data) distribution using a diffusion
process. Rather than use this Markov chain to approximately evaluate a
model which has been otherwise defined, one can explicitly define the
probabilistic model as the endpoint of the Markov chain. Since each
step in the diffusion chain has an analytically evaluable probability,
the full chain can also be analytically evaluated.
</p>

<!-- !split -->
<h2 id="summary-of-course-and-topics-covered" class="anchor">Summary of course and topics covered </h2>
<h2 id="diffusion-learning" class="anchor">Diffusion learning </h2>

<p>Learning in this framework involves estimating small perturbations to
a diffusion process. Estimating small, analytically tractable,
perturbations is more tractable than explicitly describing the full
distribution with a single, non-analytically-normalizable, potential
function. Furthermore, since a diffusion process exists for any
smooth target distribution, this method can capture data distributions
of arbitrary form.
</p>

<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
Expand Down
82 changes: 80 additions & 2 deletions doc/pub/week16/html/week16-reveal.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,22 @@




<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>




<section>
<!-- ------------------- main content ---------------------- -->
<center>
Expand Down Expand Up @@ -192,14 +208,76 @@ <h2 id="plans-for-the-week-of-may-6-10-2024">Plans for the week of May 6-10, 202
</center>
</section>

<section>
<h2 id="diffusion-models-basics">Diffusion models, basics </h2>

<p>Diffusion models are inspired by non-equilibrium thermodynamics. They
define a Markov chain of diffusion steps to slowly add random noise to
data and then learn to reverse the diffusion process to construct
desired data samples from the noise. Unlike VAE or flow models,
diffusion models are learned with a fixed procedure and the latent
variable has high dimensionality (same as the original data).
</p>
</section>

<section>
<h2 id="problems-with-probabilistic-models">Problems with probabilistic models </h2>

<p>Historically, probabilistic models suffer from a tradeoff between two
conflicting objectives: \textit{tractability} and
\textit{flexibility}. Models that are \textit{tractable} can be
analytically evaluated and easily fit to data (e.g. a Gaussian or
Laplace). However, these models are unable to aptly describe structure
in rich datasets. On the other hand, models that are \textit{flexible}
can be molded to fit structure in arbitrary data. For example, we can
define models in terms of any (non-negative) function \( \phi(\boldsymbol{x}) \)
yielding the flexible distribution \( p\left(\boldsymbol{x}\right) =
\frac{\phi\left(\boldsymbol{x} \right)}{Z} \), where \( Z \) is a normalization
constant. However, computing this normalization constant is generally
intractable. Evaluating, training, or drawing samples from such
flexible models typically requires a very expensive Monte Carlo
process.
</p>
</section>

<section>
<h2 id="diffusion-models">Diffusion models </h2>
<p>Diffusion models have several interesting features</p>
<ul>
<p><li> extreme flexibility in model structure,</li>
<p><li> exact sampling,</li>
<p><li> easy multiplication with other distributions, e.g. in order to compute a posterior, and</li>
<p><li> the model log likelihood, and the probability of individual states, to be cheaply evaluated.</li>
</ul>
</section>

<p>Text to come here.</p>
<section>
<h2 id="original-idea">Original idea </h2>

<p>In the original formulation, one uses a Markov chain to gradually
convert one distribution into another, an idea used in non-equilibrium
statistical physics and sequential Monte Carlo. Diffusion models build
a generative Markov chain which converts a simple known distribution
(e.g. a Gaussian) into a target (data) distribution using a diffusion
process. Rather than use this Markov chain to approximately evaluate a
model which has been otherwise defined, one can explicitly define the
probabilistic model as the endpoint of the Markov chain. Since each
step in the diffusion chain has an analytically evaluable probability,
the full chain can also be analytically evaluated.
</p>
</section>

<section>
<h2 id="summary-of-course-and-topics-covered">Summary of course and topics covered </h2>
<h2 id="diffusion-learning">Diffusion learning </h2>

<p>Learning in this framework involves estimating small perturbations to
a diffusion process. Estimating small, analytically tractable,
perturbations is more tractable than explicitly describing the full
distribution with a single, non-analytically-normalizable, potential
function. Furthermore, since a diffusion process exists for any
smooth target distribution, this method can capture data distributions
of arbitrary form.
</p>
</section>


Expand Down
88 changes: 83 additions & 5 deletions doc/pub/week16/html/week16-solarized.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,33 @@
2,
None,
'plans-for-the-week-of-may-6-10-2024'),
('Diffusion models', 2, None, 'diffusion-models'),
('Summary of course and topics covered',
('Diffusion models, basics', 2, None, 'diffusion-models-basics'),
('Problems with probabilistic models',
2,
None,
'summary-of-course-and-topics-covered')]}
'problems-with-probabilistic-models'),
('Diffusion models', 2, None, 'diffusion-models'),
('Original idea', 2, None, 'original-idea'),
('Diffusion learning', 2, None, 'diffusion-learning')]}
end of tocinfo -->

<body>



<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>


<!-- ------------------- main content ---------------------- -->
<center>
<h1>Advanced machine learning and data analysis for the physical sciences</h1>
Expand Down Expand Up @@ -111,13 +130,72 @@ <h2 id="plans-for-the-week-of-may-6-10-2024">Plans for the week of May 6-10, 202
</div>


<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="diffusion-models-basics">Diffusion models, basics </h2>

<p>Diffusion models are inspired by non-equilibrium thermodynamics. They
define a Markov chain of diffusion steps to slowly add random noise to
data and then learn to reverse the diffusion process to construct
desired data samples from the noise. Unlike VAE or flow models,
diffusion models are learned with a fixed procedure and the latent
variable has high dimensionality (same as the original data).
</p>

<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="problems-with-probabilistic-models">Problems with probabilistic models </h2>

<p>Historically, probabilistic models suffer from a tradeoff between two
conflicting objectives: \textit{tractability} and
\textit{flexibility}. Models that are \textit{tractable} can be
analytically evaluated and easily fit to data (e.g. a Gaussian or
Laplace). However, these models are unable to aptly describe structure
in rich datasets. On the other hand, models that are \textit{flexible}
can be molded to fit structure in arbitrary data. For example, we can
define models in terms of any (non-negative) function \( \phi(\boldsymbol{x}) \)
yielding the flexible distribution \( p\left(\boldsymbol{x}\right) =
\frac{\phi\left(\boldsymbol{x} \right)}{Z} \), where \( Z \) is a normalization
constant. However, computing this normalization constant is generally
intractable. Evaluating, training, or drawing samples from such
flexible models typically requires a very expensive Monte Carlo
process.
</p>

<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="diffusion-models">Diffusion models </h2>
<p>Diffusion models have several interesting features</p>
<ul>
<li> extreme flexibility in model structure,</li>
<li> exact sampling,</li>
<li> easy multiplication with other distributions, e.g. in order to compute a posterior, and</li>
<li> the model log likelihood, and the probability of individual states, to be cheaply evaluated.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="original-idea">Original idea </h2>

<p>Text to come here.</p>
<p>In the original formulation, one uses a Markov chain to gradually
convert one distribution into another, an idea used in non-equilibrium
statistical physics and sequential Monte Carlo. Diffusion models build
a generative Markov chain which converts a simple known distribution
(e.g. a Gaussian) into a target (data) distribution using a diffusion
process. Rather than use this Markov chain to approximately evaluate a
model which has been otherwise defined, one can explicitly define the
probabilistic model as the endpoint of the Markov chain. Since each
step in the diffusion chain has an analytically evaluable probability,
the full chain can also be analytically evaluated.
</p>

<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="summary-of-course-and-topics-covered">Summary of course and topics covered </h2>
<h2 id="diffusion-learning">Diffusion learning </h2>

<p>Learning in this framework involves estimating small perturbations to
a diffusion process. Estimating small, analytically tractable,
perturbations is more tractable than explicitly describing the full
distribution with a single, non-analytically-normalizable, potential
function. Furthermore, since a diffusion process exists for any
smooth target distribution, this method can capture data distributions
of arbitrary form.
</p>

<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2024, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
Expand Down
Loading

0 comments on commit bb3d370

Please sign in to comment.