Diffusion Models

Tldr

Generative Model used to generate new data by learning how data changes through time through a process of adding noise and then reversing this process.

Forward Process (diffusion)

• The model starts with a clean image and adds noise step by step. [Fixed].
• After many steps, the image becomes completely unrecognizable noise.
  • the clean image $x_0$ transitions to noisy images $x_1,x_2,...,x_T$ with each step adding more and more noise.
  • Mathematically this is a Markov Chain , where at each step Gaussian noise is added:
  • $x_{t+1}=\sqrt{{\alpha }_t}{x}_t+\sqrt{1-{\alpha }_t}{ϵ}_t$
    • ${ϵ}_t\in N(0,I)$
    • ${\alpha }_t$ controls how much noise is added at each step

Reverse Process (Denoising)

• The reverse process involves recovering the original clean image $x_0$ step by step from the noisy image $x_T$.
• This is modeled as another Markov Chain, where at each step noise is subtracted to move closer to the original image:
  • Mathematically, the reverse step can be expressed as: $x_t=\frac{1}{\sqrt{{\alpha }_t}}\left(x_{t+1}-\sqrt{1-{\alpha }_t}{ϵ}_t\right)+{\sigma }_{t}z$
    • Here:
      • $z\sim N(0,I)$ (optional noise for stochasticity)
      • ${\sigma }_t$ controls the variance during the reverse step.
• ==The reverse process is learned using a neural network that predicts the amount of noise ${ϵ}_t$ added at each step. This prediction is used to iteratively denoise.==

Training Objective

• The model is trained to minimize the difference between the predicted noise and the actual noise added at each forward step:

• Training is performed by maximizing the ELBO, which leads to maximizing log likelihood.

  ${\mathcal{L}}_{\text{simple}}={\mathbb{E}}_{x_0,ϵ\sim N(0,I),t}\left[\Vert ϵ-{ϵ}_{\theta }(x_t,t){\Vert }^2\right]$

  • ${ϵ}_{\theta }(x_t,t)$: Neural network’s prediction of noise for input $x_t$ at time $t$.

Key Components

1. Forward Process (Noise Addition):

   • Progressively adds Gaussian noise to transition $x_0\to x_T$.
   • Defined by: $q(x_t|x_{t-1})=N(x_t;\sqrt{{\alpha }_t}{x}_{t-1},(1-{\alpha }_t)I)$

2. Reverse Process (Noise Removal):

   • Approximates the denoising distribution: $\boxed{p_{\theta }(x_{t-1}|x_t)=N(x_{t-1};{\mu }_{\theta }(x_t,t),{\Sigma }_{\theta }(x_t,t))}$
   • Where:
     • ${\mu }_{\theta }(x_t,t)$: Mean predicted by the neural network.
     • ${\Sigma }_{\theta }(x_t,t)$: Variance (can be fixed or learned).

Connection to Variational Inference

• The forward process defines a fixed distribution $q(x_t|x_0)$.
• The reverse process tries to approximate $q(x_{t-1}|x_t,x_0)$ by optimizing the Evidence Lower Bound (ELBO): $\mathcal{L}={\mathbb{E}}_q\left[{\sum }_{t=1}^T{D}_{\text{KL}}\left(q(x_{t-1}|x_t,x_0)\Vert p_{\theta }(x_{t-1}|x_t)\right)\right]$

Resources

Diffusion Without Tears Step-by-Step Diffusion: An Elementary Tutorial (arxiv.org)