ChungMG

DDIM (Denoising Diffusion Implicit Models)

DDIM is a faster deterministic sampling method for [[Diffusion Model|diffusion models]] that generalizes [[Diffusion Model|DDPM]] by relaxing the Markovian assumption. It enables 10-50× fewer sampling steps while maintaining comparable quality, and introduces the crucial concept of deterministic inversion between noise and data.

---

1. Core Concept

1.1 Motivation

[[Diffusion Model|DDPM]] problem: Requires 1000 steps for high-quality sampling.

Root cause: The reverse process must closely follow the forward process’s path, which was defined as a [[Markov Process|Markov chain]] with small Gaussian steps.

DDIM insight: The [[Diffusion Model|DDPM]] objective only depends on marginals $q(x_t\mid x_0)$ , not on the specific joint distribution $q(x_{1:T}\mid x_0)$ . This means we can define a different forward process that shares the same marginals but allows faster reverse sampling.

1.2 Key Innovation

DDIM defines a non-Markovian forward process:

• [[Diffusion Model|DDPM]] forward: $q(x_t\mid x_{t-1})$ (Markovian)
• DDIM forward: $q(x_{t-1}\mid x_t,x_0)$ (non-Markovian, conditions on $x_0$ )

Both share the same marginal distribution:

$$q(x_t\mid x_0)=\mathcal{N}(x_t;\sqrt{{\overline{\alpha }}_t}{x}_0,(1-{\overline{\alpha }}_t)I)$$

[!NOTE] Core Insight
DDIM proves that the [[Diffusion Model|DDPM]] training objective is valid for a family of inference distributions, not just the Markovian one. By choosing a non-Markovian inference distribution, we can produce higher quality samples with fewer steps.

---

2. Mathematical Foundation

2.1 [[Diffusion Model|DDPM]] Review

[[Diffusion Model|DDPM]] forward process (Markov):

$$q(x_{1:T}\mid x_0)=\prod _{t=1}^{T}q(x_t\mid x_{t-1})$$

where $q(x_t\mid x_{t-1})=\mathcal{N}(x_t;\sqrt{1-{\beta }_t}{x}_{t-1},{\beta }_{t}I)$ .

Marginal $q(x_t\mid x_0)$ :

$$x_t=\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ,ϵ\sim \mathcal{N}(0,I)$$

where ${\overline{\alpha }}_t=\prod _{s=1}^t{\alpha }_s$ and ${\alpha }_t=1-{\beta }_t$ .

2.2 DDIM Inference Distribution

DDIM defines a non-Markovian forward process:

$$q_{\sigma }(x_{1:T}\mid x_0)=q_{\sigma }(x_T\mid x_0)\prod _{t=2}^T{q}_{\sigma }(x_{t-1}\mid x_t,x_0)$$

where:

$$q_{\sigma }(x_T\mid x_0)=\mathcal{N}(x_T;\sqrt{{\overline{\alpha }}_T}{x}_0,(1-{\overline{\alpha }}_T)I)$$

and for $t>1$ :

$$q_{\sigma }(x_{t-1}\mid x_t,x_0)=\mathcal{N}(x_{t-1};\sqrt{{\overline{\alpha }}_{t-1}}{x}_0+\sqrt{1-{\overline{\alpha }}_{t-1}-{\sigma }_t^2}\cdot \frac{x_t-\sqrt{{\overline{\alpha }}_t}{x}_0}{\sqrt{1-{\overline{\alpha }}_t}},{\sigma }_t^{2}I)$$

Parameter ${\sigma }_t$ controls stochasticity:

$${\sigma }_t=\eta \sqrt{\frac{1-{\overline{\alpha }}_{t-1}}{1-{\overline{\alpha }}_t}}\sqrt{1-\frac{{\overline{\alpha }}_t}{{\overline{\alpha }}_{t-1}}}$$

• $\eta =1$ : [[Diffusion Model|DDPM]] (fully stochastic)
• $\eta =0$ : DDIM (fully deterministic)

2.3 Marginally Consistent

Theorem: For any choice of ${\sigma }_t$ , the DDIM forward process satisfies:

$$q_{\sigma }(x_t\mid x_0)=\mathcal{N}(x_t;\sqrt{{\overline{\alpha }}_t}{x}_0,(1-{\overline{\alpha }}_t)I)$$

This means all ${\sigma }_t$ choices produce the same marginals as [[Diffusion Model|DDPM]].

Consequence: A [[Diffusion Model|DDPM]]-trained model (which only depends on these marginals) can be used with any ${\sigma }_t$ !

---

3. DDIM Sampling

3.1 Generative Process

DDIM reverse process:

Given a noisy sample $x_t$ and prediction $x_0^{(t)}$ , generate $x_{t-1}$ :

$$x_{t-1}=\sqrt{{\overline{\alpha }}_{t-1}}\underset{\text{predicted }{x}_0}{\underbrace{\left(\frac{x_t-\sqrt{1-{\overline{\alpha }}_t}{ϵ}_{\theta }(x_t,t)}{\sqrt{{\overline{\alpha }}_t}}\right)}}+\underset{\text{direction pointing to }{x}_t}{\underbrace{\sqrt{1-{\overline{\alpha }}_{t-1}-{\sigma }_t^2}\cdot {ϵ}_{\theta }(x_t,t)}}+\underset{\text{random noise}}{\underbrace{{\sigma }_t{z}_t}}$$

where $z_t\sim \mathcal{N}(0,I)$ .

Three components:

1. Predicted $x_0$ : Estimate of clean data from noisy $x_t$
2. Direction to $x_t$ : Points toward the current noisy sample
3. Random noise: Controlled by ${\sigma }_t$ (zero for deterministic case)

3.2 Deterministic DDIM ( $\eta =0$ )

When $\eta =0$ (no random noise):

$$x_{t-1}=\sqrt{{\overline{\alpha }}_{t-1}}\left(\frac{x_t-\sqrt{1-{\overline{\alpha }}_t}{ϵ}_{\theta }(x_t,t)}{\sqrt{{\overline{\alpha }}_t}}\right)+\sqrt{1-{\overline{\alpha }}_{t-1}}\cdot {ϵ}_{\theta }(x_t,t)$$

Properties:

• Deterministic mapping: $x_0$ uniquely determined by $x_T$
• Invertible: Can compute $x_T$ from $x_0$ (DDIM inversion)
• Consistent: Same noise produces same output

3.3 Accelerated Sampling

DDIM can use a subsequence of timesteps:

Full schedule: $\tau =[T,T-1,\dots ,1]$

Subsampled: $\tau =[{\tau }_S,{\tau }_{S-1},\dots ,{\tau }_1]$ where $S\ll T$

Example (T=1000, S=50):

• Original: $[1000,999,998,\dots ,1]$ (1000 steps)
• Subsampled: $[1000,980,960,\dots ,20,1]$ (50 steps)

[!TIP] Practical Choice
DDIM with 50-100 steps typically achieves quality close to full [[Diffusion Model|DDPM]] (1000 steps), giving 10-20× speedup.

3.4 Sampling Pseudocode

# Deterministic DDIM Sampling
def ddim_sample(model, x_T, timesteps, eta=0.0):
    """
    model: Noise prediction network epsilon_theta(x, t)
    x_T: Initial noise ~ N(0, I)
    timesteps: Subsequence of [T, ..., 1]
    eta: 0 for DDIM, 1 for DDPM
    """
    x_t = x_T
    
    for i in range(len(timesteps)):
        t = timesteps[i]
        
        # Predict noise
        eps_theta = model(x_t, t)
        
        # Predict x_0
        x0_pred = (x_t - sqrt(1 - alpha_bar[t]) * eps_theta) / sqrt(alpha_bar[t])
        
        # Compute next timestep
        if i < len(timesteps) - 1:
            t_next = timesteps[i + 1]
        else:
            t_next = 0
        
        # Compute sigma (stochasticity)
        sigma_t = eta * sqrt((1 - alpha_bar[t_next]) / (1 - alpha_bar[t])) * \
                  sqrt(1 - alpha_bar[t] / alpha_bar[t_next])
        
        # Direction pointing to x_t
        direction = sqrt(1 - alpha_bar[t_next] - sigma_t**2) * eps_theta
        
        # Random noise (only for DDPM, eta=1)
        z = torch.randn_like(x_t) if eta > 0 else 0
        
        # Update
        x_t = sqrt(alpha_bar[t_next]) * x0_pred + direction + sigma_t * z
    
    return x_t

---

4. DDIM Inversion

4.1 Forward (Encoding) Process

DDIM inversion reverses the sampling process: given a real image $x_0$ , find the noise $x_T$ that would generate it.

Algorithm:

$$x_{t+1}=\sqrt{{\overline{\alpha }}_{t+1}}\left(\frac{x_t-\sqrt{1-{\overline{\alpha }}_t}{ϵ}_{\theta }(x_t,t)}{\sqrt{{\overline{\alpha }}_t}}\right)+\sqrt{1-{\overline{\alpha }}_{t+1}}\cdot {ϵ}_{\theta }(x_t,t)$$

# DDIM Inversion (encoding real image to noise)
def ddim_inversion(model, x_0, timesteps):
    """
    Encode real image x_0 to noise x_T
    """
    x_t = x_0
    
    for i in range(len(timesteps)):
        t = timesteps[i]
        
        # Predict noise
        eps_theta = model(x_t, t)
        
        # Predict x_0
        x0_pred = (x_t - sqrt(1 - alpha_bar[t]) * eps_theta) / sqrt(alpha_bar[t])
        
        # Compute next timestep
        t_next = timesteps[i + 1] if i < len(timesteps) - 1 else T
        
        # DDIM forward step
        direction = sqrt(1 - alpha_bar[t_next]) * eps_theta
        x_t = sqrt(alpha_bar[t_next]) * x0_pred + direction
    
    return x_t  # This is x_T (the noise code)

4.2 Applications of Inversion

1. Real Image Editing:

• Encode image to noise: $x_0\to x_T$
• Modify or guide the reverse process
• Decode back: $x_T\to {\hat{x}}_0$ (edited)

2. Semantic Interpolation:

• Encode two images: $x_0^{(1)}\to x_T^{(1)}$ , $x_0^{(2)}\to x_T^{(2)}$
• Interpolate in noise space: $x_T=(1-\lambda )x_T^{(1)}+\lambda x_T^{(2)}$
• Decode: $x_T\to {\hat{x}}_0$ (interpolated)

3. Attribute Manipulation:

• Encode image
• Apply semantic direction in noise space
• Decode for controlled editing

---

5. Theoretical Analysis

5.1 Why Fewer Steps Work

[[Diffusion Model|DDPM]] problem: Each reverse step is a small Gaussian step that assumes close proximity between $x_t$ and $x_{t-1}$ .

DDIM solution: The generative process directly jumps to the predicted $x_0$ , then mixes back with noise.

Mathematical justification:

The [[Diffusion Model|DDPM]] training loss:

$$\mathcal{L}=\sum _{t=1}^T\mathbb{E}[\parallel ϵ-{ϵ}_{\theta }(x_t,t){\parallel }^2]$$

depends only on the marginal $q(x_t\mid x_0)$ , not the joint $q(x_{1:T}\mid x_0)$ . Therefore, any inference distribution with matching marginals is valid.

5.2 Consistency Properties

Theorem (Consistency): For the same initial noise $x_T$ and model ${ϵ}_{\theta }$ , DDIM with different numbers of steps produces samples that converge to the same output as the number of steps increases.

Practical implication:

• 10 steps: Approximate result, some artifacts
• 50 steps: Good quality, minor differences from 1000-step [[Diffusion Model|DDPM]]
• 100 steps: Nearly identical to [[Diffusion Model|DDPM]]

5.3 Connection to [[Probability Flow ODE]]

DDIM as ODE discretization:

As $\eta =0$ and $T\to \mathrm{\infty }$ , DDIM converges to the [[Probability Flow ODE]]:

$$\frac{\mathrm{d}x}{\mathrm{d}t}=f(t)x-\frac{1}{2}g(t{)}^2{\mathrm{\nabla }}_x\log p_t(x)$$

DDIM vs ODE solvers:

Method	Type	Steps	Quality
DDIM-50	Discrete ODE	50	High
DDIM-100	Discrete ODE	100	Very High
[[DPM-Solver]]-2	Higher-order ODE	20	Very High
Euler ODE	1st-order ODE	100	Medium-High

[!NOTE] Historical Significance
DDIM was the first to show that diffusion models can be sampled with far fewer steps, paving the way for subsequent ODE-based samplers like [[Probability Flow ODE]] and [[DPM-Solver]].

---

6. Comparison with [[Diffusion Model|DDPM]]

6.1 Forward Process

Aspect	[[Diffusion Model|DDPM]]	DDIM
Type	Markovian	Non-Markovian
Joint distribution	$q(x_{1:T}\mid x_0)=\prod _{t}q(x_t\mid x_{t-1})$	$q_{\sigma }(x_{1:T}\mid x_0)$ (not factorized)
Marginals	$\mathcal{N}(\sqrt{{\overline{\alpha }}_t}{x}_0,(1-{\overline{\alpha }}_t)I)$	Same
Inference	$q(x_{t-1}\mid x_t)$	$q_{\sigma }(x_{t-1}\mid x_t,x_0)$

6.2 Reverse (Sampling) Process

Aspect	[[Diffusion Model|DDPM]]	DDIM ( $\eta =1$ )	DDIM ( $\eta =0$ )
Stochasticity	Random	Random	Deterministic
Noise injection	Yes	Yes	No
Invertible	No	No	Yes
Steps	1000	10-1000	10-1000

6.3 Quality vs Speed Trade-off

Method	FID (CIFAR-10)	Steps	Time (relative)
[[Diffusion Model|DDPM]]	3.17	1000	1.0×
DDIM	4.16	100	0.1×
DDIM	6.84	50	0.05×
DDIM	13.36	20	0.02×
DDIM	23.05	10	0.01×

[!TIP] Speed-Quality Balance
DDIM-100 provides an excellent balance: nearly the same quality as [[Diffusion Model|DDPM]]-1000 but 10× faster.

---

7. Stochastic DDIM ( $\eta >0$ )

7.1 Continuous Stochasticity Control

General DDIM update:

$$x_{t-1}=\sqrt{{\overline{\alpha }}_{t-1}}{x}_0^{(t)}+\sqrt{1-{\overline{\alpha }}_{t-1}-{\sigma }_t^2}\cdot {ϵ}_{\theta }(x_t,t)+{\sigma }_t{z}_t$$

Effects of $\eta$ :

$\eta$	Stochasticity	Diversity	Determinism	Best For
0.0	None	Fixed output	Perfect	Editing, inversion
0.2	Low	Some variation	Near-deterministic	Balanced quality
0.5	Medium	Moderate	Partial	General sampling
0.8	High	High	Low	Diverse generation
1.0	Full ([[Diffusion Model|DDPM]])	Maximum	None	Maximum quality

7.2 When to Use Each Mode

Deterministic ( $\eta =0$ ):

• Image editing (need reproducibility)
• DDIM inversion
• Semantic interpolation
• Latent space exploration

Stochastic ( $\eta =1$ ):

• Maximum sample diversity
• Unconditional generation
• When quality trumps speed

Intermediate ( $0<\eta <1$ ):

• Controlled diversity
• Balanced speed-quality trade-off

---

8. Applications

8.1 Real Image Editing

DDIM Inversion + Editing Pipeline:

1. Encode: Use DDIM inversion to map real image to noise
2. Modify: Apply text-guided editing in the denoising process
3. Decode: Generate edited image

Key advantage: DDIM inversion preserves image structure better than random noise.

Example: Prompt-to-Prompt, Null-text Inversion, EDICT.

8.2 Semantic Interpolation

Process:

1. Encode image A and B via DDIM inversion
2. Interpolate noise codes: $x_T^{\lambda }=(1-\lambda )x_T^A+\lambda x_T^B$
3. Decode interpolated noise

Result: Smooth semantic transition between images.

8.3 Latent Space Manipulation

Finding semantic directions:

1. Encode many images
2. Find directions in noise space corresponding to attributes
3. Apply directional shifts for controlled editing

8.4 Accelerated Training

Progressive distillation:

1. Train teacher model with [[Diffusion Model|DDPM]]
2. Use DDIM to generate high-quality samples
3. Train student model with fewer steps
4. Repeat for further acceleration

---

9. Practical Implementation

9.1 Training

Key fact: DDIM uses the same training as [[Diffusion Model|DDPM]]!

# DDIM Training = DDPM Training
# No modification needed!

def ddpm_training_loss(model, x_0):
    # Sample timestep
    t = random.randint(1, T)
    
    # Sample noise
    eps = torch.randn_like(x_0)
    
    # Forward diffusion
    x_t = sqrt(alpha_bar[t]) * x_0 + sqrt(1 - alpha_bar[t]) * eps
    
    # Predict noise
    eps_theta = model(x_t, t)
    
    # MSE loss
    loss = F.mse_loss(eps_theta, eps)
    
    return loss

# Same model can be used for DDPM and DDIM sampling!

9.2 Timestep Selection

Strategies for subsampling:

1. Uniform: Select every $k$ -th step

   • Simple but suboptimal

2. Quadratic: More steps near $t=0$

   def quadratic_schedule(T, S):
       steps = np.linspace(0, T, S+1)**2
       steps = (steps / steps[-1] * T).astype(int)
       return sorted(set(steps))

3. Linear: Evenly spaced steps

   def linear_schedule(T, S):
       return np.linspace(0, T, S+1).astype(int)

Recommendation: Linear schedule works well for 50+ steps; quadratic better for very few steps (< 20).

9.3 Debugging Checklist

• [ ] Verify ${\alpha }_t$ and ${\overline{\alpha }}_t$ values are correct
• [ ] Check $x_0$ prediction formula matches noise schedule
• [ ] Test with $\eta =0$ (deterministic) first
• [ ] Compare output with [[Diffusion Model|DDPM]] baseline
• [ ] Verify inversion consistency: $x_0\to x_T\to x_0$
• [ ] Monitor numerical stability (no NaN/Inf)
• [ ] Test with different step counts (10, 20, 50, 100)

---

10. Core Formula Cards

[!QUOTE] DDIM Marginal Distribution

$$q_{\sigma }(x_t\mid x_0)=\mathcal{N}(x_t;\sqrt{{\overline{\alpha }}_t}{x}_0,(1-{\overline{\alpha }}_t)I)$$

[!QUOTE] DDIM Generative Process

$$x_{t-1}=\sqrt{{\overline{\alpha }}_{t-1}}\left(\frac{x_t-\sqrt{1-{\overline{\alpha }}_t}{ϵ}_{\theta }(x_t,t)}{\sqrt{{\overline{\alpha }}_t}}\right)+\sqrt{1-{\overline{\alpha }}_{t-1}-{\sigma }_t^2}\cdot {ϵ}_{\theta }(x_t,t)+{\sigma }_t{z}_t$$

[!QUOTE] Deterministic DDIM ( $\eta =0$ )

$$x_{t-1}=\sqrt{{\overline{\alpha }}_{t-1}}\left(\frac{x_t-\sqrt{1-{\overline{\alpha }}_t}{ϵ}_{\theta }(x_t,t)}{\sqrt{{\overline{\alpha }}_t}}\right)+\sqrt{1-{\overline{\alpha }}_{t-1}}\cdot {ϵ}_{\theta }(x_t,t)$$

[!QUOTE] Predicted $x_0$ (One-step Estimation)

$$x_0^{(t)}=\frac{x_t-\sqrt{1-{\overline{\alpha }}_t}{ϵ}_{\theta }(x_t,t)}{\sqrt{{\overline{\alpha }}_t}}$$

[!QUOTE] DDIM Inversion (Forward)

$$x_{t+1}=\sqrt{{\overline{\alpha }}_{t+1}}\left(\frac{x_t-\sqrt{1-{\overline{\alpha }}_t}{ϵ}_{\theta }(x_t,t)}{\sqrt{{\overline{\alpha }}_t}}\right)+\sqrt{1-{\overline{\alpha }}_{t+1}}\cdot {ϵ}_{\theta }(x_t,t)$$

[!QUOTE] Stochasticity Parameter

$${\sigma }_t=\eta \sqrt{\frac{1-{\overline{\alpha }}_{t-1}}{1-{\overline{\alpha }}_t}}\sqrt{1-\frac{{\overline{\alpha }}_t}{{\overline{\alpha }}_{t-1}}}$$

---

11. Extensions and Variants

11.1 DDIM with Classifier Guidance

Combine DDIM sampling with classifier guidance:

$${\widetilde{ϵ}}_{\theta }(x_t,t)={ϵ}_{\theta }(x_t,t)-w\sqrt{1-{\overline{\alpha }}_t}{\mathrm{\nabla }}_{x_t}\log p_{\varphi }(y\mid x_t)$$

where $w$ is the guidance scale.

11.2 DDIM with Classifier-Free Guidance

$${\widetilde{ϵ}}_{\theta }(x_t,t,c)={ϵ}_{\theta }(x_t,t,\varnothing )+w({ϵ}_{\theta }(x_t,t,c)-{ϵ}_{\theta }(x_t,t,\varnothing ))$$

11.3 Spherical DDIM

Motivation: $x_T\sim \mathcal{N}(0,I)$ has norm $\approx \sqrt{d}$ .

Fix: Normalize to unit sphere for better interpolation.

$$x_T^{\text{sphere}}=\sqrt{d}\cdot \frac{x_T}{\parallel x_T\parallel }$$

11.4 Comparison with Other Fast Samplers

Method	Type	Steps	Inversion	Training Required
DDIM	Discrete ODE	10-100	Yes	No (uses [[Diffusion Model|DDPM]] model)
[[Diffusion Model|DDPM]] (few-step)	[[Stochastic Differential Equation (SDE)|SDE]]	50-100	No	No
[[DPM-Solver]]	High-order ODE	10-20	Possible	No
[[Flow Matching]]	ODE	10-100	Yes	Separate training
Consistency Models	Direct mapping	1-8	No	Separate training

---

Related Concepts

• [[Diffusion Model]]
• [[Probability Flow ODE]]
• [[Stochastic Differential Equation (SDE)]]
• [[Score Function]]
• [[DPM-Solver]]
• [[Langevin Dynamics]]
• [[Flow Matching]]
• [[Consistency Models]]
• [[Wiener Process|Wiener Process]]
• [[Markov Process]]
• [[Neural ODE]]
• [[Prompt-to-Prompt]]

---

Dataview Query

LIST
FROM #ddim OR #diffusion_model OR #fast_sampling
SORT file.ctime DESC

---

References

• Paper: Denoising Diffusion Implicit Models (Song et al., 2021)
• Paper: Denoising Diffusion Probabilistic Models (Ho et al., 2020)
• Paper: Score-Based Generative Modeling through SDEs (Song et al., 2021)
• Blog: What are Diffusion Models? - Lilian Weng
• Blog: DDIM Explained - Papers with Code
• GitHub: https://github.com/ermongroup/ddim
• Course: CS236 Deep Generative Models (Stanford)