ChungMG

Score Function

The score function of a probability density $p(x)$ is its gradient with respect to the input: ${\mathrm{\nabla }}_x\log p(x)$ . It points in the direction of steepest increase in log-density and is fundamental to score-based generative modeling, [[Diffusion Model|diffusion models]], and statistical inference.

---

1. Core Concept

1.1 Definition

For a probability density function $p(x)$ , the score function is:

$$s(x)={\mathrm{\nabla }}_x\log p(x)=\frac{{\mathrm{\nabla }}_{x}p(x)}{p(x)}$$

Key properties:

• Points in the direction of maximum probability increase
• Magnitude indicates how fast probability changes
• Invariant to scaling: ${\mathrm{\nabla }}_x\log (c\cdot p(x))={\mathrm{\nabla }}_x\log p(x)$ for $c>0$

[!NOTE] Intuitive Understanding
Imagine standing on a hill in fog. The score function tells you which direction goes uphill most steeply and how steep it is, without knowing your absolute elevation.

1.2 Why Score Function?

Advantages over density estimation:

1. No normalization constant: ${\mathrm{\nabla }}_x\log p(x)$ doesn’t require knowing $\int p(x)\mathrm{d}x$
2. Geometric interpretation: Reveals structure of probability landscape
3. Sampling: Can generate samples using [[Langevin Dynamics|Langevin dynamics]]
4. Flexible models: Learn unnormalized densities

Applications:

• [[Diffusion Model|Diffusion models]] (core component)
• Score-based generative modeling
• Statistical inference (Fisher information)
• Optimization (natural gradient)
• Anomaly detection

---

2. Mathematical Properties

2.1 Basic Properties

Zero mean:

$${\mathbb{E}}_{x\sim p(x)}[s(x)]=\int p(x){\mathrm{\nabla }}_x\log p(x)\mathrm{d}x=\int {\mathrm{\nabla }}_{x}p(x)\mathrm{d}x=0$$

Fisher Information Matrix:

$$\mathcal{I}=\mathbb{E}[s(x)s(x{)}^{\mathrm{\top }}]=\text{Cov}[s(x)]$$

Measures the amount of information that observations carry about parameters.

2.2 Score of Gaussian Distribution

For $x\sim \mathcal{N}(\mu ,\mathrm{\Sigma })$ :

$$\log p(x)=-\frac{1}{2}(x-\mu {)}^{\mathrm{\top }}{\mathrm{\Sigma }}^{-1}(x-\mu )+\text{const}$$ $$s(x)={\mathrm{\nabla }}_x\log p(x)=-{\mathrm{\Sigma }}^{-1}(x-\mu )$$

Special case (standard normal $\mathcal{N}(0,I)$ ):

$$s(x)=-x$$

2.3 Score of Mixture Distribution

For mixture $p(x)=\sum _k{w}_k{p}_k(x)$ :

$$s(x)={\mathrm{\nabla }}_x\log \sum _k{w}_k{p}_k(x)=\frac{\sum _k{w}_k{\mathrm{\nabla }}_x{p}_k(x)}{\sum _k{w}_k{p}_k(x)}$$ $$s(x)=\sum _k\underset{{\gamma }_k(x)}{\underbrace{\frac{w_k{p}_k(x)}{\sum _j{w}_j{p}_j(x)}}}{s}_k(x)$$

where ${\gamma }_k(x)$ is the responsibility of component $k$ , and $s_k(x)$ is its score.

Interpretation: The score is a weighted average of component scores.

2.4 Score and Hessian

Hessian of log-density:

$$H(x)={\mathrm{\nabla }}_x^2\log p(x)={\mathrm{\nabla }}_{x}s(x)$$

Relationship:

$$\mathbb{E}[H(x)]=-\mathbb{E}[s(x)s(x{)}^{\mathrm{\top }}]=-\mathcal{I}$$

This is a fundamental identity in statistics.

---

3. Score Estimation

3.1 The Challenge

Problem: We don’t know $p(x)$ , so we can’t compute ${\mathrm{\nabla }}_x\log p(x)$ analytically.

Solution: Learn a neural network $s_{\theta }(x)$ to approximate the score function.

3.2 Score Matching

Objective: Minimize Fisher divergence between true score $s(x)$ and model score $s_{\theta }(x)$ :

$$\mathcal{L}(\theta )=\frac{1}{2}{\mathbb{E}}_{p(x)}[\parallel s_{\theta }(x)-s(x){\parallel }^2]$$

Problem: Requires knowing $s(x)={\mathrm{\nabla }}_x\log p(x)$ , which is intractable.

3.3 Denoising Score Matching

Key insight (Hyvärinen, 2005): Add noise to data and learn to recover it.

Algorithm:

1. Perturb data: $\tilde{x}=x+ϵ$ , where $ϵ\sim \mathcal{N}(0,{\sigma }^{2}I)$
2. Learn score of perturbed distribution: $s_{\theta }(\tilde{x})\approx {\mathrm{\nabla }}_{\tilde{x}}\log p_{\sigma }(\tilde{x})$

Loss function:

$$\mathcal{L}(\theta )=\frac{1}{2}{\mathbb{E}}_{x,\tilde{x}}[\parallel s_{\theta }(\tilde{x})-{\mathrm{\nabla }}_{\tilde{x}}\log p_{\sigma }(\tilde{x}\mid x){\parallel }^2]$$

For Gaussian noise $\tilde{x}\mid x\sim \mathcal{N}(x,{\sigma }^{2}I)$ :

$${\mathrm{\nabla }}_{\tilde{x}}\log p_{\sigma }(\tilde{x}\mid x)=-\frac{\tilde{x}-x}{{\sigma }^2}$$

Final loss:

$$\mathcal{L}(\theta )=\frac{1}{2{\sigma }^2}{\mathbb{E}}_{x,ϵ}[\parallel s_{\theta }(x+\sigma ϵ)+ϵ{\parallel }^2]$$

3.4 Sliced Score Matching

Idea: Use random projections to avoid computing Hessian.

Loss:

$$\mathcal{L}(\theta )={\mathbb{E}}_{p(x),p(v)}[v^{\mathrm{\top }}{\mathrm{\nabla }}_x{s}_{\theta }(x)v+\frac{1}{2}\parallel s_{\theta }(x){\parallel }^2]$$

where $v\sim \mathcal{N}(0,I)$ is a random vector.

Advantage: Only requires Jacobian-vector products (efficient via autodiff).

---

4. Score-Based Generative Modeling

4.1 Langevin Dynamics

Core idea: Use score function to generate samples via stochastic dynamics.

Algorithm:

$$x_{t+1}=x_t+\frac{\eta }{2}{s}_{\theta }(x_t)+\sqrt{\eta }{z}_t$$

where:

• $\eta$ : Step size
• $z_t\sim \mathcal{N}(0,I)$ : Random noise

Theorem: As $\eta \to 0$ and $T\to \mathrm{\infty }$ , $x_T\sim p(x)$ .

# Langevin Dynamics Sampling
def langevin_sampling(score_model, x_init, n_steps, step_size):
    x = x_init
    
    for t in range(n_steps):
        # Compute score
        score = score_model(x)
        
        # Add noise
        noise = torch.randn_like(x)
        
        # Update
        x = x + (step_size / 2) * score + torch.sqrt(step_size) * noise
    
    return x

4.2 Noise Conditional Score Networks (NCSN)

Problem: Single noise level doesn’t work well for complex distributions.

Solution: Train score model with multiple noise levels.

Algorithm:

1. Noise schedule: ${\sigma }_1<{\sigma }_2<\cdots <{\sigma }_L$
2. Perturb data: $\tilde{x}=x+{\sigma }_iϵ$ for random $i$
3. Learn: $s_{\theta }(\tilde{x},i)\approx {\mathrm{\nabla }}_{\tilde{x}}\log p_{{\sigma }_i}(\tilde{x})$

Loss:

$$\mathcal{L}(\theta )=\sum _{i=1}^L{\lambda }_i\mathbb{E}[\parallel s_{\theta }(\tilde{x},i)-{\mathrm{\nabla }}_{\tilde{x}}\log p_{{\sigma }_i}(\tilde{x}\mid x){\parallel }^2]$$

Sampling (annealed Langevin dynamics):

def annealed_langevin_sampling(score_model, n_steps_per_scale, noise_schedule):
    x = torch.randn_like(data_shape)  # Start from noise
    
    for sigma in reversed(noise_schedule):
        for t in range(n_steps_per_scale):
            score = score_model(x, sigma)
            noise = torch.randn_like(x)
            x = x + sigma**2 * score + sigma * noise
    
    return x

4.3 Score [[Stochastic Differential Equation (SDE)|SDE]]

Unified framework (Song et al., 2021): Connect score matching with SDEs.

Forward [[Stochastic Differential Equation (SDE)|SDE]]:

$$\mathrm{d}x=f(x,t)\mathrm{d}t+g(t)\mathrm{d}{W}_t$$

Reverse-time [[Stochastic Differential Equation (SDE)|SDE]]:

$$\mathrm{d}x=[f(x,t)-g(t{)}^2{\mathrm{\nabla }}_x\log p_t(x)]\mathrm{d}t+g(t)\mathrm{d}{\overline{W}}_t$$

Key insight: The reverse [[Stochastic Differential Equation (SDE)|SDE]] requires the score function ${\mathrm{\nabla }}_x\log p_t(x)$ at all times $t$ .

Training:

$$\mathcal{L}(\theta )={\mathbb{E}}_{t,x_0,x_t}[\parallel s_{\theta }(x_t,t)-{\mathrm{\nabla }}_{x_t}\log p_{0t}(x_t\mid x_0){\parallel }^2]$$

---

5. Score Function in [[Diffusion Model|Diffusion Models]]

5.1 Connection to [[Diffusion Model|DDPM]]

[[Diffusion Model|DDPM]] objective:

$$\mathcal{L}=\mathbb{E}[\parallel ϵ-{ϵ}_{\theta }(x_t,t){\parallel }^2]$$

Relationship to score:

For $x_t=\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ$ :

$${\mathrm{\nabla }}_{x_t}\log q(x_t\mid x_0)=-\frac{ϵ}{\sqrt{1-{\overline{\alpha }}_t}}$$

Therefore:

$${ϵ}_{\theta }(x_t,t)=-\sqrt{1-{\overline{\alpha }}_t}{s}_{\theta }(x_t,t)$$

Key insight: [[Diffusion Model|DDPM]] learns noise prediction, which is equivalent to learning the score function!

5.2 Unified View

Framework	Learns	Relationship
Score Matching	$s_{\theta }(x)\approx {\mathrm{\nabla }}_x\log p(x)$	Direct score estimation
[[Diffusion Model|DDPM]]	${ϵ}_{\theta }(x_t,t)$	${ϵ}_{\theta }=-{\sigma }_t{s}_{\theta }$
Score [[Stochastic Differential Equation (SDE)|SDE]]	$s_{\theta }(x,t)\approx {\mathrm{\nabla }}_x\log p_t(x)$	Time-dependent score

5.3 [[Probability Flow ODE]]

The [[Probability Flow ODE]] explicitly uses the score function:

$$\mathrm{d}x=[f(x,t)-\frac{1}{2}g(t{)}^2{\mathrm{\nabla }}_x\log p_t(x)]\mathrm{d}t$$

Interpretation:

• $f(x,t)$ : Drift term (deterministic dynamics)
• $-\frac{1}{2}g(t{)}^2{\mathrm{\nabla }}_x\log p_t(x)$ : Score-guided correction toward high-density regions

---

6. Advanced Topics

6.1 Fisher Divergence

Definition:

$$D_F(p\parallel q)=\frac{1}{2}{\mathbb{E}}_{p(x)}[\parallel {\mathrm{\nabla }}_x\log p(x)-{\mathrm{\nabla }}_x\log q(x){\parallel }^2]$$

Properties:

• Not symmetric: $D_F(p\parallel q)\ne D_F(q\parallel p)$
• Zero iff $p=q$ (up to normalization)
• Related to KL divergence via Taylor expansion

6.2 Stein’s Identity

Theorem: For any function $h(x)$ :

$${\mathbb{E}}_{p(x)}[{\mathrm{\nabla }}_{x}h(x)+h(x){\mathrm{\nabla }}_x\log p(x)]=0$$

Proof:

$$\int {\mathrm{\nabla }}_{x}h(x)p(x)\mathrm{d}x=-\int h(x){\mathrm{\nabla }}_{x}p(x)\mathrm{d}x=-\int h(x){\mathrm{\nabla }}_x\log p(x)p(x)\mathrm{d}x$$

Applications:

• Stein variational gradient descent (SVGD)
• Goodness-of-fit tests
• Variational inference

6.3 Stein Variational Gradient Descent (SVGD)

Idea: Use score function to transport particles to target distribution.

Update rule for particle $x_i$ :

$$x_i\leftarrow x_i+ϵ\sum _{j=1}^n[k(x_j,x_i){\mathrm{\nabla }}_{x_j}\log p(x_j)+{\mathrm{\nabla }}_{x_j}k(x_j,x_i)]$$

where $k(\cdot ,\cdot )$ is a kernel function.

Advantage: Non-parametric, deterministic particle-based sampling.

6.4 Score and Information Geometry

Fisher information metric:

$$g_{ij}(\theta )=\mathbb{E}\left[\frac{\partial \log p(x;\theta )}{\partial {\theta }_i}\frac{\partial \log p(x;\theta )}{\partial {\theta }_j}\right]$$

Defines a Riemannian geometry on the space of probability distributions.

Natural gradient:

$${\tilde{\mathrm{\nabla }}}_{\theta }\mathcal{L}=\mathcal{I}(\theta {)}^{-1}{\mathrm{\nabla }}_{\theta }\mathcal{L}$$

Accounts for the geometry of the statistical manifold.

---

7. Practical Implementation

7.1 Network Architectures

Common choices:

1. U-Net (for images):

   • Multi-scale processing
   • Skip connections
   • Attention mechanisms

2. MLP (for low-dimensional data):

   • Simple and efficient
   • Good for toy examples

3. Transformer (for sequences):

   • Global receptive field
   • Self-attention

Time/noise conditioning:

class ScoreNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.time_embedding = SinusoidalEmbedding(dim=128)
        self.backbone = UNet()
    
    def forward(self, x, sigma):
        # Embed noise level
        t_emb = self.time_embedding(sigma)
        
        # Predict score
        score = self.backbone(x, t_emb)
        
        return score

7.2 Training Best Practices

1. Noise schedule design:

• Geometric: ${\sigma }_i={\sigma }_{min}({\sigma }_{max}/{\sigma }_{min}{)}^{i/L}$
• Cover wide range: ${\sigma }_{min}\approx 0.01$ , ${\sigma }_{max}\approx 1.0$

2. Loss weighting:

• ${\lambda }_i={\sigma }_i^2$ (balances contribution from different scales)
• Importance sampling for critical noise levels

3. Regularization:

• Gradient clipping
• Weight decay
• EMA (exponential moving average) of parameters

4. Evaluation:

• Sample quality (FID, IS)
• Likelihood estimation
• Score accuracy (if ground truth available)

7.3 Sampling Tips

Langevin dynamics parameters:

• Step size: $\eta \propto {\sigma }^2$
• Steps per noise level: 10-100
• Noise schedule: 10-50 levels

Common issues:

• Too large step size: Instability, divergence
• Too small step size: Slow mixing, poor samples
• Too few steps: Incomplete convergence

---

8. Applications

8.1 Image Generation

Score-based models:

• NCSN (Noise Conditional Score Network)
• NCSN++ (improved architecture)
• Score [[Stochastic Differential Equation (SDE)|SDE]] (continuous-time)

Advantages:

• High-quality samples
• Exact likelihood computation
• Continuous latent space

8.2 [[Diffusion Model|Diffusion Models]]

Role of score:

• Forward process: Add noise
• Reverse process: Learn score to denoise
• Sampling: Use score for [[Langevin Dynamics|Langevin dynamics]] or ODE integration

Key papers:

• [[Diffusion Model|DDPM]]: Implicit score learning via noise prediction
• Score [[Stochastic Differential Equation (SDE)|SDE]]: Explicit score matching framework
• [[DPM-Solver]]: Fast ODE solver using score function

8.3 Audio Synthesis

Score-based audio generation:

• Waveform-level modeling
• High-fidelity audio
• Faster than autoregressive models

8.4 Molecular Generation

Applications:

• Drug discovery
• Material design
• Protein folding

Advantages:

• Continuous representation
• Exact likelihood
• Controllable generation

8.5 Anomaly Detection

Idea: Anomalies have low score magnitude (flat probability landscape).

Method:

1. Train score model on normal data
2. Compute score norm $\parallel s_{\theta }(x)\parallel$ for test points
3. Flag points with small score norm as anomalies

8.6 Energy-Based Models

Connection: Score function of energy-based model $p(x)=\frac{1}{Z}{e}^{-E(x)}$ :

$${\mathrm{\nabla }}_x\log p(x)=-{\mathrm{\nabla }}_{x}E(x)$$

Applications:

• Image inpainting
• Super-resolution
• Compositional generation

---

9. Comparison with Related Methods

9.1 Score Matching vs Variational Inference

Aspect	Score Matching	Variational Inference
Objective	Match gradients	Maximize ELBO
Normalization	Not required	Required
Flexibility	High (unnormalized)	Constrained (bound)
Computation	Score estimation	Optimization

9.2 Score-Based vs Likelihood-Based

Aspect	Score-Based	Likelihood-Based
Target	$\mathrm{\nabla }\log p(x)$	$p(x)$
Normalization	Not needed	Required
Training	Score matching	Maximum likelihood
Sampling	[[Langevin Dynamics]]	Direct or inverse

9.3 Generative Model Comparison

Model	Uses Score?	Sampling	Likelihood	Training
GAN	No	Fast (1 step)	Intractable	Adversarial
VAE	No	Fast (1 step)	Lower bound	ELBO
Normalizing Flow	No	Parallel	Exact	Maximum likelihood
Score-Based	Yes	[[Langevin Dynamics]]	Tractable	Score matching

---

10. Core Formula Cards

[!QUOTE] Score Function Definition

$$s(x)={\mathrm{\nabla }}_x\log p(x)=\frac{{\mathrm{\nabla }}_{x}p(x)}{p(x)}$$

[!QUOTE] Score of Gaussian

$$x\sim \mathcal{N}(\mu ,\mathrm{\Sigma })⟹s(x)=-{\mathrm{\Sigma }}^{-1}(x-\mu )$$

[!QUOTE] Denoising Score Matching Loss

$$\mathcal{L}(\theta )=\frac{1}{2{\sigma }^2}\mathbb{E}[\parallel s_{\theta }(x+\sigma ϵ)+ϵ{\parallel }^2]$$

[!QUOTE] Langevin Dynamics

$$x_{t+1}=x_t+\frac{\eta }{2}{s}_{\theta }(x_t)+\sqrt{\eta }{z}_t$$

[!QUOTE] Reverse-Time [[Stochastic Differential Equation (SDE)|SDE]]

$$\mathrm{d}x=[f(x,t)-g(t{)}^2{\mathrm{\nabla }}_x\log p_t(x)]\mathrm{d}t+g(t)\mathrm{d}{\overline{W}}_t$$

[!QUOTE] Score-Noise Relationship ([[Diffusion Model|DDPM]])

$${ϵ}_{\theta }(x_t,t)=-\sqrt{1-{\overline{\alpha }}_t}{s}_{\theta }(x_t,t)$$

[!QUOTE] Fisher Information Matrix

$$\mathcal{I}=\mathbb{E}[s(x)s(x{)}^{\mathrm{\top }}]$$

[!QUOTE] Stein’s Identity

$${\mathbb{E}}_{p(x)}[{\mathrm{\nabla }}_{x}h(x)+h(x){\mathrm{\nabla }}_x\log p(x)]=0$$

---

Related Concepts

• [[Diffusion Model]]
• [[Probability Flow ODE]]
• [[Stochastic Differential Equation (SDE)]]
• [[Fokker-Planck Equation]]
• [[Kolmogorov Equations]]
• [[DDIM]]
• [[DPM-Solver]]
• [[Flow Matching]]
• [[Wiener Process|Wiener Process]]
• [[Langevin Dynamics]]
• [[Denoising Score Matching]]
• [[Score SDE]]
• [[U-Net]]
• [[Fisher Information]]
• [[Energy-Based Model]]
• [[Variational Autoencoder (VAE)]]
• [[Martingale]]

---

Dataview Query

LIST
FROM #score_function OR #score_matching OR #diffusion_model
SORT file.ctime DESC

---

References

• Paper: Generative Modeling by Estimating Gradients of the Data Distribution (Song & Ermon, 2019)
• Paper: Score-Based Generative Modeling through SDEs (Song et al., 2021)
• Paper: Denoising Diffusion Probabilistic Models (Ho et al., 2020)
• Paper: A Connection between Score Matching and Denoising Autoencoders (Vincent, 2011)
• Paper: Estimation of Non-Normalized Statistical Models by Score Matching (Hyvärinen, 2005)
• Blog: Generative Modeling by Estimating Gradients - Lilian Weng
• Course: CS236 Deep Generative Models (Stanford)