Likelihood Weighting

likelihood weighting = importance sampling

Javier Burroni

2026-06-22

Today

Learning Objectives

Recognize likelihood weighting = importance sampling with the prior as proposal, estimated by the self-normalized average \(\sum w f / \sum w\).

Question 3

One run by hand worked for one program. Here is the general evaluator: a single recursive function that threads the inference state state (it carries the rng and the running log_w). Four forms are special; everything else is a primitive call.

def evaluate(expr, env, state):
    if isinstance(expr, Symbol):              # variable
        return env[expr]
    if not isinstance(expr, list):            # constant
        return expr
    op, *args = expr
    if op == "let":                           # (let [v e ...] body)
      ...

In your notebook, complete the if case so the evaluator runs. Hint: if is lazy and threads state like every other form, evaluate the test, then evaluate only the branch it selects.

Section 1: Likelihood Weighting is Importance Sampling

Importance sampling, in one slide

We want \(\mathbb{E}_{p(X \given y)}[r(X)]\). Suppose we can take samples of \(q(X)\) and evaluate its density.

Reweight:

\[\mathbb{E}_{p(X \given y)}[r(X)] = \mathbb{E}_{q(X)}\!\left[ \frac{p(X \given y)}{q(X)}\, r(X) \right], \qquad q = \text{the proposal}\]

One problem

We cannot evaluate \(p(X \given y)\). However…

\[\mathbb{E}_{q(X)}\!\left[ \frac{p(X \given y)}{q(X)}\, r(X) \right] = \mathbb{E}_{q(X)}\!\left[ \frac{p(X, y)}{p(y)\,q(X)}\, r(X) \right] = \frac{1}{p(y)}\mathbb{E}_{q(X)}\!\left[ \frac{p(X, y)}{q(X)}\, r(X) \right]\]

The same trick gives the normalizer:

\[p(y) = \int p(X, y) \, dX = \mathbb{E}_{q(X)}\!\left[\frac{p(X, y)}{q(X)}\right] = \mathbb{E}_{q(X)}\!\left[ \frac{p(X, y)}{q(X)}\mathbf{1}(X) \right]\]

Reading off any expectation

Define the unnormalized weight \(W = p(X, y) / q(X)\).

We get

\[\mathbb{E}_{p(X \given y)}[r(X)] =\frac{\mathbb{E}_{q(X)}[W r(X)]}{\mathbb{E}_{q(X)}[W]}.\]

With samples \(X^{(1)},\ldots,X^{(L)} \sim q\):

\[\mathbb{E}_{p(X \given y)}[r(X)] \approx \frac{\sum_{\ell=1}^L W^{(\ell)} r(X^{(\ell)})}{\sum_{\ell=1}^L W^{(\ell)}}.\]

This is called a self-normalized estimate: the unknown normalizer \(p(y)\) is replaced by the sum of the same unnormalized weights.

Different choices of \(r\) give different posterior queries.

Likelihood weighting

Likelihood weighting takes the prior as the proposal, \(q(X) = p(X)\). Then the prior cancels:

\[W = \frac{p(y, X)}{p(X)} = p(y \given X)\]

The name explained

With the prior as proposal, the raw weight is the likelihood of the data, \(W = p(y \given X)\).
The evaluator accumulates the corresponding log weight, \(\log W = \log p(y \given X)\), at observe sites.

Explaining Fig 4.1

Using the weighted runs

Same weighted runs answer any query \(r\), chosen after sampling: \(r(x)=x\) gives the mean, \(r(x)=\mathbb{I}(x > c)\) a tail probability.
The self-normalized estimate is consistent: as \(L \to \infty\) it converges to the true posterior expectation.
Numerically stable: the raw weights \(e^{\log W}\) may underflow to \(0\) or overflow. Shift the log weights by their max \(m = \max_\ell \log W^{(\ell)}\) before exponentiating; \(m\) is a common factor that cancels in the ratio, so the estimate is unchanged.

def run_once():
  x = random.gauss(0, 1)
  logW = norm_logpdf(2.3, loc=x, sd=1)
  return x, logW

runs = [run_once() for _ in range(100_000)]
xs   = [x  for x, _  in runs]
lws  = [lw for _, lw in runs]
m    = max(lws)                          # largest log weight
ws   = [math.exp(lw - m) for lw in lws]  # shift first: no overflow
post_mean = sum(w*x for x, w in zip(xs, ws)) / sum(ws)  # ws/sum(ws) = softmax(lws)

Question 4

Likelihood weighting barely works when the prior sits far from the data. Measure how badly, then fix it.

(let [mu (sample (normal 0 1))]   ; prior far from the data
  (observe (normal mu 1) 8.0)
  mu)

In your notebook, two steps:

From a run’s normalized weights, implement the effective sample size \(\text{ESS} = 1 / \sum_\ell \bar{w}_\ell^{\,2}\) and report it for this model.
Keep the same observation, but write a new model with a different prior on mu that makes the ESS much larger. What prior did you choose, and what ESS did you get?

def ess(w):                # w: normalized weights, summing to 1
    return 1.0 / np.sum(w**2)

A prior centered near the data makes most runs explain the observation, so the weights even out and the ESS climbs toward \(L\). In likelihood weighting the prior is the proposal, and the ESS is a verdict on how well it matches where the data pull. A prior squeezed onto the data is a different model with a different posterior, so the real goal is to propose near the posterior without changing the model.

When likelihood weighting struggles

Likelihood weighting proposes latent values from the prior:

\[ X^{(\ell)} \sim p(X). \]

Then it scores them by the observations:

\[ w^{(\ell)} = p(Y_{\text{obs}} \mid X^{(\ell)}). \]

This works well when prior samples often explain the data.

It struggles when most prior samples are incompatible with the observations: then almost all normalized weight is carried by a few executions.

Summary

Likelihood weighting

Run the program forward many times. It is importance sampling with the prior as proposal, so the weight is the likelihood \(p(Y \given X)\) (only observes contribute). Read off any query with \(\sum w f / \sum w\).