← Zoe Wu

Diverse Low-Temperature Diffusion
Sampling through Replica Exchange

Zoe Wu, Haldun Balim, Yilun Du
Harvard University
Paper Code

Abstract

Diffusion models achieve strong generative performance across domains. However, achieving high sample quality under hard physical and dynamical constraints, as in protein design and robotic manipulation, remains difficult. Existing inference-time methods either require a costly or unavailable verifier, or avoid a verifier at the cost of reduced diversity. To relieve this tension, we introduce Score Tempering with Exchange between Exploratory Replicas (STEER), a verifier-free inference-time method that replaces single-chain sharpening with global search using multiple diffusion chains. During sampling, neighboring chains, each at different sharpening strengths, exchange particles according to an energy-based acceptance rate, propagating exploration to outputted samples. Across a diverse range of domains, from synthetic multimodal distributions to image generation, STEER achieves a better fidelity–diversity trade-off and improves success rates on continuous-control tasks in both verifier-based and verifier-free settings. Our findings suggest that effective diffusion tempering requires going beyond local updates to population-based global search that enhance fidelity and diversity.

How it works

Animation of the STEER framework: replicas start together, fan out during score-rescaled diffusion steps, and exchange samples between neighboring chains

STEER achieves a better fidelity–diversity trade-off

Images

Lowering the temperature sharpens samples, but a single chain pushed to low temperature tends to lose semantic fidelity to the prompt. STEER exchanges samples with higher-temperature replicas, allowing low-temperature sampling that stays sharp without sacrificing semantic fidelity.

Fidelity vs. prompt alignment

FID vs. CLIP score across λ; down and right is better. Pick a replica count below (or click a curve) to highlight it.

No Swaps STEER
STEER replicas

Image Samples

No Swaps vs. STEER with 4 replicas, same prompts

No Swaps
STEER
λ = 0.95

With the same prompt, replica exchange yields more diverse samples: more varied scenes, more fine-grained detail, and fewer artifacts than sampling without swaps.

Prompt“A man riding a bicycle past a train.”

No Swaps
Swaps

Protein generation

Each point is a temperature λ on protein backbone generation; STEER shifts the fidelity–designability trendline favorably. Its advantage holds as proteins get longer and we give it more replicas.

Fidelity vs. designability

FID vs. designability across λ, with linear trendlines; down and right is better.

No Swaps STEER

Longer proteins, more replicas

44.2% vs. 35.9% designable overall (+8.3 pts)

No Swaps STEER

STEER excels at continuous control tasks, with or without verifiers.

STEER excels at long-range horizon planning when composed with verifiers

High-temperature replicas explore the space of possible trajectories more broadly, while swaps with low-temperature replicas let promising trajectories cool down into precise plans that follow the maze walls.

Success rate vs. number of particles

Mean ± std over 5 seeds. Click a method to highlight it.

With swaps, replicas sample a more diverse set of pathways through the same maze, rather than collapsing onto similar routes.

No Swaps
Swaps

STEER improves imitation learning success without any verifier or reward signal

On Robomimic manipulation tasks, STEER samples actions from the same diffusion policy using only its learned score, with no reward or success check, and matches or raises success rates over the baseline and other tempering methods.

Success rate on Square

Mean ± std; axis zoomed per task.

STEER

Baseline

BibTeX

@article{wu2026steer,
  title   = {Diverse Low-Temperature Diffusion Sampling through Replica Exchange},
  author  = {Wu, Zoe and Balim, Haldun and Du, Yilun},
  year    = {2026}
}