AnimateDiff Prompt Guide

AnimateDiff prompt guide

AnimateDiff animates a Stable Diffusion checkpoint by layering a motion module over it, optionally steered by motion LoRAs and prompt travel. Pick your base model and the motion you want, and this guide recommends the right module, LoRA and a prompt-travel schedule template.

How AnimateDiff works

1. Motion module — the temporal backbone, version-locked to your model (mm_sd_v15_v2 for SD 1.5; dedicated beta modules for SDXL). Pick the one that matches your checkpoint or you get flicker.
2. Motion LoRAs — optional camera-move add-ons (pan, zoom, tilt, roll) stacked on the module, controlled by a weight around 0.6–1.0.
3. Prompt travel — frame-indexed prompts that morph the scene over time, e.g. 0: "closed bud" → 16: "open flower", interpolated automatically.

Tips for coherent animation

• Lock the seed and keep the base prompt constant; vary only travel terms.
• One motion module at a time — stacking modules fights itself.
• Keep LoRA weight ≤ 1.0 — overdriving camera LoRAs causes jitter.
• Match context length to frame count so the module doesn’t loop oddly.
• Start on SD 1.5 for the richest tooling before attempting SDXL.

Understanding context length and frame count

Context length is one of the most important settings for AnimateDiff coherence. The motion module was trained to process a fixed-length window of frames at once, typically 16 frames. If you generate more frames than the context length, the module processes them in overlapping windows, with the overlap size controlling how much neighboring windows “see” of each other.

A mismatch between your total frame count and the context/overlap settings can cause the animation to loop, stutter, or lose temporal coherence at the window boundaries. Common configurations:

• 16 frames total, context 16: the module sees all frames at once; maximum coherence
• 24 frames, context 16, overlap 4: two windows overlap by 4 frames; watch for a visible seam at frame 13
• 48 frames, context 16, overlap 8: three overlapping windows; coherence degrades noticeably

For longer animations, consider using a dedicated long-context AnimateDiff variant or generating multiple short clips and concatenating them in a video editor.

Prompt travel scheduling in practice

Prompt travel is most effective for slow, smooth transitions where the two prompts share most of their content. A schedule like {0: "a young woman standing in a field of flowers", 16: "a young woman standing in a field of snow"} transitions cleanly because only the environment changes.

Dramatic subject changes — from a person to a landscape, or from daytime to nighttime with different architecture — often produce a muddled middle section rather than a clean transition. For those, consider a hard cut between two separate clips rather than prompt travel.

Practical frame schedule format for ComfyUI’s AnimateDiff Evolved node:

"0": "opening description, 4k, cinematic",
"12": "transitional state, natural light",
"24": "final state, golden hour"

Numbers are frame indices (zero-based). Each index is a keyframe; frames between indices interpolate the text embedding.

Motion LoRA strength guide

Stack at most two motion LoRAs, and reduce each weight when stacking to avoid fighting effects. A pan-left at 0.8 combined with a zoom-in at 0.8 will often produce erratic motion; try 0.6 each instead.