ControlNet Mode Reference

ControlNet mode reference

ControlNet conditions an image model on a structural map extracted from a reference image — edges, a depth map, a pose skeleton, a segmentation mask — so the diffusion process follows that structure instead of starting from pure noise. The hard part is remembering which of the dozen-plus preprocessors does what and how hard to push it. Pick your goal above and this card surfaces the right mode, a sane weight range and the models it pairs with.

How the modes differ

Every ControlNet mode is a pair: a preprocessor that turns your reference into a control map, and a model trained to read that specific map. The map type is what matters:

• Edge maps (Canny, Lineart, Softedge, MLSD) preserve outlines and detail.
• Geometry maps (Depth, Normal) preserve 3D form and spatial layout.
• Pose maps (OpenPose) preserve human/animal articulation only.
• Semantic maps (Segmentation) preserve what is where by region.
• Loose maps (Scribble, Tile, Reference) give the model the most freedom.

The tighter the map, the more faithfully output matches the reference — and the less room the prompt has to change things. That trade-off is exactly what the control weight tunes.

Choosing between Canny, Lineart, and Softedge

These three edge-map modes are frequently confused because they all extract contours, but they serve different goals:

Canny uses a hard two-threshold edge detector. It captures every sharp boundary, including surface texture detail and fine lines that may not be artistically intended. Canny is ideal for precise structural subjects like architecture, product photography, or mechanical drawings where you want the output to match the geometry exactly. It is less forgiving — small variations in the reference appear as control artifacts.

Lineart is trained on hand-drawn illustration lines and produces clean, stroke-like maps that follow the intent of the drawing rather than every pixel edge. It is the preferred mode for character art, concept illustration, and anime linework, where you want the output to feel like it was drawn rather than photographed.

Softedge (HED or PIDI preprocessor) generates softer, more probabilistic edge maps that give the model more latitude in how it fills in the shapes. It is useful when the reference structure should guide the output loosely — keeping the overall composition without locking down every line.

OpenPose keypoints: body, hands, and face

OpenPose extracts a skeleton made of keypoints joined by limbs. The base preprocessor captures the main body joints. Two additional variants add hand and face keypoints respectively — a full-body + hands + face map is far more controllable for characters in dynamic poses but adds more constraints.

Use the face keypoints variant when expression and head angle need to transfer from the reference. Use the hands variant when precise finger and wrist positions matter (holding objects, gesturing). When you only need broad body pose and do not care about face expression, the body-only variant is lighter and gives the model more room for facial interpretation.

Control weight tuning guide

Tips

• Stack modes carefully. OpenPose + Depth is a powerful combo for posed characters in a scene, but two strong controls fight each other — drop each weight to ~0.6 when stacking.
• Match resolution. Generate at a size close to your reference’s aspect ratio so the control map is not stretched.
• Use guidance end to free the late steps. Ending control around 0.7–0.8 lets the final denoising steps add texture and realism the rigid map would otherwise suppress.
• Preprocess first and inspect the map. Run the preprocessor on your reference and look at the intermediate control map before generating. A bad map — too noisy, too sparse, or with broken keypoints — will produce a bad output regardless of weight settings.