Fine-tuning vs prompting decision helper
“Should we fine-tune?” is one of the most expensive questions a team can answer wrongly. Fine-tuning sounds powerful, but it adds data labelling, training cost, and a maintenance treadmill — and it solves the wrong problem if what you actually needed was retrieval or a better prompt. This helper asks the questions that separate those cases and points you to the cheapest path that closes your gap.
How it works
You answer a short set of questions: what the task is, whether the gap is knowledge (the model lacks your facts) or behaviour (it knows enough but won’t follow your format or tone), how much labelled data you have, how stable the task is, and your latency and budget constraints. The helper scores those toward four outcomes — prompt engineering, RAG, fine-tuning, or a hybrid — using the standard decision rules: knowledge gaps point to RAG, behaviour gaps with ample stable data point to fine-tuning, everything else starts with prompting. It explains the reasoning so the recommendation is auditable.
The decision tree in plain terms
Step 1 — Is the gap knowledge or behaviour?
If the model doesn’t know your company’s products, your internal policies, or facts from after its training cutoff, that is a knowledge gap. Injecting documents via RAG is faster, cheaper, and updatable without retraining. Fine-tuning cannot reliably bake in factual knowledge — models trained on specific facts still hallucinate.
If the model knows enough but consistently produces the wrong format, the wrong tone, or ignores a constraint no matter how clearly you write it in the prompt — that is a behaviour gap. Fine-tuning can encode those patterns.
Step 2 — Have you exhausted prompt engineering?
Better system prompts, few-shot examples in the context, chain-of-thought instructions, and output schema enforcement (structured outputs, constrained decoding) can close large behaviour gaps without any training. This step is reversible in minutes; fine-tuning is not.
Step 3 — Do you have the data?
Meaningful fine-tuning typically requires several hundred to a few thousand high-quality labelled input-output pairs. If you have fewer than 100 clean examples, the signal is too weak. Generating synthetic examples with a teacher model can bridge the gap — see the dataset cost estimator for budget figures.
Step 4 — Is the task stable?
Fine-tuned weights are static. If your task definition, policies, or required outputs change frequently, you will re-train regularly. A stable extraction task with a fixed schema is a good fine-tune candidate; a support task where policy changes monthly is not.
Common real-world patterns
| Situation | Recommendation |
|---|---|
| Model ignores JSON format despite explicit prompt | Fine-tune on format examples, or use structured output APIs |
| Model lacks knowledge of your product catalogue | RAG with a product database |
| Model writes in the wrong brand tone | Fine-tune on tone examples after exhausting few-shot |
| Model answers outdated questions about live data | RAG against a fresh data source |
| All of the above at once | Hybrid: RAG for knowledge, light fine-tune for tone, strong prompt for format |
Tips and notes
Default to prompting first; it is reversible and often gets you 80% of the way for a fraction of the effort. Reach for RAG when the model needs facts that change or that it never saw in training. Only fine-tune when you have a behaviour or format gap, hundreds-plus clean labelled examples, and a stable task — and budget for re-training as your data drifts. In production the strongest setup is frequently a hybrid: RAG for fresh knowledge, a light fine-tune for consistent style, behind a sharp prompt. Plan the retrieval side with the RAG architecture planner once you land on RAG or hybrid.