Open-Source vs API Cost Comparison

Open-source vs API cost comparison

Renting an LLM API is pay-as-you-go: cheap at low volume, expensive at scale. Self-hosting an open-source model like Llama 3 flips that — a fixed monthly GPU bill that gets cheaper per request the more you use it. This tool finds the break-even volume where self-hosting starts to win.

How it works

The API side is simple: total tokens (volume × tokens per request, split into input and output) priced at the model’s per-million rates gives a monthly bill that scales linearly with usage.

The self-hosted side is a fixed cost: a dedicated GPU instance billed by the hour, running continuously (~730 hours/month), regardless of how busy it is. Setting the two equal and solving for volume gives the break-even — the monthly request count above which the fixed GPU cost is spread thinly enough to beat per-request API pricing.

API monthly cost      = (requests × avg_tokens) / 1,000,000 × price_per_million
Self-hosted monthly   = gpu_hourly_rate × 730
Break-even requests   = Self-hosted monthly / (avg_tokens × price_per_million / 1,000,000)

Illustrative break-even scenarios

These are illustrative examples using representative numbers — verify current pricing with the specific provider before making infrastructure decisions.

At higher API prices per token, the break-even volume is lower — meaning self-hosting wins at less traffic. At very low API prices (cheap small models), the fixed GPU cost has to be spread over an enormous request count to pay off.

The hidden costs of self-hosting

The GPU hourly rate is only the visible line item. Real self-hosting involves:

• Setup and integration time — configuring inference servers (vLLM, TGI, Ollama), load balancers, and authentication takes real engineering hours
• Monitoring and reliability — you own the uptime; a GPU crash at 3am is your on-call’s problem, not the API vendor’s
• Scaling complexity — bursty traffic means GPUs sit idle most of the time or you need to auto-scale (adding complexity and cost)
• Model updates — when a new version of the open-source model releases, you re-download, re-quantize, re-test, and re-deploy it yourself
• Security — the inference server is now inside your infrastructure and carries its own attack surface

A common rule of thumb: if your engineering team costs more than the API bill and you cannot dedicate someone to maintaining inference infrastructure, the API wins on total cost even above the theoretical break-even volume.

When self-hosting genuinely wins

Self-hosting makes financial sense when:

• Your traffic is high and consistent (the GPU runs above ~60% utilisation)
• You have a dedicated ML platform team who can maintain the stack
• You need data privacy that prevents sending inputs to a third-party API
• You are using a model small enough to fit on a single GPU and your quality bar is met (for example, a fine-tuned 7B or 13B model for a narrow task)

Tips and notes

• Self-hosting is a fixed cost, not free. A 24/7 GPU costs the same whether you serve ten requests or ten million — utilisation is everything.
• Add the operational tax. Setup, autoscaling, monitoring, and on-call carry real engineering cost and risk that this raw compute comparison omits.
• Batch and autoscale to improve economics. Bursty traffic wastes a dedicated GPU; serverless GPU or batching can lower the effective self-hosted cost below the always-on assumption used here.