Self-hosted LLM cost vs API: your break-even depends on which API you would otherwise buy

Together AI will sell you gpt-oss-120b at $0.60 per million output tokens. Renting two H100s to run those exact weights yourself costs $6,117 a month. To get your own cost down to $0.60, the cards have to produce 3,880 output tokens every second, day and night, weekends included. On an ordinary business schedule you can saturate them through every working hour and still land at $1.28, twice what the provider charges for the identical model. That is the comparison people are not running right now, and it is the one most teams are actually facing.

The break-even for self-hosting depends on which API you would otherwise have bought. The two possible answers sit more than an order of magnitude apart. Against Claude Sonnet 5 at $10 per million output tokens, those same two H100s pay for themselves at 612 million output tokens a month, or 233 tokens per second, which is very reachable. Against the same open weights on someone else's endpoint, the number is 10.2 billion. Published thresholds range from thirty million to three billion because each one quietly picked a side without saying so. This piece gives you both numbers and the utilisation correction that decides where you land. It also covers the latency you pay for the batch depth that makes the money work, a calculator to run on your own figures, and what to do when the arithmetic says no but you still want the hardware.

Here is the shape this review usually arrives in. A platform lead has done the homework: monthly token estimate, GPU price, division, sixty percent saving, approved.

In month four the finance partner asks why the inference line went up instead of down. The honest answer is that two numbers on that slide were wrong in the same direction. The comparison row was a frontier model the team had already stopped using for this workload. The GPU row assumed a card that was busy. Neither error looks large on its own. Together they move the answer by a factor of ten.

The people in that room were being careful. The arithmetic is easy and the two mistakes are hard to see, which is a bad combination and worth taking apart slowly.

Two questions wearing the same headline

Search this and you will find break-even thresholds from thirty million tokens a month to more than three billion. That spread looks like a disagreement about facts. It is two different questions sharing a title.

The first question is whether running open weights yourself beats calling a frontier model. That one is generous to self-hosting. Frontier output tokens cost between $5 and $50 per million, and you are proposing to replace them with electricity and depreciation, so the maths tends to come out flattering.

The second question is whether running open weights yourself beats paying a specialist to run the same weights for you. That one is brutal, and the reason is worth sitting with. The provider is executing the identical model on hardware that stays busy. They are selling you the marginal token near their marginal cost. Their utilisation is their product. Yours is whatever your traffic happens to be.

Most people who say "we should self-host to save money" are picturing the first question. The ones who actually have the workload described are living in the second. If the task runs acceptably on an open-weight model, the honest alternative is the hosted version of those same weights, and the frontier API stopped being relevant a while ago.

So the bar you have to clear is the cheap thing that happens to be the identical model. That bar sits a long way above the other one.

The numbers, on today's published prices

Everything below came off a vendor's own page on 20 September 2026. Substitute your own. The arithmetic is what carries over; my rates will be stale by the time you read this.

The hardware. Lambda lists H100 SXM on-demand at $4.19 per GPU-hour on a two-GPU instance. Two cards across a 730-hour month is $6,117. That bill holds steady whatever your traffic does.

The throughput. The GPUStack performance lab benchmarked gpt-oss-120b on two H100 SXM under vLLM v0.10.2 against ShareGPT. The default server managed 2,901 output tokens per second. With async scheduling and tensor parallelism across the pair it reached 7,648. I use the tuned figure throughout, which is the most generous honest assumption available to self-hosting.

The alternatives. Anthropic lists Claude Opus 5 at $25 per million output tokens and Sonnet 5 at $10. Google lists Gemini 3.8 Flash at $3.75 output, an introductory rate that runs to 31 December 2026 and then becomes $7.50. Together AI lists gpt-oss-120b, the same model as the benchmark, at $0.15 input and $0.60 output.

Divide the monthly bill by the alternative's output price and you get the volume where the two are level. Divide that by the seconds in a month and you get the rate you have to hold.

The last column is the one worth reading, because you can hold it up against something real. Ninety-three tokens a second is a handful of people using a chat window. Three thousand eight hundred and eighty is half the measured maximum of the hardware, held continuously, through every night and weekend of the month. No amount of spreadsheet formatting makes those look like the same request.

A rented GPU bills all 730 hours

That sustained column assumes the cards stay busy, and real workloads go quiet constantly. Internal traffic follows the working day: eight hours, five days a week. That is 174 hours of a 730-hour month, or 23.8 percent. The invoice covers all 730 regardless.

So take the required sustained rate and divide by the duty cycle. You get the peak rate you need during the hours you are actually serving. Hold that against the 7,648 tokens per second the hardware was measured at.

Look at the last row. On a normal business schedule, two H100s running gpt-oss-120b cannot reach Together's price for that same model. Saturate the cards through every working hour and you still fall short, because the rate you would need is more than twice what the hardware can physically produce. There is no flag you can pass to vLLM that closes a gap that size. The workload is not dense enough to pay for something that bills continuously.

Turn it into cost per million output tokens and it reads more plainly. Run those cards at their ceiling for every working hour and you pay $1.28 per million against Together's $0.60. Run them at a more believable thirty percent average load while live and you pay $4.26, roughly seven times the hosted price of the identical model.

And that same $4.26 is a fifty-seven percent saving against Sonnet 5, and an eighty-three percent saving against Opus 5. Both of those are true at the same time. Which one describes you comes down to what you would otherwise have bought. So the first question in these reviews should be what the workload runs on this morning. What it could theoretically run on is a different conversation, and it belongs later.

Where the utilisation number comes from

Three figures decide this whole question, and they get wildly different levels of care.

Throughput gets measured, because measuring it feels like engineering. Somebody runs a benchmark, screenshots the result, pastes it into the deck. The alternative's price gets copied off a pricing page in about nine seconds. And average utilisation, the figure that moves the answer more than either of them, gets a round number somebody said out loud in a meeting.

Watch what it does. At a hundred percent utilisation those two H100s cost $0.30 per million output tokens and beat everything in this article, Together included. At the thirty percent I used above, $4.26. At ten percent, which is ordinary for an internal tool in its first quarter, $12.77.

At that point you have quietly built something more expensive than Claude Sonnet 5 while telling the board you were saving money. The hardware and the model stayed exactly the same. The only thing that moved was a figure somebody guessed.

What makes this genuinely hard is that utilisation is decided by your users, and you do not have any users yet when you are writing the business case. The honest move is to stop guessing and go and look. Instrument the API traffic you already send to whichever provider you use today. Bucket the output tokens by hour for a fortnight. You will have a real duty cycle and a real peak.

It costs nothing, it takes an afternoon to set up, and it is the highest-value thing you can do before signing a GPU contract.

The pattern that usually turns up in that data: a sharp weekday peak, a long dead evening, a weekend near zero. That is exactly the shape that makes dedicated hardware expensive, and exactly the shape a serving provider absorbs for free, because your trough is somebody else's peak.

The batch depth that pays for the GPU is one your users can feel

There is a second cost folded into the throughput number, and it rarely reaches the spreadsheet.

Serving engines get high aggregate throughput by batching. Many requests ride the same compute pass, which is what makes the tokens cheap. Push the batch deeper and the economics improve while each individual person waits longer.

In that same GPUStack run, the default configuration produced 2,901 output tokens per second with a mean time to first token of 24.8 seconds. The tuned configuration produced 7,648 with a mean time to first token of 4.6 seconds. After that it delivered 86.5 milliseconds per output token, which one person experiences as roughly 11.6 tokens per second on their own stream.

So the configuration that makes self-hosting arithmetically attractive is one where somebody waits about five seconds for the first word. Then they read along more slowly than they are used to from the hosted APIs they have been using all year. For an overnight document pipeline that is completely fine. For a support agent typing back at a customer it is the product, and you have just made it worse to save money you may not be saving.

You can trade back, obviously. Shallower batches buy a faster first token and cost you tokens per second per dollar, which pushes every threshold in this article in the unhelpful direction. The clean way to say it: interactivity and unit cost are the same dial. A business case that assumes the throughput number while promising the latency number has spent one dial twice.

Cheaper GPUs move the threshold less than you would hope

The obvious objection is that Lambda's on-demand rate is nowhere near the cheapest capacity on the market, and that is true. Community and marketplace capacity is cheaper, reserved contracts cheaper still, and owning the hardware changes the problem shape entirely.

So here is the same break-even against Together's $0.60, recomputed at three published prices. The last column is the share of all the hours in the month you would need the cards running at their measured ceiling.

Halving the GPU price takes you from needing the cards flat out for half the month to needing them flat out for a third of it. For a business-hours workload, a third of the month is still about one and a half times the hours that exist in your working week. The answer holds.

That is worth knowing before you spend a quarter negotiating capacity to fix a problem the contract was never going to fix.

Where the GPU price genuinely bites is on the other comparison. Against Sonnet 5, cheaper capacity turns a comfortable win into an overwhelming one. Against Opus 5 it was a wide margin to begin with.

Run it on your own numbers

All of the above is four divisions. Here they are in a form you can paste into a terminal and argue with. It asks for throughput you measured yourself, because that is the only kind that belongs in a business case.

GPU_HOURLY   = 4.19   # per GPU, your quoted rate
GPU_COUNT    = 2
HOURS        = 730    # a rented GPU bills all of them
OUT_TOK_S    = 7648   # measured output tok/s, your box
ALT_PER_M    = 0.60   # alternative's $ per M output
LIVE_HOURS   = 174    # 8h x 5d: when traffic exists
AVG_LOAD     = 0.30   # mean share of ceiling while live

monthly = GPU_HOURLY * GPU_COUNT * HOURS
ceiling = OUT_TOK_S * 3600 * HOURS / 1e6   # M tok/mo

breakeven = monthly / ALT_PER_M            # M tok/mo
sustained = breakeven * 1e6 / (HOURS * 3600)
duty      = LIVE_HOURS / HOURS
peak      = sustained / duty

served    = OUT_TOK_S * 3600 * LIVE_HOURS * AVG_LOAD / 1e6
your_cost = monthly / served

print(f"bill          ${monthly:,.0f}/month")
print(f"break-even    {breakeven:,.0f}M out-tok/month")
print(f"  sustained   {sustained:,.0f} tok/s, 24/7")
print(f"  peak needed {peak:,.0f} tok/s while live")
print(f"  that is     {peak/OUT_TOK_S:.0%} of your ceiling")
print(f"you serve     {served:,.0f}M out-tok/month")
print(f"you pay       ${your_cost:,.2f} per M output")
print(f"they charge   ${ALT_PER_M:,.2f} per M output")

Change AVG_LOAD first, before you touch anything else. Watching the last two lines cross over as you move one assumption you invented is the fastest way to understand why the published thresholds disagree so violently.

If this decision is live for you, two neighbours to this piece. The self-hosted agent stack covers the five layers you sign up for once the model server is running. Ollama for model serving covers the concurrency behaviour behind the throughput number you just typed into the calculator.

What if the hardware is already yours

Plenty of teams arrive here having already bought the GPUs, either because a research group needed them or because somebody made this call last year. The arithmetic changes, and mostly in your favour.

Owned hardware turns the monthly bill into a sunk cost plus power, cooling and rack. The marginal token gets very cheap. The question stops being whether to self-host and becomes how to fill the trough.

That is a scheduling problem with good answers. Batch the overnight work that can wait. Move evaluation runs and embedding backfills onto the same cards. Let asynchronous jobs soak up the hours your interactive traffic leaves empty. A card at eighty percent utilisation across the whole day is a completely different business case from the same card at thirty.

The trap here is the reverse of the one above. Because the hardware is paid for, teams route everything to it, including the workloads that need a stronger model, and quality degrades in places you are not measuring. Sunk hardware is a reason to use the model more. It is a poor reason to use a worse model for work that needed a better one.

What this model leaves out

The arithmetic is honest about its own scope, and you should be too when you quote it.

Output tokens only. I compare output prices against output throughput and ignore input completely. That flatters self-hosting on prefill-heavy work such as long-context retrieval, where the GPU burns real time on tokens this comparison does not charge for. If your prompts are long and your answers short, redo it with total token throughput and both sides of the price.

Quantisation is unresolved. Serving at lower precision lifts throughput and changes behaviour. I have not priced that trade because it cannot be priced without your evals. Anyone telling you the quality cost is zero has not measured it, and anyone telling you it is catastrophic usually has not either.

No people in the numbers. The $6,117 is cards. It leaves out the engineer who tunes the server, and the on-call rotation for a component that now belongs to you. Nothing in it covers the eval suite you need before every weight change, or the week you will lose discovering that your inference box and your retrieval box disagree about tokenisation. On a small team that total is regularly larger than the GPU line.

On-demand, one model, no redundancy. No spot, no reserved pricing, no owned hardware, no second instance. The hosted alternatives include availability you are not buying here, and a single-instance deployment stops being comparable to an API the moment something breaks at two in the morning.

Three of those four make self-hosting look better than it is, so the conclusion survives them.

The reasons that survive a finance review

When the savings claim collapses, most teams drop the project. That is usually the wrong reaction. The savings claim was rarely why anybody wanted this. It was the argument that seemed easiest to defend, and it turned out to be the only one a spreadsheet can falsify.

Here is what holds up once the cost argument has gone. All five are worth real money; they just have to be named and budgeted as themselves.

That last row is not hypothetical. Gemini 3.8 Flash is $3.75 per million output tokens today and $7.50 on 1 January 2027, published on Google's own pricing page with the end date printed next to it. A break-even calculated against an introductory price expires along with the promotion. The case for cheap hosted inference rests, in part, on prices that are cheap deliberately and temporarily, and that risk goes uncosted.

Model permanence deserves the same seriousness. The API version you qualified against can be retired on the provider's schedule. When it is, every eval you ran becomes evidence about a model you can no longer call.

Owning the weights turns that from something that happens to you into something you decide. It is a governance property, and it belongs in the business case under its own heading where finance can weigh it. Hiding it inside a savings number is the move that sinks most of these programmes, which is why so many AI projects die at the business case long before anybody gets to blame the technology.

What I'd do

Write down two sentences before anybody prices hardware: the model this workload genuinely needs, and the hosted price of that exact model. If the honest answer is an open-weight model, you are in the $0.60 row, the economic case is almost certainly gone, and your time is better spent finding out which of the five other reasons applies to you. And if the honest answer is a frontier model, instrument a fortnight of your existing API traffic and bucket the output tokens by hour, because your real duty cycle decides this and it is the figure most teams invent. Then run the script above on throughput you measured on your own box, and put both break-evens in the deck so the flattering one has company. The version of this analysis that gets approved and then quietly fails costs far more than the version that gets rejected in September. Keep the decision written down with its inputs and the date you checked them, the way we teach operators to keep the AI Operating System as a living document, so that when prices move you can rerun it in ten minutes instead of relitigating it for a quarter. If you want the deployment side done properly instead of assembled from blog posts, that is what Engineering Foundations covers, and if this decision is being made for a whole organisation, the team programmes exist for exactly this conversation.

FAQ

Is self-hosting an LLM cheaper than using an API?

It depends which API. Against a frontier model such as Claude Sonnet 5 at $10 per million output tokens, two H100s at $6,117 a month break even at 612 million output tokens, or 233 tokens per second sustained, and most serious workloads clear that. Against the same open-weight model on a serving provider at $0.60, the break-even is 10.2 billion output tokens a month, which very few workloads reach.

How many tokens per month do you need to break even on self-hosting?

On two H100 SXM at $4.19 per GPU-hour: roughly 245 million output tokens a month against Claude Opus 5, and 612 million against Sonnet 5. Against Gemini 3.8 Flash at its introductory rate it is 1.6 billion, and against gpt-oss-120b on Together AI it is 10.2 billion. A single number quoted without naming the alternative it was measured against tells you very little.

How much does it cost to run an LLM on your own GPU?

Two H100 SXM on demand is about $6,117 a month at published September 2026 rates, billed across all 730 hours whether or not you send traffic. At the measured ceiling of 7,648 output tokens per second that is about $0.30 per million output tokens. At a realistic business duty cycle with thirty percent average load it is $4.26, and at ten percent it is $12.77.

Is it cheaper to self-host than to use an open-weight API like Together or Groq?

Usually the opposite. Serving providers run the same weights on hardware that stays busy and sell the marginal token near marginal cost. Matching Together AI's $0.60 per million output tokens on gpt-oss-120b means running your cards at their measured maximum for more than half of every hour of the month, and that requirement barely moves even if you find much cheaper GPUs.

What are the real reasons to self-host an LLM?

Data residency inside a boundary your regulator recognises. A latency floor with no shared-tenancy queue. A genuine air gap. Model permanence, so the weights you qualified stay the weights you serve. And price certainty for the day a provider's introductory rate ends. Each is worth real money, and each should be budgeted under its own name instead of disguised as a saving.

Do you need an H100 to self-host an LLM?

Not for smaller models. A 27B model quantised to four bits fits comfortably on an L40S at around $1.09 an hour, and the same arithmetic applies with different constants. The H100 pair is used here because it is the configuration with a published, reproducible throughput benchmark for gpt-oss-120b, which is what makes the comparison checkable.

The slide said sixty percent. The honest answer ranged from a large saving to a loss of seven times, and it all turned on a row that was missing: which model the workload actually needed. Establishing that row takes an afternoon and settles the entire question, which is a very good trade and a rare one.

Self-host when you want something a GPU gives you and an API cannot. Saving money you have not measured is a thinner reason than it looks from inside the meeting.

Charafeddine MouzouniSeptember 20, 2026