Self-Hosting a 70B Model Breaks Even at 2,131 Tokens per Second, Not 5.6 Billion per Month

Self-hosting a 70B open model breaks even at 5.60 billion billed tokens per month. That number is almost useless.

It is useless because it is nothing but fixed cost divided by the API unit price, so it comes out to 5.60 billion no matter what your workload looks like. Chat, long-document summarization, agentic tool loops — same number. If you are deciding by volume, you are deciding on the wrong axis.

Two things actually decide it. Does your overnight trough still carry 2,131 billed tokens per second? And when you compute required utilization using throughput that actually holds your latency target, does it stay under 100%? The second condition can become impossible to satisfy depending on your workload, and when it does, adding GPUs does not fix it.

Here is the arithmetic. Every input and source is on the table, so substitute your own numbers and rerun it.

Three inputs

Input	Value	Source (retrieved 2026-08-23)
Together AI, Llama 3.3 70B	$1.04 / 1M tokens, input and output alike	pricing page
Lambda, H100 SXM	$3.99 / GPU-hour	pricing page
Llama 3.3 70B FP8 throughput	4,181.06 output tokens/sec	TensorRT-LLM performance docs

The $3.99 GPU rate is the bottom of a published $3.99–4.29 range, before tax. Take the top and everything below gets 7.5% more expensive: break-even moves from 5.60B to 6.02B tokens, required utilization from 25.5% to 27.4%. Every number in this post leans in favor of self-hosting.

A throughput figure means nothing without its conditions. That 4,181.06 is measured on 2× H100 SXM 80GB, tensor parallel 2, FP8 quantization, 1,000 input tokens and 1,000 output tokens. Any inference speed claim missing those conditions is unusable for comparison. By that same standard, one condition is missing from my own source — the document does not state batch size.

The configuration below matches that benchmark: 2× H100 SXM, and 730 hours per month.

Per hour  = 2 × $3.99         = $7.98
Per month = $7.98 × 730 hours = $5,825.40

Break-even volume is workload-independent

Self-hosting is a fixed cost. You pay $5,825.40 a month even at zero requests. The API bills per token. The two lines cross here.

Break-even billed tokens = fixed cost ÷ API unit price
                         = $5,825.40 ÷ ($1.04 / 1M)
                         = 5.60 billion tokens / month

Throughput does not appear in that equation. Whether the GPUs emit 4,181 tokens per second or 411, break-even is 5.60 billion billed tokens. So the sentence “we burn X tokens a month, therefore self-hosting wins” does not hold up.

This holds only with the hardware fixed. Go to 4 GPUs and fixed cost doubles, so break-even climbs to 11.20 billion with it. Workload-independent means independent at constant fixed cost.

Break-even volume is also a necessary condition, not a sufficient one. Below it there is nothing to discuss, but above it you still lose money if you cannot fill the required utilization. That is why the decision guide at the end screens on volume first and looks at utilization second.

Spread 5.60 billion evenly across 24 hours and you get 2,131 billed tokens per second — for a 1:1 input-to-output workload, 1,066 output tokens per second. That is the first threshold.

What the workload changes is required utilization

Throughput does not set break-even volume. It sets how much two GPUs can absorb at all, that is, capacity. Once capacity is fixed, you can ask what fraction of it you must fill to reach break-even. That fraction is required utilization.

Running the other rows of the same performance document:

Input / output tokens	Output throughput (tok/s)	Monthly capacity (billed tokens)	Required utilization	Savings at saturation
2,048 / 128	723.40	32.32B	17.3%	5.77×
5,000 / 500	865.55	25.02B	22.4%	4.47×
1,000 / 1,000	4,181.06	21.98B	25.5%	3.92×
128 / 2,048	5,892.94	16.45B	34.0%	2.94×
2,048 / 2,048	2,785.53	14.64B	38.3%	2.61×
20,000 / 2,000	411.85	11.91B	47.0%	2.13×

Capacity counts billed tokens, not output tokens. You pay for input too, so the longer the input, the more billed tokens the same GPUs push through.

The last column is the prize: how many times cheaper than the API you are when capacity is full, which is simply 1 ÷ required utilization. Once required utilization passes 100%, that number drops below 1× and self-hosting is the expensive option.

Notice the ordering flips. By raw throughput, 2,048/128 looks like one of the slowest rows. By required utilization it is the most favorable. A RAG-shaped workload that reads a lot and answers briefly is prefill-dominated, so output tokens per second are low while a large volume of input tokens gets billed.

The opposite end is worse than it looks. A workload taking 20,000 tokens in and returning 2,000 must fill 47% of capacity to break even. The mechanism behind that collapse is the KV cache: FP8 weights take roughly 70GB, leaving about 85GB of the 160GB, and holding KV for 20,000-token sequences caps concurrent requests before compute ever becomes the limit. Long context is a memory problem, not a FLOPs problem.

4,181 tokens per second is a ceiling

The TensorRT-LLM documentation describes the benchmark behind that number itself: an offline maximum-throughput scenario with all requests queued in rapid succession. It is the ceiling you reach by saturating the GPUs with no regard for latency.

An interactive service has to hold a time-to-first-token target and an inter-token latency target. Enforcing those means fewer concurrent requests, and effective throughput lands below the ceiling. How far below depends on your targets and traffic shape, and I could not find a credible primary source for it.

Since I cannot state a multiplier, here is the sensitivity instead. On the least favorable workload, 20,000/2,000:

Effective throughput (vs ceiling)	Monthly capacity	Required utilization
100%	11.91B	47.0%
50%	5.95B	94.1%
25%	2.98B	188.2%

Once required utilization exceeds 100%, no amount of volume gets you to break-even. It is not that you lack traffic. On this hardware with this workload, self-hosting cannot be cheaper than the API.

This is not confined to long context. Apply the same sensitivity to the most favorable workload, chat at 1,000/1,000:

1,000/1,000 scenario	Monthly capacity	Required utilization
Effective throughput 100%	21.98B	25.5%
Effective throughput 50%	10.99B	51.0%
Effective throughput 100% + standby node	21.98B	51.0%
Effective throughput 25%	5.49B	102.0%
Effective throughput 50% + standby node	10.99B	102.0%

Even chat becomes unwinnable at a quarter of the ceiling. Add a standby node and losing half is enough. The strongest claim this data supports is not about long context — it is that with a realistic latency target and a standby node, a 2× H100 deployment struggles to beat serverless API pricing regardless of workload shape.

Adding GPUs looks like the fix. It is not.

GPUs	Monthly fixed cost	Monthly capacity	Required utilization
2	$5,825	2.98B	188.2%
4	$11,651	5.95B	188.2%
8	$23,302	11.91B	188.2%

Capacity and fixed cost scale together, so required utilization does not move. Required utilization is scale-invariant, which means this is not a problem you solve with volume. You change the workload, change the hardware, or use the API.

That table assumes you scale by replicating the tensor-parallel-2 configuration — four GPUs means two 2-GPU nodes. That is what makes capacity proportional to GPU count. Widening tensor parallelism to 4 or 8 instead loses ground to communication overhead, and required utilization would land above 188.2%. My data covers only tensor parallel 2, so I used the assumption that flatters self-hosting; the conclusion holds either way.

Utilization has two definitions

The utilization above is not time-based. It is actual throughput against the maximum throughput of two GPUs. Conflating the two will throw off your judgment.

Time utilization — the fraction of hours the GPUs are powered. Autoscaling reduces it.
Saturation — how full the batch runs while they are powered. Sparse arrivals leave saturation at 20–30% even with the GPUs up.

They multiply. Consider internal tooling. It is used during business hours, so eight to ten hours of traffic a day, and 40% time utilization is a generous ceiling. With sparse arrivals, saturation sits near 25%.

0.40 × 0.25 = 0.10

Effective utilization of 10%, which is 39% of the 25.5% you need. At internal-tooling scale, self-hosting does not reach break-even. Both ratios are illustrative rather than measured, so substitute your own traffic curve. The point is that you have to multiply them — judging by time utilization alone leaves you two to three times too optimistic.

Autoscaling should raise time utilization, shouldn’t it? This is where model load time bites. FP8 weights for a 70B model are roughly 70GB. Reading that from local NVMe at 3 GB/s takes 23 seconds; pulling from object storage at 1 GB/s takes 70 seconds. Engine load and warmup sit on top of that. An instance that takes minutes to come up cannot chase a traffic spike that arrives in seconds.

So in practice you keep a minimum instance always on. That minimum instance is the fixed cost. Autoscaling does not eliminate the fixed cost; it trims only what sits above it. These load times are calculated from weight size and bandwidth assumptions, not measured.

Add the operator’s salary

GPU cost is one line item. Running a model server means somebody keeps doing deployments, monitoring, incident response, model and framework upgrades, quantization revalidation, and capacity planning.

With monthly operator cost $E:

Monthly operator cost	Break-even billed tokens	Required trough throughput	Required utilization (1:1 workload)
$0	5.60B	2,131 tok/s	25.5%
$2,000	7.52B	2,863 tok/s	34.2%
$4,000	9.45B	3,595 tok/s	43.0%
$8,000	13.29B	5,058 tok/s	60.5%

break-even = ($5,825.40 + E) ÷ API unit price

$E is an assumption with no source behind it. Organizations differ, so put your own number in. What matters is the slope, not the absolute value. Adding operator cost equal to 70% of the $5,825.40 GPU bill raises the required volume by 1.7×. What dominates the self-hosting break-even is not GPU hours but people.

This is the line item that goes missing most often. Procurement will bring you a GPU quote. Nobody quotes the hours of the person who has to run it.

The decision you actually make is hybrid

Everything above framed it as 100% self-hosted against 100% API. What teams actually pick sits in between: self-host the baseline, send peak and failover to the API.

Size the self-hosted portion to your overnight trough and those GPUs run effectively saturated, with the API absorbing everything above. The utilization problem disappears structurally.

So does the standby node. A standby doubles fixed cost while adding exactly zero capacity, which doubles required utilization — 25.5% to 51.0% for chat. Every calculation above assumes a single node, so that penalty is not yet included. A customer-facing service has to pay it somewhere, and the hybrid never does, because a dead self-hosted node just fails over to the API.

The decision criterion shifts from total volume to trough throughput. If your overnight trough still carries 2,131 billed tokens per second, moving that slice in-house pays, and it pays by the savings multiple in the capacity table — 3.92× on that slice for chat. A huge total volume buys you nothing if the trough sits below the line, and a modest total volume pays if the trough sits above it.

One caveat on that 2,131: it assumes zero operator cost. At $4,000 the threshold rises to 3,595 billed tokens per second, 69% higher. The practical threshold is the one with salary in it, and the title’s 2,131 should be read as the floor this arithmetic produces.

What this calculation leaves out

Omitted	Direction for self-hosting
1–3 year committed and reserved discounts (on-demand only here)	favorable
Prefix caching	unclear
Batchable offline work	unclear
Overnight and weekend idle time	unfavorable
Autoscaling that cannot respond because of model load time	unfavorable
Standby node (minimum two nodes unless hybrid)	unfavorable
GPU procurement lead time and inventory risk	unfavorable
Revalidation cost when swapping models	unfavorable
Dedicated endpoint and volume contract pricing on the API side	unfavorable
Batch processing discounts on the API side	unfavorable
Ongoing API price cuts	unfavorable

Committed discounts lower fixed cost, so they lower break-even proportionally. With discount rate d, break-even = $5,825.40 × (1 - d) ÷ API unit price. At 20% that is 4.48B tokens (20.4% required utilization); at 40%, 3.36B (15.3%). Discount rates vary by provider and term, so I am not putting a number on d.

Commitments only pay when utilization is certain, though. Signing one while volume is still unpredictable means absorbing the utilization risk yourself.

Prefix caching and batching are marked unclear for a reason. High cache hit rates or batchable work raise effective throughput for self-hosting. But API providers also discount cache hits and batch requests, so the API bill falls too. Which side falls further depends on the provider’s discounted rates, and I could not obtain them.

That leads to the biggest limitation in this comparison. The $1.04 I used is the serverless list price. Nobody paying for 5.6 billion tokens a month pays list. Dedicated endpoints, volume contracts, and batch discounts all lower the effective API rate. Not having those rates, I calculated at list, which tilts every number in this post toward self-hosting. Put real rates in and both break-even volume and required utilization go up. Modeling GPU commitment discounts while omitting API-side discounts is asymmetric, so those rates are the first thing I need to report for the follow-up.

Hyperscaler on-demand

Instance	GPU	GPU memory	Per hour
AWS g6e.12xlarge (Seoul, on-demand)	L40S ×4	192 GB	$12.90
Lambda H100 SXM ×2	H100 ×2	160 GB	$7.98

The AWS figure comes from a third-party aggregator (Vantage, retrieved 2026-08-23), because AWS renders its per-instance pricing tables in JavaScript and they are not machine-readable. Only the instance specifications are confirmed against AWS documentation.

GPU memory differs by 20%, 192 GB against 160 GB, while the hourly rate is 1.6× — normalized per GB-hour, 1.35×. But L40S and H100 are different hardware.

Without an L40S throughput figure measured under the same conditions, I am not computing required utilization for AWS. No capacity, no required utilization, and comparing hourly rates alone means nothing in this model.

The direction is all I can state. Higher hourly fixed cost raises both break-even volume and required utilization. If you are already on hyperscaler on-demand and cannot leave that contract, your self-hosting case is worse than the numbers above.

How to decide

Start by opening the API invoice and reading the monthly billed token count. A word on why a post that called volume the wrong axis starts with volume.

For 2,131 billed tokens per second to flow at the trough, sustaining that rate for a month totals 5.60 billion. So a 5.60 billion total is the minimum under perfectly flat traffic, and real traffic has peaks, meaning your total has to be considerably larger. Total volume is the screen you cannot pass without; trough throughput and required utilization are what you look at after passing it. In that order.

Under 1 billion billed tokens per month. Close the cost-driven self-hosting review. You are at 18% of break-even, and even a hybrid leaves the trough far below the threshold.

The legitimate reasons in this range are data sovereignty, regulation, latency, and model customization. If one of those applies, decide on it directly instead of dressing it up as a cost argument. That review moves faster and does not get reversed later.

1 to 5 billion. The API is still cheaper. This is the range where you watch the growth rate, and the work to do now is measurement, not migration.

Above 5 billion. Worth calculating, with your own numbers. You need four:

The distribution of input and output token lengths per request (a histogram)
Effective throughput measured under your latency target
Traffic by hour, and specifically billed tokens per second at the overnight trough
The real cost of the operator you can actually assign

Without the first three, the quote is theater. And there is an order. Get capacity from the distribution, get required utilization from capacity, and check whether required utilization stays under 100% before anything else. If it does not, the volume conversation is moot.

Small models are a different story. An 8B model fits on one GPU with much higher throughput, so capacity rises and required utilization falls sharply. Do not carry these numbers over. The structure is the same though: fixed cost against capacity, and required utilization under 100% before anything else.