Self-Hosting a 70B Model Is Chartering a Bus - You Need a 25% Load Factor to Beat Taxis

“Wouldn’t running the model ourselves be cheaper than the API?” Answering that takes more than counting monthly tokens. What you have to count is load factor.

Stated in that language, this post’s conclusions are:

• Charter fee divided by taxi fare puts the crossover at 5.60 billion billed tokens a month. That number comes out the same for every workload, which is exactly why it cannot be your decision criterion
• For a conversational workload the bus needs a 25.5% load factor to win. Long documents push that to 47.0%
• Add a realistic latency target and a standby bus and the required load factor passes 100%. Past that, no number of passengers and no number of extra buses beats the taxi

Here is the arithmetic. Every input and source is on the table, so substitute your own numbers.

Three inputs

Input	Value	Source (retrieved 2026-08-23)
Taxi fare - Together AI, Llama 3.3 70B	$1.04 / 1M tokens, input and output alike	pricing page
Charter fee - Lambda, H100 SXM	$3.99 / GPU-hour	pricing page
Seats on the bus - Llama 3.3 70B FP8 throughput	4,181.06 output tokens/sec	TensorRT-LLM performance docs

The $3.99 GPU rate is the bottom of a published $3.99-4.29 range, before tax. Take the top and everything below gets 7.5% more expensive: the crossover moves from 5.60B to 6.02B tokens and the required load factor from 25.5% to 27.4%. Every number here leans in favor of self-hosting.

A throughput figure means nothing without its conditions. That 4,181.06 is measured on 2x H100 SXM 80GB, tensor parallel 2, FP8 quantization, 1,000 input tokens and 1,000 output tokens. Any inference speed claim missing those conditions is unusable. By that same standard, one condition is missing from my own source - the document does not state batch size.

The bus matches that benchmark: 2x H100 SXM, and 730 hours per month.

Per hour  = 2 x $3.99         = $7.98
Per month = $7.98 x 730 hours = $5,825.40   <- charter fee

The crossover is 5.6 billion tokens, and that number is useless

Divide the charter fee by the taxi fare and you get the point where the two lines meet.

crossover = charter fee / taxi fare
          = $5,825.40 / ($1.04 per 1M)
          = 5.60 billion billed tokens / month

Notice that throughput does not appear in that equation. Whether the bus has forty seats or four, the crossover is 5.60 billion. Which is why “we burn X tokens a month, therefore self-hosting wins” does not hold up.

Spread 5.60 billion evenly across 24 hours and you get 2,131 billed tokens per second - for a 1:1 input-to-output workload, 1,066 output tokens per second. That is the first threshold.

What fraction of the seats do you have to fill?

What throughput sets is not the crossover but how many seats the bus has, that is, capacity. Once capacity is fixed you can ask what fraction of it you must fill to stay under the taxi fare. That fraction is the required load factor.

Input / output tokens	Output throughput (tok/s)	Monthly capacity (billed tokens)	Required load factor	Savings at full
2,048 / 128	723.40	32.32B	17.3%	5.77x
5,000 / 500	865.55	25.02B	22.4%	4.47x
1,000 / 1,000	4,181.06	21.98B	25.5%	3.92x
128 / 2,048	5,892.94	16.45B	34.0%	2.94x
2,048 / 2,048	2,785.53	14.64B	38.3%	2.61x
20,000 / 2,000	411.85	11.91B	47.0%	2.13x

The last column is the prize: how many times cheaper than the taxi you are with the seats full, which is simply 1 / required load factor. Once that passes 100% the number drops below 1x and the bus is the expensive option.

The ordering flip is worth noticing. By raw throughput, 2,048/128 looks like one of the slowest rows; by required load factor it is the most favorable. A RAG-shaped workload that reads a lot and answers briefly is dominated by the reading stage, so output tokens per second are low - but everything it read gets billed.

4,181 tokens per second was measured on an empty road

The TensorRT-LLM documentation describes the benchmark behind that number itself: an offline maximum-throughput scenario with all requests queued in rapid succession.

An interactive service has to hold a time-to-first-token target and an inter-token latency target. Enforcing those means fewer concurrent requests, and effective throughput lands below the ceiling. How far below depends on your targets and traffic shape, and I could not find a credible primary source for it.

Since I cannot state a multiplier, here is the sensitivity instead. On the least favorable workload, 20,000/2,000:

This is not confined to long context. Apply the same sensitivity to the most favorable workload, chat at 1,000/1,000:

1,000/1,000 scenario	Monthly capacity	Required load factor
Effective throughput 100%	21.98B	25.5%
Effective throughput 50%	10.99B	51.0%
Effective throughput 100% + one standby bus	21.98B	51.0%
Effective throughput 25%	5.49B	102.0%
Effective throughput 50% + one standby bus	10.99B	102.0%

Even chat becomes unwinnable at a quarter of the ceiling. Add the standby bus and losing half is enough. The strongest claim this data supports is not about long context - it is that with a realistic latency target and a standby node, a 2x H100 deployment struggles to beat serverless API pricing regardless of workload shape.

Chartering more buses looks like the fix. It is not.

Buses (GPUs)	Monthly charter fee	Monthly capacity	Required load factor
2	$5,825	2.98B	188.2%
4	$11,651	5.95B	188.2%
8	$23,302	11.91B	188.2%

Seats and charter fee scale together, so the required load factor does not move. It is scale-invariant, which means this is not a problem you solve with volume. You change what the passengers are like, change the bus, or take the taxi.

That table assumes you scale by replicating the tensor-parallel-2 configuration - four GPUs means two 2-GPU nodes. That is what makes capacity proportional to GPU count. Widening tensor parallelism to 4 or 8 loses ground to communication overhead, and the required load factor would land above 188.2%. My data covers only tensor parallel 2, so I used the assumption that flatters self-hosting; the conclusion holds either way.

Load factor has two meanings, and they multiply

The load factor above is not time-based. It is actual throughput against the maximum throughput of two GPUs. Conflating the two will throw off your judgment.

• Time utilization - the share of hours the bus is running. Autoscaling reduces it
• Saturation - how full the seats are while it runs. With sparse arrivals the bus rolls with only 20-30% of seats taken

Multiply them for an internal tool. It is used during business hours, so eight to ten hours of traffic a day, and 40% time utilization is a generous ceiling. With sparse arrivals, saturation sits near 25%. Effective load factor of 10% - 39% of the 25.5% you need. At internal-tooling scale, self-hosting does not reach break-even.

Both ratios are illustrative rather than measured, so substitute your own traffic curve. The point is that you have to multiply them. Judging by time utilization alone leaves you two to three times too optimistic.

Autoscaling should raise time utilization, shouldn’t it? This is where model load time bites. FP8 weights for a 70B model are roughly 70GB. Reading that from local NVMe at 3 GB/s takes 23 seconds; pulling from object storage at 1 GB/s takes 70 seconds. Engine load and warmup sit on top. An instance that takes minutes to come up cannot chase a traffic spike that arrives in seconds.

So in practice you keep a minimum instance always on. That minimum instance is the charter fee, and autoscaling does not eliminate it - it only trims what sits above. These load times are calculated from weight size and bandwidth assumptions, not measured.

The driver’s wages dominate

GPU cost is one line item. A bus needs a driver. Somebody keeps doing deployments, monitoring, incident response, model and framework upgrades, quantization revalidation, and capacity planning.

Monthly operator cost	Crossover	Required trough throughput	Required load factor (1:1)
$0	5.60B	2,131 tok/s	25.5%
$2,000	7.52B	2,863 tok/s	34.2%
$4,000	9.45B	3,595 tok/s	43.0%
$8,000	13.29B	5,058 tok/s	60.5%

crossover = ($5,825.40 + E) / taxi fare

E is an assumption with no source behind it. Organizations differ, so put your own number in. What matters is the slope. Adding operator cost equal to 70% of the $5,825.40 GPU bill raises the required volume by 1.7x. What dominates the self-hosting break-even is not GPU hours but people.

This is the line item that goes missing most often. Procurement will bring you a quote for the bus. Nobody quotes the hours of the person who has to drive it.

In practice you only put the regulars on the bus

Everything above framed it as 100% bus against 100% taxi. What teams actually pick sits in between.

Size the self-hosted portion to your overnight trough and those GPUs run effectively saturated, with the API absorbing everything above. The load factor problem disappears structurally.

So does the standby bus. A standby doubles the charter fee while adding exactly zero seats, which doubles the required load factor - 25.5% to 51.0% for chat. Every calculation above assumes a single node, so that penalty is not yet included. A customer-facing service has to pay it somewhere, and the hybrid never does, because a dead self-hosted node just fails over to the API.

The decision criterion shifts from total volume to trough throughput. If your overnight trough still carries 2,131 billed tokens per second, moving that slice in-house pays, and it pays by the savings multiple in the capacity table - 3.92x on that slice for chat. A huge total volume buys you nothing if the trough sits below the line, and a modest total pays if it sits above.

One caveat: that 2,131 assumes zero operator cost. At $4,000 the threshold rises to 3,595 billed tokens per second, 69% higher. The practical threshold is the one with wages in it.

Two companies, run through the numbers

Both examples show the same thing. Total volume is only the first gate; what you look at after passing it is trough throughput and required load factor.

What this calculation leaves out

Omitted	Direction for self-hosting
1-3 year committed and reserved discounts (on-demand only here)	favorable
Prefix caching	unfavorable (closed in part two)
Batchable offline work	unclear
Overnight and weekend idle time	unfavorable
Autoscaling that cannot respond because of model load time	unfavorable
Standby bus (minimum two nodes unless hybrid)	unfavorable
GPU procurement lead time and inventory risk	unfavorable
Revalidation cost when swapping models	unfavorable
Dedicated endpoint and volume contract pricing on the API side	unfavorable
Batch processing discounts on the API side	unfavorable
Ongoing API price cuts	unfavorable

Committed discounts lower the charter fee, so they lower the crossover proportionally. With discount rate d, crossover = $5,825.40 x (1 - d) / taxi fare. At 20% that is 4.48B tokens (20.4% required load factor); at 40%, 3.36B (15.3%). Discount rates vary by provider and term, so I am not putting a number on d.

Commitments only pay when the load factor is certain. Signing one while volume is still unpredictable means absorbing the load-factor risk yourself.

High cache hit rates or batchable work raise effective throughput for self-hosting. But API providers also discount cache hits and batch requests, so the API bill falls too. Which side falls further depended on the provider’s discounted rates, and part two obtained them and put them back into this model. The answer is unfavorable to self-hosting — a 70% cache hit rate plus the 50% batch discount takes the “3.92x cheaper at full load” below down to 1.34x.

That leads to the biggest limitation here. The $1.04 I used is the serverless list price. Nobody paying for 5.6 billion tokens a month pays list. Dedicated endpoints, volume contracts, and batch discounts all lower the effective API rate. Not having those rates, I calculated at list, which tilts every number toward self-hosting. Modeling GPU commitment discounts while omitting API-side discounts is asymmetric, so those rates are the first thing I need to report for the follow-up.

Chartering the bus from a hyperscaler

Instance	GPU	GPU memory	Per hour
AWS g6e.12xlarge (Seoul, on-demand)	L40S x4	192 GB	$12.90
Lambda H100 SXM x2	H100 x2	160 GB	$7.98

The AWS figure comes from a third-party aggregator (Vantage, retrieved 2026-08-23), because AWS renders its per-instance pricing tables in JavaScript and they are not machine-readable. Only the instance specifications are confirmed against AWS documentation.

GPU memory differs by 20%, 192 GB against 160 GB, while the hourly rate is 1.6x - normalized per GB-hour, 1.35x. But L40S and H100 are different hardware.

Without an L40S throughput figure measured under the same conditions, I am not computing a required load factor for AWS. No seat count, no load factor, and comparing hourly rates alone means nothing in this model.

The direction is all I can state. A higher charter fee raises both the crossover and the required load factor. If you are already on hyperscaler on-demand and cannot leave that contract, your self-hosting case is worse than the numbers above.

How to decide

Start by opening the API invoice and reading the monthly billed token count. A word on why a post that called volume useless starts with volume.

For 2,131 billed tokens per second to flow at the trough, sustaining that rate for a month totals 5.60 billion. So 5.60 billion is the minimum under perfectly flat traffic, and real traffic has peaks, meaning your total has to be considerably larger. Total volume is the screen you cannot pass without; trough throughput and required load factor are what you look at after passing it. In that order.

Under 1 billion billed tokens per month. Close the cost-driven self-hosting review. You are at 18% of the crossover, and even a hybrid leaves the trough far below the threshold. Example A lives here.

The legitimate reasons in this range are data sovereignty, regulation, latency, and model customization. If one applies, decide on it directly instead of dressing it up as a cost argument. That review moves faster and does not get reversed later.

1 to 5 billion. The API is still cheaper. This is the range where you watch the growth rate, and the work to do now is measurement, not migration.

Above 5 billion. Worth calculating, with your own numbers. You need four:

• The distribution of input and output token lengths per request (a histogram)
• Effective throughput measured under your latency target
• Traffic by hour, and specifically billed tokens per second at the overnight trough
• The real cost of the operator you can actually assign

Without the first three, the quote is theater. And there is an order. Get seat count from the distribution, get the required load factor from seat count, and check whether it stays under 100% before anything else. If it does not, the volume conversation is moot.

Small models are a different story. An 8B model fits on one GPU with much higher throughput, so seat count rises and the required load factor falls sharply. Do not carry these numbers over. The structure is the same though: charter fee against seat count, and required load factor under 100% before anything else.