Does building your own infrastructure to run open models pay off? We did the maths

Every few weeks a company comes to us asking what hardware we would recommend. The plan is almost always the same:

"The models are open. We'll download them and we'll run them ourselves."

And often it makes sense. Especially if the company is large and already has hardware and people to make it available.

Open models let you keep more control over your data, choose where to run them, and avoid depending on one particular vendor. We exist, in large part, out of that same instinct.

The problem shows up when the only thing we think about is the price of the GPU, which is scary enough on its own for most people.

Because the weights are the free part.

What costs money is everything else: keeping the GPUs powered on even when they are not working, reaching enough utilisation to amortise them, optimising serving, maintaining drivers and inference engines, responding when something breaks, and assigning people to all of it.

So let us try to answer a much simpler question:

What does it really cost me to run an open model myself, compared with paying for an API?

And then we will look at everything sitting behind that figure.

If you only want the quick answer

We will use GLM-5.2 as our example.

To serve it in FP8 we take a node of 8x NVIDIA H200 as our reference.

If you rent that hardware, our baseline comes out roughly like this:

This is probably the most important figure in the whole article.

Before you know how many tokens you are going to serve, your self-hosted infrastructure already costs you around €25,000 a month.

An API works exactly the other way round.

You do not pay to have eight GPUs waiting for traffic. You pay only for the tokens you consume.

That is why the comparison changes enormously depending on your volume.

Let us take three possible prices for GLM-5.2 to see it:

• Promotional API price: the cheapest price available on OpenRouter right now.
• Competitive API price: a reference such as DeepInfra's base price.
• Standard price: the $1.40/M input and $4.40/M output that appear as the undiscounted price on OpenRouter.

For every calculation we use a ratio of 3 input tokens for every output token.

With that mix, approximately:

Now we can make the comparison we care about, as a function of your monthly consumption:

A much clearer picture starts to appear here.

At 5 or 10 billion tokens a month, renting eight H200s for yourself alone barely makes economic sense, and your CFO is never going to approve it.

From 15 to 20 billion tokens a month, the answer starts to depend enormously on the price per token you can get.

And at tens of billions of tokens a month, with a stable load and a normal API price, self-hosting starts to look very interesting.

But there is a rather curious exception.

The current promotional price for GLM-5.2 on OpenRouter carries an enormous discount. At that price, we cannot compete with the API even running our node at practically full capacity.

That says more about the discount than about self-hosting. You cannot compete by buying fixed capacity against someone who at that moment is selling the same capacity 77% below their list price. So I would advise you to run your numbers against the standard or the competitive price, and treat the promotional one for what it is: an offer that can disappear any day.

And there is another important cost: the capacity you assign to one model stops being available to the others.

If your 8x H200 are serving GLM-5.2 at full capacity, you cannot use those same GPUs simultaneously to serve another large model without splitting resources or reducing the capacity available to GLM-5.2.

This means self-hosting does not just force you to reach high utilisation. You also have to reach it on the right model.

You can have GLM-5.2 at 90% while DeepSeek sits at 10%, and you cannot automatically move that capacity from one model to the other. If you want guaranteed capacity for both, you need to size infrastructure for both, or introduce a scheduling and model load/unload layer, with its own costs and trade-offs.

What counts is the utilisation of the GPUs assigned to each specific model, and that is always worse than the utilisation of the fleet as a whole.

Before we go on: what does "building my own infrastructure" actually mean?

It is worth separating three decisions that often get lumped together.

Shared inference platform. You neither buy nor operate GPUs. You pay to use inference capacity. It is particularly interesting when demand is irregular, when you want to try different models, or when you do not want to make inference operations part of your product. (OpenRouter)

Managed dedicated infrastructure. You have capacity dedicated to you, but you do not necessarily want to take charge of the whole inference stack. It is a middle option: more control and predictability than shared infrastructure, without taking on all the operational work. (Helmcode)

Self-hosting. You rent or buy the GPUs and operate the infrastructure yourself, from deployment through optimisation, monitoring and maintenance.

And within self-hosting there is another important decision:

Do I rent the GPUs or do I buy them?

They are very different sums.

If I rent the GPUs: around €25,000 a month

Let us take the case we are analysing.

GLM-5.2 is a 744B parameter model. For this calculation we take an FP8 deployment on 8x H200 with 141 GB as our reference.

Public rental prices vary quite a lot by provider, region, commitment and availability.

In August 2026 we found references ranging from roughly $3.95 to $4.50 per GPU/hour at specialist providers, up to $50.44/hour for a complete 8x H200 node at CoreWeave, which is around $36,800 a month.

We take $4.30 per GPU/hour, which sits in the middle of that range. At that price, our bill lands at around:

~€21,500/month for the node.

But that is only the first invoice. We have not operated anything yet.

The part we tend to forget

The "we'll run it ourselves" plan also includes a new job: operating inference.

Models change constantly.

Every time one appears that might improve your stack, you have to check what quality it delivers, what throughput it reaches, how much memory it needs, which inference engine supports it, how it behaves with your traffic, what happens with long contexts and, above all, whether it works for your real cases.

We know what that costs because we do it ourselves.

Then comes optimisation.

Deploying a model does not mean you have obtained the performance that hardware can deliver. You have to decide precision, quantisation, tensor parallelism, batching, engine configuration, context, KV cache and scheduling.

In our own deployments we have measured meaningful gains simply by optimising these variables. For example, quantising Qwen 3.6 to NVFP4 improved its throughput by 11% and made room for 32 more clients.

That is inference FinOps. Because improving throughput is money.

If you manage to serve twice the tokens on the same hardware, you have just halved the infrastructure needed to reach that throughput.

And then you have to maintain it: inference engines, drivers, kernels, CUDA versions, security, monitoring, alerts, capacity, deployments and model changes.

And then there is on-call.

Inference stops being "something you use" and becomes a system you have to operate.

If it breaks at three in the morning, that is your problem. If it breaks in August, also your problem.

To make the comparison fair we have charged in:

0.5 FTE of MLOps / infrastructure engineering.

At a fully loaded cost of €90,000/year, that is approximately:

€3,750/month.

We are not saying every self-hosting setup needs exactly half an engineer.

It is an assumption.

A team that already has MLOps, GPUs and operational capacity can charge in far less incremental cost. A company starting from zero may need considerably more.

So our complete sum is:

€21,500 of infrastructure + €3,750 of operations = €25,250/month.

The problem: you pay that €25,250 whether you have traffic or not

This is probably the biggest difference between self-hosting and an API.

Your node is available around 730 hours a month. Your traffic is not. If you do not have a constant volume of requests, the sum does not work.

Picture a company whose main traffic is concentrated during working hours.

There will be peaks. There will be troughs. There will be weekends. There will be nights. There will be moments when the GPUs are waiting for work.

But your invoice will still be:

€25,250.

We could talk here about "effective utilisation", which is the technically correct metric. It is easier to see it in tokens.

How many tokens can I get out of that €25,250?

For a reference we use a public benchmark of GLM-5.2 in FP8 on exactly 8x H200 with SGLang.

In the *balanced* configuration, at concurrency 64 with a synthetic workload of 8,192 input tokens and 1,024 output tokens, the node reached approximately:

17,993 aggregate tokens per second.

If we sustained that throughput across the 730 hours of a month, we would be talking about roughly:

47 billion tokens a month of theoretical capacity.

Now we can understand the cost better.

The infrastructure costs the same every month.

If you serve only 5B tokens, each million is costing you around €5.

If you serve 30B, it drops to around €0.84.

And if you managed to keep the node practically full all month, you would approach €0.53/M.

So when does it start to pay off?

There is no single break-even for self-hosting. It depends on which model and which API you are comparing against.

With our assumptions:

Against an API at standard price, at around 14 billion tokens a month your own hardware starts to make economic sense. Economic sense only, mind you.

Against a very competitive API, you need to get close to 25 billion and keep more than half the node's capacity genuinely busy.

Against the promotional price that exists right now for GLM-5.2, you do not reach break-even even by filling the node all month.

That is why looking only at the price of an H200, or only at the price of a million tokens, gets you nowhere.

Why can a platform do this?

How can a platform charge less to serve those tokens than the node it rents or operates itself costs?

Two reasons.

The first is utilisation. An individual company has one demand curve. A platform serving hundreds of companies has hundreds of demand curves.

When one client is in a trough, another can be at a peak.

That is statistical multiplexing: using the same capacity for loads that do not coincide in time.

An individual company cannot multiplex against its own clients. A platform can aggregate the demand curves of many clients.

The second is operations. The half FTE we charged in full to our hypothetical company gets spread across many clients on a platform.

There is no magic here. There is higher hardware utilisation and a team whose job is precisely to operate that infrastructure.

And if I buy the H200s instead of renting them?

Here comes another reasonable objection:

"Renting GPUs is expensive. I'll buy them and amortise them."

Correct.

In 2026 we find complete 8-GPU HGX H200 systems at around $300,000 to $420,000, depending on manufacturer and configuration. There are even public Exxact configurations starting at roughly $296,000.

So if you want to "build your own infrastructure" by buying the hardware, think roughly in terms of:

€300,000 to €400,000 of upfront investment for an 8x H200 node.

And you need electricity, cooling, rack or datacenter, networking, maintenance, component replacement and operations.

An 8x H200 server can draw several kilowatts. Some commercial configurations publish system draw of around 6 to 7 kW, before accounting for datacenter cooling overhead.

If we take a €320,000 node as our example and amortise it linearly over 36 months:

€320,000 / 36 = ~€8,900/month of amortisation.

That is considerably less than renting the same hardware at €21,500/month. But it does not mean your infrastructure costs €8,900/month.

It means that the hardware costs you that once amortised.

You still have to add power, cooling, rack, network, maintenance and people. And on top of that you have fronted hundreds of thousands of euros.

There is also another cost that is harder to put in a spreadsheet: technological depreciation.

An H200 you buy today will still be working three years from now.

The question is whether three years from now it will still be the GPU you want to serve inference on.

Buying hardware can be an excellent option when you have very high, predictable and sustained utilisation. But it does not make capacity free.

When self-hosting wins

After all these sums, there are scenarios where self-hosting makes a great deal of sense.

You have a genuinely sustained load. Tens of billions of tokens a month, relatively stable traffic, and a model you are not going to swap every two weeks. Batch work, continuous research or certain 24/7 services can come very close to this scenario.

You already have infrastructure and a team. If you already have a cluster, MLOps, a datacenter or contracted capacity, the calculation changes completely. The incremental cost of operating one more model can be far lower than the €3,750 we have used here.

You can use the GPUs for more things. Perhaps GLM-5.2 does not fill the node, but when there is no inference you use the GPUs for fine-tuning, batch, research or other models. Your aggregate utilisation can be much higher.

You need control for reasons that are not economic. There are companies whose compliance, security or architecture requires on-premise or dedicated infrastructure. In those cases the requirement has already decided part of the infrastructure, and the arithmetic in this article only tells you what meeting it costs.

When paying for an API wins

There is also a set of situations where the answer is fairly clear.

If you consume 2B, 5B or even 10B tokens a month and your traffic is irregular, buying or renting eight H200s exclusively for yourself probably does not make much economic sense.

If you change model constantly, it does not either.

If you do not have a team that wants to become an inference infrastructure operator, it does not either.

In summary, the decision by situation:

Do the sums with your own numbers

The numbers in this article are deliberately reproducible. Change the GPU price, the number of GPUs, the throughput, the utilisation, the monthly tokens, the engineering cost and the price of the alternative, and calculate again.

The formulas are these:

Monthly TCO = infrastructure + operations

>

Cost per million tokens = monthly TCO / tokens served x 1,000,000

There is no universal figure for what it costs to self-host an LLM. There is a figure for your model, your hardware, your traffic and your team.

If you want to run the comparison against your own consumption without building the spreadsheet, we have a calculator that does exactly this arithmetic with market prices kept up to date.

We like open models precisely because they let you choose. Our value is in offering you those models on a flat rate without you having to operate or buy infrastructure. And if your case is one of those that demands dedicated capacity, we operate that too.

And to finish, an honest piece of advice. At the speed models, quantisations and chips are coming out, buying iron makes sense in very few cases. Especially if that is not your business. Focus on building a product people want and pay for, and use an AI provider the same way you use a cloud provider.

---

Assumptions and sources

*Reference model:* GLM-5.2, 744B parameters, in an FP8 deployment on a node of 8x H200 with 141 GB. The real configuration depends on precision, quantisation, context, KV cache, batch and inference engine.

*Rental prices:* public prices as of August 2026. CoreWeave ($50.44/hour for an 8x H200 HGX node), Jarvislabs ($3.99/GPU/hour), Nebius ($4.50/GPU/hour), RunPod ($4.39/GPU/hour). The €21,500/month node corresponds to $4.30/GPU/hour across 730 hours.

*Exchange rate:* EUR/USD 0.857.

*Engineering cost:* 0.5 FTE at a fully loaded cost of €90,000/year. It is an assumption of the model and not a universal cost.

*Input/output ratio:* 3 input tokens for every output token, applied to the API prices.

*Throughput:* 17,993 aggregate tokens/second, measured with SGLang in the *balanced* configuration, concurrency 64, and a synthetic workload of 8,192 input tokens and 1,024 output tokens. That workload has a different ratio from the 3:1 we use for API prices, so the theoretical capacity of 47 billion tokens a month moves with the real mix of your traffic.

*API prices:* promotional price and list price for GLM-5.2 on OpenRouter, and DeepInfra's base price as the competitive reference.

*Purchase prices:* 8-GPU HGX H200 systems between $300,000 and $420,000 depending on manufacturer and configuration, with public Exxact configurations from around $296,000. System draw of 6 to 7 kW in commercial configurations, before datacenter cooling overhead.

Provider prices are a snapshot of the market on the date given and can change. Repeat the arithmetic with your own prices, utilisation and volume: the result may be different. If it comes out in favour of self-hosting, publish it. That is precisely the reason for making the assumptions visible.