A local LLM in the company: hardware, cost, reality
A local LLM needs less hardware than you think. What matters is matching the model size to the task.
This translation was produced automatically using AI. The German version is the editorially reviewed original.
Running a language model in-house is closer to reach for the Mittelstand today than the discussion suggests. Many practical use cases run on a single professional graphics card, not a data centre. The most common expensive mistake is not too little hardware, but choosing a model too large for a task a smaller one would handle just as well.
This piece sets out which hardware carries which model size, what the cost calculation looks like over three years, and what operating it day to day actually demands.
01. What hardware a local LLM really needs
The decisive factor is graphics memory, that is, the GPU's VRAM. It determines which model can be loaded at all. Through quantisation, meaning a more economical numerical representation of the model weights, this requirement drops significantly, usually without noticeable quality loss for domain-specific tasks.
As a rough rule: a model in the 7 to 8 billion parameter class runs quantised on a card with about 16 to 24 gigabytes of VRAM. Models around 30 billion parameters usually need a card with around 48 gigabytes. For the truly large models beyond 70 billion parameters, multiple cards become necessary. Many Mittelstand cases stay within the first two classes.
02. Model size: smaller is often enough
The temptation is strong to pick the most powerful model available. In practice, the right model size is a function of the task, not of ambition. For classification, extraction from documents, or answering questions from a knowledge base, a smaller model is often enough, as soon as good retrieval delivers the right passages.
This is exactly where the connection we have shown elsewhere pays off: a well-tuned retriever makes a smaller model usable and saves on hardware. How far open models now carry can be seen in recent releases, for instance Kimi K2.7 Code and NVIDIA's Nemotron 3, which are openly available and run on your own hardware.
03. The cost calculation over three years
A single professional GPU costs a mid to high four-figure amount depending on the class, plus server, power, and operation. That is a one-time investment. Set against this is a cloud service's pay-per-request model, which grows with usage.
The fair comparison does not calculate the daily rate, but the total cost over two to three years at your expected load. At constant, high utilisation, local hardware often pays for itself within this period. At low or heavily fluctuating load, the cloud usually remains cheaper. The full trade-off is set out in On-prem vs. cloud LLM: when local AI is the right choice.
04. Operations, maintenance, and the reality check
The hardware is the visible part, operations the underestimated one. A local model needs to be updated, monitored, and secured. It needs someone who deploys models, maintains the vector database, and steps in when something fails. For smaller teams, this is the real effort, not buying the card.
The advantage: tools that work locally and share nothing externally fit into this architecture. One example from our own work is Graphify, which turns a codebase into a knowledge graph locally, without any code leaving the device. Such local building blocks lower operational effort, because they require no additional external data sharing.
05. A realistic starting point
The sensible starting point is small and concrete: a clearly defined use case, a model from the smaller class, a single GPU, and cleanly set-up retrieval. From this, the team learns operations, and the next expansion stage is decided from experience, not from the spec sheet.
If you want to start with knowledge from documents, you will find the right setup in What is RAG? Retrieval-augmented generation for the Mittelstand explained. Why operational control over the models matters in the long run is set out in The Fable 5 sovereignty lesson.
