Local language models: what open-weight models can do and what they need

01

What is a local language model, and what actually runs in-house?

A local language model consists of a file of trained weights and a runtime that computes answers from them. Both run on hardware you control, your own data centre or a hosting provider of your choice. Requests, documents and answers never leave that environment. Only model and software updates come in from outside, and only when you fetch them.

A large language model is, at its core, a very large set of learned numbers, the weights. A runtime loads them into the memory of a GPU, breaks requests into tokens and computes the answer token by token. "Local" therefore describes no property of the model itself, but an architectural decision: where do the weights, runtime, requests and logs live, and who decides on changes?

Whether local or cloud fits better comes down to your data situation, load and desired control, as On-premises vs. cloud LLM sets out. Knowledge management with AI shows how the model fits into a knowledge system alongside search, access rights and upkeep.

02

Are open-weight models the same thing as open source?

No. Open weights only mean that the parameters can be downloaded. The Open Source Initiative's Open Source AI Definition additionally requires the freedoms to use, study, modify and share the system, plus the full code and detailed information about the training data. A model with open weights can meet that bar, but does not have to.

The Open Source Initiative published version 1.0 of its Open Source AI Definition on 28 October 2024 (OSI, 2024). It carries the familiar freedoms of free software over to AI systems: use for any purpose without permission, study, modify, including the output, and share. In return it requires three components: information about the training data detailed enough that a skilled person could build a substantially equivalent system, the full code for training and inference, and the model parameters. The training data itself does not have to be published.

For operation, what counts first is simply that the weights are available. For procurement and compliance, what counts is the licence text of the specific model version, because some licences attach conditions to use case, user numbers or modifications. LLM comparison and model selection covers which clauses should be clarified before a test. Open-weight models for German B2B offers an assessment of widely used open models for German text.

03

What is quantisation, and what does it cost in quality?

Quantisation stores a model's weights with fewer bits, say 8 or 4 instead of 16 bits per value. That cuts the weights' memory footprint to a half or a quarter, and answers often come back faster. The quality loss is small at a moderate level, grows with heavier compression, and hits languages and difficult tasks to different degrees.

Every weight is a decimal number, usually stored with 16 bits during training. Quantisation maps these values onto a coarser grid, 256 levels at 8 bits or 16 levels at 4 bits, and stores a scaling factor per group of weights that lets the original magnitude be reconstructed. The real work of these methods lies in placing the grid so that the values that matter most for the result lose as little as possible.

Research shows how far this carries. LLM.int8() (Dettmers et al., NeurIPS 2022) halves the memory needed for inference by keeping a handful of outlier dimensions at 16 bits, with no loss of accuracy according to the paper for models up to 175 billion parameters. GPTQ and AWQ go further, compressing weights down to a few bits after training, with AWQ specifically protecting the small number of especially important weights.

A fourth piece of work matters for anything beyond English. Marchisio et al. (EMNLP 2024) found that automatic measurements underestimate the damage from quantisation, that languages with non-Latin scripts suffer the most, and that difficult tasks such as mathematical reasoning degrade faster than easy ones. English benchmark figures for a quantised model are therefore no substitute for testing on German technical text with a human sample.

04

How much GPU memory does a local model need?

GPU memory has to accommodate three items: the weights, whose size comes from the parameter count times bytes per parameter, the key-value cache (KV cache), which grows with context length and the number of simultaneous requests, and a margin for the runtime. With long contexts and many users, the KV cache quickly exceeds the weights themselves.

This calculation uses a hypothetical model, so the numbers do not go stale with any particular model version. The assumptions are stated explicitly, so you can swap in the values from your own candidate's configuration file. A gigabyte here means one billion bytes.

• Parameters: 30 billion
• Layers: 64
• Key-value heads: 8, with 128 dimensions each (grouped-query attention, covered shortly)
• KV cache: 16 bits, so 2 bytes per value
• Weights: quantised to an effective 4.5 bits per parameter, the extra half-bit covering scaling factors
• Runtime margin: a flat 2 GB for activations and overhead

Weights      = parameters x bits per parameter / 8
             = 30,000,000,000 x 4.5 / 8            = approx. 16.9 GB

KV per token = 2 (key and value) x layers x KV heads x head dimension x bytes
             = 2 x 64 x 8 x 128 x 2                 = 262,144 bytes, approx. 0.26 MB

KV total     = KV per token x context length x concurrent requests
             = 262,144 x 8,000 x 4                  = approx. 8.4 GB

Total        = weights + KV total + runtime overhead
             = 16.9 + 8.4 + 2                       = approx. 27 GB

Two lessons follow from this. First, doubling context length or parallelism doubles the KV cache, while the weights stay the same. Second, architecture matters too: grouped-query attention (Ainslie et al., EMNLP 2023) shares keys and values across fewer heads, shrinking the cache accordingly, and reaches quality close to full multi-head attention according to the paper. The number of layers and KV heads is listed in every model's configuration file.

Local LLM in the enterprise: hardware, cost, reality maps out which hardware classes fit which model sizes.

05

How is a local model made available to applications?

A local model is served through an inference server that accepts requests, batches several together, and streams the answer back token by token. Applications talk to it over an HTTP interface, commonly in a format compatible with the chat interfaces of popular hosted services. That way the model can be swapped out without rewriting the application.

The two compute phases load the hardware differently. During prefill, the model processes the entire context in parallel, which sets the wait until the first token. During decode, every further token is generated one at a time, and each step reaches into the weights and the growing cache again. A server that batches many requests into shared passes uses the GPU far more efficiently than one that works through them one after another.

How much memory gets wasted depends on how the KV cache is managed. PagedAttention (Kwon et al., SOSP 2023) manages it in blocks modelled on virtual memory management in operating systems, achieves almost no waste according to the paper, and raises the throughput of popular models two to four times over at the same latency. The resulting server, vLLM, offers endpoints compatible with OpenAI's chat completions interface, according to its documentation.

GGUF, according to the GGML project's specification, is a binary format that stores tensors and metadata in a single file and is built for fast loading, including via memory mapping. For operation with many users, though, what counts is batching, cache management and monitoring.

06

What does running a local model long-term actually require?

Long-term operation requires four things: tracked model and software versions, so results stay traceable, measurements of latency, throughput, memory and answer quality, a vetted path for updating the model and the runtime, and security, because an unprotected inference server is an open compute service on the network. That work starts once installation ends.

1. Track versions

   The model file with its checksum, runtime version, quantisation level and configuration all belong in version control. Only that way can a changed answer be traced back to a cause.

2. Maintain a test set

   Real tasks with verified expected results run before every change of model, quantisation or runtime. Checking AI answers describes how to build one.

3. Measure operations

   Time to first token, tokens per second, memory utilisation, queue length and error rate show when the hardware is reaching its limit.

4. Roll out updates deliberately

   New model versions are checked against the test set in a test environment and only adopted afterwards. Security patches for the runtime get their own, shorter path.

5. Secure access

   The server is reachable internally only, requires authentication, and logs access without personal content. Access control in AI knowledge systems covers who is allowed to see which answers.

The German Federal Office for Information Security (BSI) gives a structured overview of risks and countermeasures across the whole lifecycle of generative models, from planning to operation, in a 2025 publication on the opportunities and risks of generative AI models for industry and public bodies.

07

When is a hosted model the better choice?

A hosted model is the better choice when the task needs a level of performance unreachable on available hardware, when load swings sharply or occurs rarely, when nobody can take on operations, or when the data is not sensitive. A split often works well: confidential tasks stay local, non-sensitive ones go through a service.

On-premises and offline LLM and AI with EU data residency go deeper on hybrid setups and their criteria. A Signal on the lesson of operational control describes how even a leading hosted model can disappear from the market at short notice.

08

Which common assumptions about local models do not hold up?

Several common assumptions do not survive scrutiny: that open weights mean a free licence, that the largest model that fits is the safe choice, that quantisation costs nothing, that local automatically means secure, and that an installed model needs no ongoing work. Each one leads either to the wrong procurement decision or to a poorly maintained system.

Models that call tools carry additional requirements, described in AI agents in knowledge work.

09

How do you get started with a local language model?

Start with a narrowly scoped task and a test set of real examples, not with the hardware. Two or three open candidates of different sizes run against that test set, initially in a simple runtime. Only once quality and the memory calculation hold up do server runtime, security and choosing the right hardware follow.

1. Define the task and data class: Which requests, which documents, which level of confidentiality.
2. Build a test set: Real tasks with verified expected results, difficult cases included, in German.
3. Compare candidates: Two or three open models of different sizes and quantisation, under the same conditions.
4. Calculate memory: Using the values from the model configuration, the context length and the expected parallelism.
5. Set up operations: Server runtime, authentication, metrics, version control and a tested rollback path.

iiterate Technologies GmbH builds AI applications and custom software from architecture through implementation to operation, on client infrastructure or in EU hosting, with a focus on local language models, RAG and integration with existing systems (Services).