What makes up an AI assistant's response time

A total time answers the question of whether an assistant is too slow. It does not answer why, and therefore not which measure helps. A new graphics card speeds up only one of the items, a faster search another, and both investments can miss the actual cause.

A latency budget, as known from planning real-time systems, is more useful. You set a target response time and distribute it across the items an answer passes through. If the sum exceeds the target, the budget shows at once which item is too large.

For the machine builder in the example, such a budget would have shortened the discussion. Instead of arguing about graphics cards, the team would have seen which item exceeds the target time and whether a setting can reduce it. This breakdown can be worked through with example values.

An assistant that answers from your own documents works through the same chain for every question. How that chain is built in detail is described in the knowledge article Retrieval-Augmented Generation. For response time, five parts of it count.

Document retrieval. The search finds text sections in the index that match the question. Its duration depends on the size of the collection, the search method and how permissions are filtered.

Reranking. A second model re-orders the candidates found before the best ones go to the language model. Its duration grows with the number of candidates it has to score.

Model answer. The language model reads the instruction, the question and the sections, then writes the answer. Reading the context takes time before the first word appears, and writing takes time for every word after that.

Check step. A check compares claims with their evidence and verifies permissions and format. It makes answers more reliable and adds an item of its own in return.

Network. Request and response travel back and forth between browser, application, search and model. If the model sits with an external provider, further hops are added; if it runs in your own network, they are shorter.

A language model does not write whole sentences at once. It generates tokens, that is, word fragments, one after another, and each new token builds on all the previous ones. Writing time is therefore roughly the number of tokens times the time per token.

Time per token is not a fixed property of a model. It depends on model size, hardware and quantisation, and on how many requests are sharing the compute at that moment. Two simultaneous requests on the same graphics card both slow down, and with an external provider the value can fluctuate with the provider's load.

The number of tokens, on the other hand, is up to you. An instruction that asks for detailed introductions, restatements of the question and closing summaries quickly doubles the answer length without making the answer any better.

Many interfaces show the answer while it is being written. For users, what matters most then is when the first word appears, less when the last one is in place. Before the first word, retrieval, reranking, network and context reading all run; only then does visible writing begin.

A check step changes this calculation. If it checks the finished answer before it is displayed, the interface can show nothing in advance, and the perceived wait equals the full sum. If it checks continuously or only selected claims, progressive display remains possible, but a claim that is already visible may have to be withdrawn afterwards.

Which variant fits is a business decision. For information on which someone releases a machine, waiting for the checked answer is the lesser evil.

The demonstration adds the five items into a sum and draws it to scale against a target response time. The presets are example values for an assistant with a two-second target, where the sum lands just below it. The values are not measurements; they only make the proportions tangible.

Set the time per token from 9 to 14 milliseconds, as if two requests were sharing one graphics card. Then shorten the answer length from 180 to 110 tokens and watch the target line.

At 14 milliseconds per token the bar overshoots the target by 880 milliseconds. With 110 tokens the sum is 1,900 milliseconds and back under the target, without any change to hardware or model.

The model answer dominates the sum in almost every setting. Dropping reranking and the check step entirely saves 190 milliseconds together in the preset, while 70 fewer tokens at 9 milliseconds per token save 630 milliseconds. Shorter answers are therefore often the first and cheapest fix.

The calculation has three gaps that matter in operation. It uses one fixed value per item, although real timings vary and users notice slow answers more than the average. It knows no queue, which forms when more questions arrive than can be processed at the same time. And it does not separate reading from writing, although long sections passed to the model mainly delay the first word.

A reliable statement therefore needs measurements on your own system. Useful are timestamps at every hand-over and an evaluation that shows the slowest requests alongside the mean, for instance via the 95th percentile.

Where the language model runs changes two items above all. With an external provider, network hops and the provider's load are added; in your own network the hops are shorter, but all users share your own compute. A graphics card that is fast enough for one specialist team can become noticeably slower when a whole department asks questions at the same time.

The operating modes themselves, meaning a cloud interface, hosting in the EU and running in-house, differ in far more than response time. A comparison is on the page AI development, and the trade-off from the perspective of data and cost is in On-premise vs. cloud LLM. What hardware is realistic for running it yourself is described in Local LLM in the enterprise, and which open models can be run locally in the knowledge article Local language models.

For the machine builder this means: before a bigger graphics card is bought, a measurement with simultaneous questions shows whether the time per token rises under load. If it rises, more compute helps. If it stays stable, the cause lies in another item, and the new card would not solve the problem.

The order follows the ratio of effect to effort. The first steps change configuration and instruction; only the last ones cost money.

1. Set timestamps per item and evaluate the slowest requests separately.
2. Limit answer length: ask for a short form in the instruction and remove restatements of the question and closing formulas.
3. Pass fewer and shorter sections if the test questions show that the correct passages still arrive.
4. Check the number of candidates for reranking, because it often grows unnoticed with the collection.
5. Place the check step so that nothing is checked twice, for example permissions before the search instead of after the answer.
6. Cache frequent questions with stable answers, provided their sources rarely change.
7. Only then decide on a smaller model, different hardware or a different operating mode.

Each of these changes can affect answer quality. That is why the same test set runs after every step, as described in Why AI features fail in practice.

Not every AI application keeps a person waiting. Extraction from documents, overnight evaluations or summaries for a report run in the background, and there throughput per hour matters more than seconds per request.

Conversely, there are applications for which a two-second budget is too generous, such as voice dialogues in which a pause feels like a fault. There the calculation shifts towards short answers, smaller models and a check that does not wait for the finished answer. The use case decides which target time is appropriate, and the budget makes visible what that target costs.