What does Inference mean?

Inference is the application of a fully trained model to new data. The model runs an input through its fixed weights and returns a prediction, a classification, or generated text. Unlike training, the model itself does not change during inference, which is why the term describes a model’s ongoing operation.

Technically, inference consists of a single forward pass through the network, with no backward pass and no adjustment of the weights. Compute cost depends on model size and, for language models, additionally on input length. Runtimes such as ONNX Runtime or vLLM shorten response time by batching requests and caching intermediate results that have already been computed. Memory demand in production determines which hardware an organisation needs to provide.

The question of inference arises as soon as a model moves from a pilot into regular operation. What needs clarifying then is the expected number of requests per hour, the acceptable response time, and where the model runs. These three factors determine whether dedicated hardware, a rented server, or an external service is the right fit.

Planning around peak load has the advantage that response time holds up even when many people work at the same time. Anyone who knows the pattern of demand across the day can match model size and hardware to it. A smaller model with a short response time is often more usable day to day than a larger one with noticeable delay.

The ongoing cost of a model arises from inference, not from the one-time training run. Where inference runs also decides which inputs leave the organisation: with an external service, every request travels to the provider.