DeskTheory is where founder-CEOs learn to run their companies on AI leverage.

What is inference?

Inference is the AI actually doing its job: you type a prompt, the model reads it and produces an answer. Training is how a model gets built, once, over months. Inference is what happens every single time you hit enter. When people say AI is fast or slow, they mean inference.

You hit enter on a question and watch the answer crawl out word by word. That crawl is inference. The model isn't learning anything from you in that moment. It already knows what it knows. It is reading your prompt and generating a response one piece at a time, as fast as the chips underneath it allow.

Training built the model. Inference is you using it. Two different events, two different price tags, and as a CEO you only ever pay for one of them directly.

What it is (in plain English)

Think of two phases in a model's life. Training is the education: a lab spends months and hundreds of millions of dollars feeding a large language model most of the public internet until it can predict the next word well enough to be useful. That happens once, before you ever touch it.

Inference is the working day. Every time you, your team, or an agent sends a prompt, the finished model runs that prompt through itself and produces output. No new learning. Just the model answering, over and over, for everyone using it at once.

The output comes out in tokens. A token is a chunk of text, roughly three-quarters of a word, and the model generates them one after another. Speed is measured in tokens per second. Most assistants you use today run between 50 and 150 tokens a second, which is why you sit and watch the answer stream in. Specialized inference chips can push the same models past 1,900 tokens a second, faster than you can read. Same model, same answer, completely different feel.

Why you should care as a CEO

Inference is the part you pay for, again and again.

First, it's the bill. You don't pay to train Claude or GPT. You pay per token of inference, input and output, every time anyone runs a prompt. As your team and your agents do more, this is the line item that grows. Training is the lab's sunk cost. Inference is your recurring one.

Second, speed decides what you'll actually use AI for. A slow model is fine for a one-shot email. But an agent that takes twenty steps to finish a job pays the latency tax on every step, and something that should take seconds starts taking minutes. Fast inference is what turns multi-step, agentic work from painful into usable.

Third, where inference runs is becoming a real competitive axis. For years it ran on Nvidia GPUs. Now wafer-scale chips from companies like Cerebras run the same frontier models many times faster, and the biggest labs are buying in: OpenAI signed a multi-year deal worth more than $10 billion for Cerebras inference, and is now putting its newest model on that hardware. When OpenAI spends that kind of money on speed, speed matters.

Where you'll see it

• On any AI pricing page. The per-token input and output prices are inference prices. That meter is the cost of running the model, not building it.
• In the phrase "inference provider." Groq, Cerebras, Together, and Fireworks sell fast inference of open models they did not train.
• In agent workflows, where slow inference compounds step by step, and a near-instant model changes what's practical to automate.
• In the Cerebras and GPT-5.6 Sol news, where 750 tokens a second is the whole story.
• Any time someone calls a model "fast" or "slow." They mean its inference speed, not how smart it is.

What you should do next

Open chat.cerebras.ai and ask it something real to feel fast inference for yourself; if you want the layer above this, read what is a frontier model.