If you have ever looked at the price list for an AI service, you may have seen something like “$1.40 per million input tokens, $4.40 output.” A token is a small chunk of text, roughly a word or a piece of one. Input is what you send in. Output is what the model writes back. And the output is always the pricier side, usually two to four times more, at pretty much every company that sells this stuff.
I had assumed that was just a pricing habit, like charging more for the large popcorn. It isn’t. The gap comes from how a model actually runs, and once I understood it, a lot of other things about AI started to make sense.
Reading is fast, writing is slow
The running part has a name: inference, which in plain English means the model doing its thinking in real time, as opposed to the long training it went through beforehand. Inference happens in two phases.
The first is prefill. The model takes in your whole prompt at once, all in parallel, like speed-reading a page. That’s the input side, and it’s cheap.
The second is decode. This is where the model writes its answer, one word at a time. It can’t choose the next word until it has the one before it, so each word means another full trip through the entire model. That’s the output side, and it’s slow and expensive.
I think about this the way I think about reading versus writing a letter. I can skim a two-page letter in a minute. Writing a reply of the same length takes me most of an evening, because every sentence depends on the one I just wrote. The model has the same problem, only it pays for it in computing power instead of patience.
The memory that keeps growing
There’s a second cost hiding in there. If the model had to reread your entire conversation before producing every new word, that would be enormously wasteful. So instead it keeps a running memory of everything so far, called the KV cache (short for key-value cache). It’s the model’s working memory, and it grows word by word.
A longer conversation means a bigger cache, and a bigger cache eats more RAM, the fast memory a computer works out of. So every word of output costs a full pass through the model plus a slightly larger pile of memory to carry along. Put those together and you get the two to four times markup.
It reminds me of a long phone call where you’re taking notes so you don’t lose track. Early on the notepad is light. An hour in, you’re flipping back through pages before you can say anything sensible. The conversation didn’t get harder, there’s just more of it to hold.
Memory is the real budget
This changed how I think about running AI models on your own computer. There, memory is usually a bigger limit than how fast the processor is. A big model needs a lot of RAM just to exist on the machine at all, and the KV cache takes more the longer you chat. If you’re ever shopping for a computer to run models at home, memory is the budget, and the KV cache is where a good part of it goes.
So how are some prices so low?
Some AI services charge remarkably little per token, and my first instinct was that the models must be worse. The lesson pointed somewhere else. Two things drive those prices down.
One is batching: grouping thousands of people’s requests together so the expensive graphics chips (GPUs) that run the model never sit idle. It’s the bus versus the taxi. The trip costs about the same to run either way, but split across a full bus, each seat is cheap.
The other is efficient design. DeepSeek, for example, uses a setup called mixture-of-experts that only wakes up a fraction of the model for each word, so each word costs less to produce.
So cheap tokens tell you the GPUs are well fed. They don’t tell you the model is weak. I find that a useful correction to a very human assumption, that price is a measure of quality. Sometimes it’s a measure of how full the bus is.
The one-line version
Your input gets speed-read once. The output gets written one word at a time, each word costing a full pass plus a little more memory, which is why you pay more for the answer than the question.
There’s something oddly familiar in that. Questions are easy to ask. Answers, the real kind, are built one careful step at a time, and each step has to account for everything that came before. Maybe that’s worth remembering the next time any of us expects a long, thoughtful answer instantly, from a machine or from a person.
Try this today
Next time you use an AI chat, try asking for a shorter answer, something like “in three sentences.” Notice whether it actually changes the usefulness of what you get. If you ever pay for AI by the token, the answer is the side billed at the higher rate, so asking for exactly what you need is the easiest saving there is. And if a chat has been running for ages, consider starting a fresh one. That long history is memory the model has to carry with every word.
Next lesson: why prompt injection can’t be patched away.