When you read a sentence, you see whole words. But an AI doesn’t read words the way we do. Instead, it chops text up into smaller, bite-sized pieces called tokens so it can process them. For example: - Short, common words usually equal one token (e.g., "cat", "the", "and"). - Long or complex words get broken into pieces (e.g., "unforgettable" might be broken into "un", "forget", and "table", that's three tokens). - Punctuation and spaces also count as their own tokens (e.g., "!" or "?").

Why AI Tokens Are So Expensive and the 7 Questions Companies Should Be Asking

EO Design Group
Jul 5
4 min read

Updated: Jul 20

The cost of doing business with artificial intelligence is shifting. As major vendors like Microsoft and OpenAI begin pulling back their initial subsidies, companies are realizing that the underlying engine of AI, tokens, comes with a hefty price tag that compounds rapidly. To control these compounding costs, organizations must first understand the infrastructure driving them. Here is a breakdown of what tokens actually are, why AI tokens are so expensive, and the 7 questions companies should be asking.

What Are Tokens?

When you read a sentence, you see whole words. But an AI doesn’t read words the way we do. Instead, it chops text up into smaller, bite-sized pieces called tokens so it can process them.

For example:

Short, common words usually equal one token (e.g., "cat", "the", "and").
Long or complex words get broken into pieces (e.g., "unforgettable" might be broken into "un", "forget", and "table", that's three tokens).
Punctuation and spaces also count as their own tokens (e.g., "!" or "?").

Why do Tokens Matter?

The era of predictable flat-rate AI pricing is fading. Tech giants are moving toward consumption-based models. Take Microsoft's recent policy change for GitHub Copilot: "Usage will be calculated based on token consumption, including input, output, and cached token". For unprepared businesses, this shift will drive costs up exponentially. To understand why, we need to look at how AI communicates compared to people.

Talking to a Person: You are chatting with a friend at a coffee shop. As the conversation flows, your friend naturally remembers the main points. You don't need to remind them every two minutes what you are talking about.
Talking to an AI: Imagine chatting with someone who has short-term memory. Every time you say something new, they completely forget who you are and what you were talking about. To fix this, you have to hand them a script of everything you’ve said since the beginning of the conversation, plus your new question. This is how AI chat works, even for the simplest chat.

Because AI has to keep track of the entire conversation, the longer you chat, the larger that "script" becomes. And remember: before the AI can even read that growing script to give you an answer, it has to chop every single word and punctuation mark into tokens. This is why:

Why AI Tokens are so Expensive

Because the AI has to re-read the whole "script" every time, the token count grows very fast. Imagine a back-and-forth chat where each message is worth 100 tokens:

Message 1: The AI reads 100 tokens.
Message 2: The AI has to re-read Message 1 plus your new message (200 tokens).
Message 3: The AI re-reads Messages 1 and 2 plus your new message (300 tokens).

By the time you are 10 messages deep, the AI is processing thousands of tokens just to answer a simple "Yes or no" question.

It's like paying a lawyer who charges you by the hour, but insists on re-reading your entire legal file from scratch every time you call them with a quick question!

7-Questions Companies Should Be Asking

AI is an incredible tool when used correctly. The goal isn't to stop using AI, but to use it efficiently. To make sure you are maximizing its value while keeping token costs under control, evaluate your AI use against these questions:

Is this task unpredictable? AI is great for summarizing documents or brainstorming, but overkill (and expensive) for predictable tasks. For example: calculating sales tax, querying data from a database, or converting currencies.
Could a traditional system do this? If your problem can be solved with a basic "If/Then" rule (like: If a customer clicks 'Billing,' then send them to the billing team), a simple database search, or a standard drop-down menu, you don’t need AI. Don't pay for AI to do a job that standard software can do instantly for free.
Do you have real-time cost alerts? Set up automated alerts that flag your engineering team the moment token consumption spikes unexpectedly.
Are you using the right-sized model? Use smaller, faster, cheaper models for simpler tasks.
Do you have a person auditing the costs? Assign a team member to review the ROI (Return on Investment) of your AI tools monthly to ensure the efficiency gained outweighs the token bill.
Do you have a fallback plan? If the AI hits its budget limit for the month, is there a seamless backup system (like a standard form or a person) ready to take over?
Are you using AI just for the hype? Ensure the AI is solving a genuine problem, not just acting as a marketing buzzword.

Want a Deeper Dive?

For an excellent, deep-dive breakdown on this topic, check out Computerphile’s video if you want to see a step-by-step breakdown of how token costs spiral out of control. It’s a must-watch for anyone looking to understand the real cost of AI.