Some links here are affiliate links — if you buy through them we may earn a commission at no extra cost to you. It never changes what we recommend. Full disclosure.
Disclosure: This article contains affiliate links. If you sign up through them, we may earn a commission at no extra cost to you — thanks for supporting the site.
When you're billed per token, every unnecessary word in your prompt is a tiny recurring cost. It's easy to ignore when it's one request — it's very easy to notice when it's a million requests a month. The fix usually isn't "use a smaller model," it's "stop sending tokens you don't need."
Why This Matters
Because serverless inference is stateless, your app resends context on every single request — system instructions, conversation history, any reference documents. If that context is bloated, you're paying for the bloat on every call, not just once.
Common sources of waste:
- System prompts that repeat themselves or over-explain simple instructions
- Sending full conversation history when only the last few turns matter
- Pasting entire documents into the prompt when a summary or relevant excerpt would do
- Requesting far more output than you actually use (long responses you then truncate)
A Simple Framework
- Audit your system prompt — read it out loud; cut anything you're saying "just in case"
- Trim conversation history to what's actually relevant to the current turn
- Summarize or chunk long reference material instead of sending it whole
- Set explicit output limits so the model doesn't generate more than you need
- Re-test after each change to confirm output quality hasn't dropped
Example
Before: "You are a highly capable, extremely knowledgeable assistant. Please read the following extremely carefully and take your time to think about every detail before responding thoughtfully and comprehensively..." followed by the entire 40-message chat history and a full product manual pasted in.
After: A concise system prompt stating the assistant's role and constraints in two sentences, the last 4 relevant messages, and a two-paragraph excerpt from the manual that's actually relevant to the current question.
> Tip: Keep a "before" and "after" token count for your main prompts. Watching the number drop is the fastest way to build the habit of writing lean prompts by default.
Common Mistakes
- Treating verbose, "polite" prompt language as harmless — it isn't, it's billed
- Sending the entire chat history on every turn out of convenience
- Pasting whole documents instead of retrieving just the relevant section
- Not setting a max output length, letting the model ramble
DigitalOcean's Serverless Inference API endpoints guide is a useful reference for how requests and context are structured across the Chat Completions and Responses APIs.
> (ad) GPU bills piling up for AI features you barely use? DigitalOcean Serverless Inference is pay-per-token, no idle infrastructure, no minimums. Ship the feature, not the server bill.
None of these fixes require new infrastructure or a smarter model — just a leaner prompt. It's the cheapest optimization available, and most teams never bother to make it.