Why Token Efficiency Matters When Evaluating On-Premises and Frontier Models

Zusammenfassung

Most enterprise AI cost conversations focus on model price. That's the wrong variable. Model price is only one lever of total cost of ownership (TCO), and it's often the least important one. The lever most teams overlook is how much work reaches the model in the first place.

Real usage data from Rocket Software shows what that difference looks like. About 300 daily users averaged roughly four million tokens per user per month. At frontier model pricing, that's an estimated $15–$25 per user per month. For a 50-person mainframe team, that adds up to about 200 million tokens and roughly $1,000 in monthly model costs. Under an existing enterprise AI agreement, that cost can approach zero.

In this paper, you'll learn:

• Why mainframe operational data turns into a token problem so easily
• The seven techniques Rocket^® EVA.Next^™ uses to cut unnecessary model consumption, including Token-Oriented Object Notation (TOON), leaner Model
• Context Protocol (MCP) and Agent2Agent (A2A) toolsets, and targeted System Management Facility (SMF) retrieval
• How on-premises and frontier models compare across every TCO dimension
• A five-question framework you can use to evaluate any AI architecture on its merits
• The real question isn't on-premises versus frontier. It's which architecture gives you the right intelligence, governance, and flexibility at the lowest total cost.
   

1. Introduction: the AI cost conversation is focused on the wrong thing

Sit in on almost any boardroom discussion about enterprise AI, and you'll hear the same debate. On-premises models or frontier models? And the deciding factor is almost always price per token.

That instinct makes sense. Price is visible, easy to compare, and simple to put on a slide. But it's incomplete.

Price is just one part of AI TCO. Infrastructure, operational overhead, data governance, and model capability all shape what you actually spend. And one more lever decides whether any of those factors matter at all: how much work reaches the model.

You can choose the cheapest model on the market and still overspend if your architecture sends every mainframe log, SMF record, and system trace straight into the model's context window. Flip that around, and a thoughtful architecture can make almost any model, on-premises or frontier, run efficiently and affordably.

This paper gives mainframe organizations and IT decision-makers a practical way to evaluate AI architectures on true total cost, not sticker price. Throughout, we'll use EVA.Next, Rocket Software's agentic AI platform for mission-critical operations, as a working example of token-efficient design in practice.

2. Why mainframe data becomes a token problem

Mainframe operational data has three traits that make it especially prone to driving up AI costs. It's high in volume, repetitive in structure, and deeply technical. A single z/OS^® environment can generate millions of SMF records, log entries, and performance metrics every day. None of it was built with a large language model's context window in mind.

When you pass that data straight to an LLM without preprocessing, three things happen at once:

1. Token counts balloon. Raw operational data is verbose and repetitive by nature.
2. Costs climb in step. Most frontier pricing charges per token, so more tokens mean a bigger bill.
3. Answer quality drops. Flooding the context window with redundant or irrelevant data makes it harder for the model to reason clearly and surface the right insight.

A fourth concern makes all three worse: control. Raw mainframe data can include sensitive customer records, transaction histories, and proprietary business logic. Sending it to an external model raises real governance and compliance questions, especially in banking, insurance, and healthcare.

So the key is clear. You need an architecture that processes data close to its source and refines it before it ever reaches a model.

3. The EVA.Next approach: process close to the mainframe, give the model only what it needs

Rocket Software didn't add token efficiency to EVA.Next as an afterthought. We built the platform around it from day one, guided by one simple principle: process data close to the mainframe, and give the model only what it needs to reason.

Two components put that principle to work:

• Automatic Context Engineering (ACE): ACE manages how your knowledge assets and Rocket Software's knowledge assets come together. It applies context refinement rules and chain-of-thought prompts to distill what's relevant before anything reaches the model.
• Rocket Intelligent Router: The router orchestrates EVA.Next's pluggable agent framework. It classifies each question, detects intent, selects the right plug-ins, and analyzes query complexity so every request takes the most efficient path.

This approach reflects Rocket Software's broader commitment to governed, deterministic agentic AI for mission-critical systems. EVA.Next uses policy-as-code to govern what actions an agent can take. It applies the same discipline to what data an agent can pass along. In EVA.Next, governance and efficiency come from the same design.

The payoff goes beyond lower cost. Every answer gets better, because the model reasons over signal instead of noise. Next, let's look at the specific techniques that make that possible.

4. Seven techniques for reducing unnecessary model consumption

EVA.Next uses seven distinct techniques to keep token consumption lean. Each one targets a different stage of the path from raw data to insight.

Compact data formats (TOON)

Token-Oriented Object Notation shrinks structured data before it reaches the model. It turns verbose data structures into a leaner format without losing meaning.

Leaner MCP and A2A toolsets

EVA.Next exposes only the tools and agent connections a task actually needs. Tool definitions stay compact, so they don't crowd the context window before the real work even starts.

Analysis outside the context window

EVA.Next's Python sandbox runs computation and data analysis outside the model's context entirely. Instead of pushing raw results through the model, EVA.Next computes first and shares only the relevant output.

Specialist agents

Domain-specific agents for CICS, Db2^®, MQ, and security narrow each query from the start. A specialist agent doesn't need the full breadth of your enterprise data. It only needs what's relevant to its domain.

Targeted SMF retrieval

Rather than pulling broad historical SMF data sets, EVA.Next uses time-aware prompts to retrieve only the window of data a question requires. That can be the difference between megabytes and gigabytes of unnecessary context.

On-demand Skills

EVA.Next loads Skills only when a task calls for them, instead of keeping them in every context by default. The model's working context stays focused on what matters right now.

SMF caching

EVA.Next caches previously retrieved and processed SMF data. When a question repeats, the platform doesn't spend tokens re-analyzing data it's already processed.

Routines and slash commands add one more layer. They give your system operators, security, and networking teams repeatable, pre-scoped actions that stay efficient by design. Each technique cuts cost on its own. Together, they compound into the efficiency gains you'll see in the usage data below.

5. Real usage data: what token efficiency looks like in practice

Architecture principles only matter if the numbers back them up. Here's what we saw in production.

In August 2026, about 300 daily users of the EVA.Next AI Framework averaged roughly four million tokens per user per month. At typical enterprise frontier model pricing, that's approximately $15–$25 per user per month. That's less than many standalone software subscriptions, for a platform that delivers end-to-end mainframe diagnostics.

Now scale that to a 50-person mainframe team, a realistic size for many enterprise operations groups:

One variable can change this math entirely. If your usage falls under an existing enterprise AI agreement, marginal model cost can approach zero. For organizations that have already negotiated AI capacity as part of a broader technology agreement, token-efficient architecture doesn't just lower cost. It can effectively remove it.

The lesson is worth remembering: architecture, not model choice alone, decides whether AI stays affordable at scale. A 50-person team running AI-powered diagnostics across its mainframe estate for about $1,000 a month isn't a hypothetical. It's what efficient design looks like in production.

6. On-premises vs. frontier models: a full TCO comparison

No single factor decides whether on-premises or frontier models fit your organization better. You only see the true cost when you weigh every dimension together.

Both columns carry real advantages. On-premises models give you strong data control and predictable infrastructure costs, which matters enormously in regulated industries. Frontier models give you elasticity, steadily improving capability, and freedom from hardware upkeep.

Neither column wins outright, and that's the point. The right answer depends on how well your architecture manages everything in between.

7. The cheapest model isn't necessarily the cheapest strategy

It's tempting to assume the lowest per-token price leads to the lowest total spend. The usage data in Section 5 tells a different story.

Architecture, including the techniques in Section 4, can matter more to total cost than any model's per-token rate. A premium model fed only the context it needs can cost less in practice than a discount model fed unfiltered mainframe data.

That's where model flexibility becomes a real cost advantage. EVA.Next supports multiple models and routes each workload by its requirements. You match model capability to task complexity instead of defaulting to one model for everything.

For example, a routine question about batch job status doesn't need the same reasoning power as a complex, cross-system root cause analysis. EVA.Next's routing reflects that difference.

So the lasting cost advantage isn't finding the cheapest model. It's building an architecture flexible enough to use the right model, with the right amount of context, for each task.

8. Beyond cost: the other benefits of token efficiency

Token efficiency isn't only about the monthly bill. It delivers three more benefits that build on each other.

• Better answers: Less noise means more accurate, relevant responses. A model reasoning over a tightly scoped, time-aware SMF retrieval gives you a sharper answer than one wading through months of undifferentiated logs.
• Less data exposure: When less data reaches the model, you shrink the surface area for governance and compliance concerns. In banking, insurance, and healthcare, that's not a minor benefit. It often decides whether an AI architecture is viable at all.
• Greater model choice: Lean, well-scoped queries make it practical to route work to a wider range of models, including smaller or on-premises models, without giving up quality. When every query doesn't need a model's full context window, you gain more freedom to choose the right model for the job.

These benefits stack on top of the cost savings. Efficiency-first architecture isn't a trade-off against quality or governance. It's what makes both possible.

9. Making AI consumption measurable

You've likely spent years applying FinOps discipline to cloud spend: tracking consumption, attributing costs to teams, and building accountability into budgets. AI consumption deserves the same rigor, and most organizations aren't there yet.

EVA.Next makes token usage visible at the user, admin, and organization levels. Instead of a single aggregated bill at month's end, you see exactly which teams, use cases, and query types drive consumption.

That visibility helps you in three ways:

1. Budget accurately with real usage data instead of guesswork.
2. Strengthen governance by giving compliance stakeholders a clear record of AI activity across your organization.
3. Prove ROI by turning an abstract line item into a measurable, defensible business case.

Measurability also sets up the framework in the next section. Without visibility into where your tokens go, you can't confidently answer any of the five questions that follow.

10. A framework for evaluating AI TCO

Whether you're evaluating on-premises, frontier, or hybrid AI, these five questions move the conversation beyond model price:

1. How much data actually reaches the model? Find out whether your architecture sends raw, unfiltered data or a refined, purpose-built context.
2. Where does analysis happen, inside or outside the model's context window? Computation outside the model, like EVA.Next's Python sandbox, cuts unnecessary token use.
3. Can you route different workloads to different models based on need? Matching model capability to task complexity often saves more than any single model's price.
4. Can you measure and attribute consumption at the user and organization levels? Without visibility, you can't manage or optimize spend.
5. What does the complete architecture cost, beyond the per-token price? Account for infrastructure, operational overhead, governance requirements, and flexibility, not just the headline rate.

This framework shifts your evaluation from model price in isolation to architectural efficiency, the real driver of value. You can apply it right away, whether you're assessing EVA.Next or any other platform.

11. Conclusion: the future isn't on-premises or frontier. It's efficient AI.

The real question for mainframe organizations isn't on-premises versus frontier models. It's which architecture delivers the right intelligence, governance, and flexibility at the lowest total cost.

EVA.Next puts that principle to work today by processing data close to the mainframe and giving the model only what it needs. The usage data proves it: a 50-person team runs AI-powered mainframe diagnostics for roughly $1,000 a month, and that cost can approach zero under an existing enterprise AI agreement.

We're not stopping there. Rocket Software's planned work includes enhancements to the RAG engine and framework orchestration tools, extending token optimization as EVA.Next's capabilities grow.

As you evaluate AI for your mainframe environment, you can ask a better question than "Which model costs less per token?" Ask which architecture gives you the most value for every token you spend. Rocket Software is here to help you answer it, one step at a time, so you can modernize with AI without disruption and without unnecessary cost.