Every agent system I have built runs on a graph of steps. In B:Side Assist, one step pulls in transactions, another categorizes them, another runs the actual financial analysis, and another summarizes what happened so the next step has context. Each of those steps needs a model behind it. And when it came time to decide which model belonged where, I did what nearly everyone building agents does. I guessed.
That is an awkward thing to admit, because I measured everything else. B:Side Assist had latency budgets, eval suites, retry logic, and dashboards for all of it. But the model assignment, the single biggest lever on both quality and cost, came down to instinct, or whatever had been praised on my timeline that week, or the simple rule that the newest frontier model is the safe choice. Nobody gets a hard time for putting the best model on the job.
Then it happened again with VCOS. Different domain, same decision. Pitch screening, due diligence, portfolio management, LP reporting, every workflow is its own chain of model calls. Twice now I had wired up an agentic system for real users, and twice I had assigned models to steps with a level of rigor I would never accept anywhere else in the stack. And with new models shipping every few weeks, each with its own benchmarks and its own promises, every release quietly asked me to re-litigate every one of those guesses.
The Safe Choice Has a Bill
Defaulting to the strongest model is defensible right up until the invoice arrives, and the invoice never arrives as one event. It accrues. Every call, every step in the graph, every retry, every user, every day. A multi-agent system might touch a frontier model several times to answer a single question, and most of those touches are bookkeeping. Routing a request to the right branch. Extracting fields from a document. Reformatting one agent's output so the next one can read it.
For an internal tool, you might never notice. For a consumer-facing product at real volume, it is brutal, and it gets worse on exactly the trajectory you want, which is up and to the right. When a small business owner asks B:Side Assist about their runway, the hard part is the analysis. It is not the step that reshapes a table, and it is definitely not the step that writes a two-line summary for the next agent in the chain. The reward for growth is a larger bill for work a cheaper model could have done.
Not Every Step Is a Hard Problem
Here is what I actually wanted, and it is not complicated. The same quality of output I was already shipping, with cheaper models running the layers that do not need a frontier brain. Most multi-agent architectures are not made of hard reasoning. They are made of plumbing. A step that extracts fields from a pitch deck. A step that routes a deal to the right workflow. A step that categorizes a transaction or summarizes a conversation. A cheap model is often perfectly good at all of it, and the expensive model should be spending your money on the genuinely hard reasoning, not on every step that happens to sit next to it.
The intuition is easy. Proving it for your own system is the actual work. A blind swap takes thirty seconds. What you cannot do in thirty seconds is know whether the swap quietly made things worse, because cheaper models fail in ways that do not throw errors. The extraction still returns valid JSON, it is just wrong on the edge cases. The summary still reads fine, it just dropped the one detail that mattered downstream. You do not find out from a stack trace. You find out from a user, later, when it is expensive. And when the product is handling someone's finances, that is not a risk you take on a hunch.
I Kept Rebuilding the Same Spreadsheet
So I hand-built the check. I took traces that had already run through the strong model, replayed them through a cheaper one, and compared the outputs step by step to see where the cheap model held and where it broke. It was never a product. It was a folder of scripts and a spreadsheet, stood up under a deadline because I needed the answer and nothing out there would give it to me. I built it for B:Side Assist. Then I built it again for VCOS. And every time I bothered, it paid for itself, usually by proving that some step I had been anxious about was safe to downgrade after all.
I assumed this was a quirk of how I worked. It was not. When I talked to other people shipping agentic systems, I kept hearing the identical description of the identical problem. Everyone had defaulted to the frontier model. Everyone suspected they were overpaying. Almost nobody had a rigorous way to know which swaps were safe, so almost nobody swapped anything, because a silent quality regression is scarier than a big bill. At some point, guessing stopped looking like a habit and started looking like a liability.
So I'm Building rightmodeler
rightmodeler replays your real agent traces through cheaper models. For each step, it judges the cheaper model's output against what you already shipped in production, the output you have already decided is good enough. Then it produces a recommendation report. Which downgrades hold up under evidence, and which do not. It runs on your own traces, not synthetic prompts, so the results reflect your actual workload instead of a benchmark that looks nothing like your system.
It is a report, not a gateway. It does not sit in your request path, route your traffic, or add a hop to your latency budget. You read the evidence, and you decide what to change and when. It is also open source, because the whole point is that you should be able to prove the savings on your own data before you believe anyone's claims, including mine.
The part I am most attached to is that it can say no. When the cheaper model's outputs do not hold up, or the sample is too thin to be sure, it declines to recommend the swap and tells you so. A tool that always finds savings is not measuring anything. The value is a verdict you can trust in both directions, including the boring, expensive verdict that the frontier model was the right call all along.
Why This Matters
Agentic systems are only getting deeper. More steps, more calls, more places where a model choice quietly compounds into real money. Every team shipping agents makes this decision at every step of every graph, and today most of them make it the way I did, by feel. I called that judgment for a long time. It was not judgment. It was a guess I had stopped noticing, and the fix was never a better instinct. It was evidence, drawn from the traces I had already run.
What started as a folder of scripts and a spreadsheet turned into something real. If you are building agents and quietly suspect that a frontier model is carrying steps that do not need it, you are probably right, and you no longer have to guess. Point rightmodeler at your own traces and read what comes back. I would love to hear what you find.