Performance Systems

Build the Scoreboard First

By Chad T Dyar 7 min read

I am telling a COO about the things I have been building with AI, and I am enjoying myself. Then she asks the question operators always ask: how do we know an agent drives revenue.

I am telling a COO about the things I have been building with AI, and I am enjoying myself. This happens when you spend a few years building: you accumulate stories, and given an audience, the stories come out. Agents that read email. Agents that enrich prospect lists. An agent whose entire job is managing my own knowledge base (I named him Corpus, and yes, I talk about him like a colleague).

The COO listens the way operators listen, which is to say patiently, and then asks the question operators ask.

How do we know an agent drives revenue?

Here is what I did not say out loud in that moment: welcome to enablement.

Because that question is not an AI question. It is the question my entire discipline has carried for as long as the discipline has existed. Enablement has always lived in the gap between activity and outcome. We train the sellers, we build the programs, we ship the content, and then a quarter later someone in finance asks what any of it earned, and the honest answers have always been harder to produce than the confident ones. AI did not create the attribution problem. AI just made it impossible to ignore, because now the activity is cheap, constant, and machine-generated, and the old habit of gesturing at a busy dashboard and calling it impact collapses under its own volume.

So I approach an agent the same way I approach any enablement challenge, with the same five questions in the same order. What outcome are we actually after? What metrics illuminate that outcome, as opposed to merely decorating it? What is the full flowchart from activity to outcome, every step, no skipped arrows? Where in that flowchart does the agent actually sit? And given all of that, what can we confidently attribute to the agent, and what can we not?

That last clause is the one people skip. Attribution is as much about what you refuse to claim as what you claim.

The interesting thing, and the reason this conversation stayed with me, is what happens when you answer those five questions before you build instead of after. Most measurement is a report card: the thing exists, and now we grade it, and the grading bends to flatter the thing. But answer the questions first and the measurement stops being a report card. It becomes an architect. It starts making decisions for you.

I watched this happen in my own work. I recently built a system where two sales agents log every action as a touchpoint tied to a lead, with the credit rules written into the database before either agent ran. And because the measurement came first, the measurement pushed back on the build. At one point an agent produced a perfectly reasonable-sounding judgment, and the schema rejected it, because the build had never defined what that judgment meant or what credit it deserved. The old me would have patched around that in an afternoon. Instead the rejection was the system working: if the measurement layer cannot account for an action, the action does not get to happen quietly. The flowchart from the five questions was not documentation of the build. It was the boss of the build.

The chefs figured this out a long time ago, by the way. I have been binging cooking shows lately (see the Shelf below), and the thing that separates a chef from a cook is not technique. It is point of view: an opinion about what the dish is for, held before the cooking starts, that every choice afterward has to answer to. Measurement-first building is the same move. The point of view comes first. The build answers to it.

There is a cost to working this way, and I will name it, because the confident version of this essay would not. Building the scoreboard first is slower at the start. It forces uncomfortable conversations early, the ones about what you will not claim, at exactly the moment when everyone wants to talk about what the shiny new thing will do. It means your first numbers arrive small and flagged instead of large and impressive, because honest instruments refuse to say more than the sample supports. I launched a public dashboard recently that displays, at this moment, essentially nothing. That was the design. It fills as the truth fills.

And the payoff is the conversation with the COO. Not that one specifically, but the one it represents, the one every builder eventually lands in, where a serious operator asks the serious question. If you built the scoreboard first, that question has an answer waiting: here is the outcome we chose, here are the metrics that reveal it, here is the flowchart, here is where the agent sits, and here is exactly what we credit it, under rules we wrote down before we had any reason to want the rules to flatter us. If you built the scoreboard after, you are negotiating with your own data, and everyone in the room can tell.

People ask the attribution question at the end, when the build is done and the honest answer is already out of reach. The question I keep turning over is what happens when you move it to the front.

What would you build differently if you had to prove it before you built it?

On the Shelf This Week

The agentic research rabbit hole. No books lately, just a steady diet of what people are building with agents: what's actually useful, how to decide what deserves to be built, and how any of it scales. The ratio of loud to useful is rough out there. The useful stuff is almost never the flashy stuff.

My own back catalogue. The strangest thing I've read this month is me. A few years of building with AI leaves a trail, and I went back through all of it, every idea and every build. I even have an agent named Corpus whose whole job is keeping that knowledge base honest. Past me had some good ideas and some terrible ones, and telling them apart is its own education.

Conversations as research. Lately my best source isn't written down anywhere. I've been talking with executives across sales, operations, and enablement about how they think about AI and what's coming, then bouncing my own ideas off them. What they push back on teaches me more than what they nod at.

Cooking shows, in bulk. I've been binging them, and not for the recipes. Watch enough chefs and you start seeing the real subject: expertise, technique, and point of view, and how much your success rides on having all three. A cook follows the recipe. A chef has an opinion about it.

Kaizen and DMAIC. The idea I keep coming back to. Lean Six Sigma principles applied to operational builds, because building with AI doesn't exempt you from needing a method for constant improvement. If anything it raises the bar. The tools got faster; the discipline didn't get optional.

P.S.

That system I mentioned, the one where the scoreboard bossed the build around? I packaged it. Two sales agents with the attribution layer built in, self-deploy, the whole methodology included so you can hand it to your own skeptic. It lives at stack.chadtdyar.com if the COO's question is one you're carrying too.

Talk in two weeks.

Back to Blog | Home