AI agents handle thousands of customer conversations, and some of the hardest problems are buried in the edge cases. Validators automatically identify specific behaviors and issues across those conversations, making it possible to find problems that would otherwise be nearly impossible to spot at scale.
So, when it comes to chat agents, everything is about conversations now, right?
Those pesky humans talking with our (amazing) chatbot. Using it gracefully. Everything works fine. There are never any issues.
Ever.
Yeah.
When things don’t exactly go our way, and some lovely human decides to say something like “last card deal” to our lovely banking agent, we have a small problem.
What do they mean?
Do they want to hear about the hottest deals on credit cards? Do they want to look at the latest deal for any of their cards? Or do they mean the deal associated with the last card they opened?
All of this ambiguity is just one of the many weird things a customer can throw at an agent.
And while it’s funny at first, it becomes a real problem at scale.
If only 1% of conversations contain some kind of ambiguity, and you’re handling thousands of conversations every day, how do you even find those conversations?
More importantly, how do you define what you’re looking for in the first place?
So yeah, we here at Avon AI built a cool feature called Validators (that I worked on for A LONG time), and I figured I’d tell you a bit about it.
So, what are Validators?
Validators are pretty simple.
You define a specific issue or behavior you care about, and the Validator looks for it across your conversations and alerts you when it finds a match.
That means instead of manually searching through thousands of conversations hoping to stumble across a problem, you can let the computer do the boring part.
For example, we could create a:
“Customer Said Something Ambiguous” Validator
And suddenly, instead of trying to find those “last card deal” conversations ourselves, we can automatically identify them.
Pretty neat.
Generally, Validators fall into two groups: Signals and Evaluators.
Signals
Signals are the more general-purpose type.
They look for things that can happen in almost any conversation, regardless of the specific business or agent.
For example:
- The agent repeats something when it shouldn’t.
- The customer becomes frustrated with the agent.
- The customer says something ambiguous.
- The conversation starts going off the rails.
These are behaviors that are useful to keep an eye on across pretty much every agent.
Evaluators
The second type is for behaviors that are much more specific to your business, your policies, or your agent.
For example:
- A customer curses, and the agent doesn’t apologize in the specific way we defined.
- A customer declares a sale, and the agent responds correctly but fails to log it properly in their account.
- A customer says “last card deal”, and instead of showing them the deal associated with their last card, we present the sweet 9% off deal they can get if they purchase a new card RIGHT NOW.
These are the kinds of things that might be perfectly fine in one business and completely unacceptable in another.
And that’s really the point.
You can define the behavior you care about, and then let Validators find it for you.
Finding issues is half the battle
At the end of the day, customers can ask an AI agent pretty much anything.
The challenge isn’t just building an agent that should behave correctly. It’s figuring out what actually happened across all those conversations.
Once we can automatically identify the conversations where something went wrong, we’re basically halfway through the improvement cycle.
Find Issues → Fix Issues → Make Sure Nothing Broke
And Validators are a core part of that first step, especially when you’re dealing with thousands (or eventually millions) of conversations.
Because you can’t improve what you can’t find.
And trust me, I’ve got some stories about some of the more interesting Validators I’ve built.
Stay tuned for Part 2.
Frequently Asked Questions
What are AI agent Validators?
AI agent Validators are automated checks that identify specific behaviors or issues in AI agent conversations. They can be used to find things like ambiguity, customer frustration, repeated responses, policy-specific behaviors, or other issues defined by the business.
What can Validators detect in AI agent conversations?
Validators can detect both general conversation patterns and business-specific behaviors. Examples include customer frustration, ambiguous requests, repeated agent responses, incorrect handling of specific customer actions, or failures to follow a defined business policy.
Why do AI agents need Validators?
AI agents can handle thousands of conversations, making it impractical for people to manually review every interaction. Validators automatically identify conversations that match behaviors a business wants to monitor, helping teams find issues at scale.
What is the difference between Signals and Evaluators?
Signals identify general behaviors that can be relevant across many AI agent conversations, such as frustration or ambiguity. Evaluators are designed around specific business rules, policies, or agent behaviors that matter to a particular organization.
How do Validators help improve AI agents?
Validators help teams find the conversations where an agent did not behave as expected. Those conversations can then be investigated and used to improve the agent, after which the agent can be tested again to make sure the change did not introduce new problems.
Can Validators monitor customer-facing AI agents at scale?
Yes. Validators are designed to analyze conversations across customer-facing AI agents, making it possible to identify specific behaviors without requiring teams to manually review every conversation.