Vijay Raina is a seasoned leader in the enterprise SaaS and software architecture space, bringing a wealth of technical depth to the rapidly evolving world of agentic AI. As companies move beyond simple chatbots toward systems that can independently plan, retrieve data, and execute complex workflows, the traditional methods of measuring success are falling short. In this discussion, we explore why the industry must shift from evaluating a single output to a rigorous, stage-by-stage governance framework. We delve into the complexities of the agentic pipeline—from initial query clarity to the nuances of retrieval-augmented generation—and explain how granular diagnostics can prevent costly debugging cycles while ensuring AI systems earn the right to act on behalf of the enterprise.
The shift from simple Large Language Models to multi-step agentic systems has introduced layers of complexity that traditional testing cannot handle. Why is it no longer sufficient to simply compute an accuracy score for an AI’s final output?
Relying on a single accuracy score is a dangerous oversimplification when you are dealing with a system that has the autonomy to call APIs, write code, and synthesize real-world decisions. In the old playbook, we treated AI like a black box where a query went in and an answer came out, but agentic AI is a multi-layered machine with at least six distinct stages, each carrying its own unique failure mode. If you only measure the final response, you might see a hallucination and blame the generation model, but the actual poison often entered the system during the second stage when the retrieval module pulled irrelevant documents. This creates a frustrating “whack-a-mole” scenario for developers who end up debugging the symptoms of a failure rather than the root cause. Without a stage-by-stage breakdown, an enterprise team lacks the necessary governance to trust an agent with sensitive tasks, as they cannot pinpoint whether a failure originated in the initial planning, the tool selection, or the final synthesis.
You’ve emphasized that the quality of an agent’s response is heavily dependent on what happens before a single word is even generated. How can teams better evaluate the input and retrieval phases to prevent downstream errors?
The first two stages—Input Reception and Retrieval—are where the most high-leverage quality checks occur because they prevent a chain of wasted computation before it even starts. At the input stage, we look for query clarity and completeness to catch ambiguous requests before they propagate, which is a low-cost way to ensure the agent isn’t being set up for failure. When we move into the retrieval and grounding phase, we rely on the RAGAS framework, which formalizes four critical metrics: context relevance, context recall, context precision, and context entity recall. These metrics are the primary defense against hallucinations, as poor retrieval doesn’t just lower accuracy; it actively feeds the model incorrect material to “reason” from. If your context precision is low, you are essentially asking your AI to find a needle in a haystack of noise, and no amount of prompt engineering at the generation stage can fix that fundamental data deficit.
When an agent moves from gathering information to actually planning and executing a task, the risks become much more tangible. What metrics should we use to monitor the “thinking” process and the actual tool interactions?
This is where the evaluation shifts from information retrieval to pure reasoning and operational reliability, and the stakes feel much heavier because the agent is now preparing to act. In the planning stage, we must measure task decomposition quality to see if the sub-goals are logically sound, and track tool selection accuracy to ensure the agent isn’t trying to use a screwdriver when it needs a hammer. I also advocate for an “action safety score,” especially for plans involving irreversible actions like deleting files or publishing content; these should trigger a mandatory human review if the irreversibility threshold is too high. Once we hit the execution layer, we monitor the task success rate and latency per step, which often reveals if a system is becoming brittle or over-engineered. A high retry rate at the execution level is a sensory red flag—it almost always tells me that the problem isn’t a broken tool, but rather a failure in the stage-three planning that sent the agent down a logical dead end.
The generation stage is often where users first notice a problem, particularly hallucinations. How do you distinguish between an agent that is “well-spoken” and one that is actually being “faithful” to its sources?
Faithfulness is the absolute cornerstone of anti-hallucination efforts, as it measures the exact proportion of claims in an output that are explicitly supported by the retrieved context. An AI can produce a response that is coherent, structurally sound, and seemingly professional, yet be completely untethered from the factual data it was given. To combat this, we use a minimum viable evaluation surface that includes faithfulness, answer relevance, the hallucination rate, and response coherence. We often use a secondary language model to act as a judge, critiquing the primary agent’s output against a strict rubric to catch unsupported factual claims. This allows us to see if the agent is actually resolving the user’s query or if it is just providing a confident-sounding answer that lacks the necessary grounding in the provided documents.
Once an agentic system is in production, the data flow becomes massive. How does having granular, stage-level diagnostics change the way a team handles a sudden drop in performance?
The diagnostic value of this framework is the difference between a quick fix and weeks of expensive, aimless debugging. Consider a scenario where your system’s faithfulness score suddenly drops from 0.87 to 0.71; without stage-level metrics, a team might waste time swapping out the entire language model or rewriting every prompt in sight. However, with the right instrumentation, you can look upstream and see that context precision at stage two has simultaneously plummeted from 0.82 to 0.54. This tells you immediately that the problem is a recently updated document corpus in the vector store, not the model’s ability to generate text. By identifying that the generation model was never the problem, you save the team from two weeks of debugging the wrong layer and refocus your resources on fixing the retrieval quality.
What is your forecast for the evolution of AI agent governance over the next few years?
I believe we are moving toward a future where “Agentic Observability” becomes a standard category of the enterprise software stack, moving far beyond the simple logging we see today. We will likely see autonomous quality gates where “critic” models are embedded into every stage of the pipeline, automatically halting execution if the task decomposition quality or action safety scores fall below a certain threshold. Enterprises will stop asking if a model is “smart” and start asking if their specific agentic architecture is “governable” across its entire lifecycle. Eventually, the ability to provide a detailed audit trail of every stage—from the 0.82 precision in retrieval to the final self-critique in the feedback loop—will be the only way AI agents earn the trust required to handle high-stakes business processes.
