As an authority in the rapidly shifting landscape of enterprise SaaS and software architecture, Vijay Raina has spent years navigating the “duct tape and glue” era of AI implementation. His work focuses on transforming brittle, manual systems into resilient, cloud-native architectures that can handle the rigorous demands of modern business. With a deep background in designing complex retrieval systems, he brings a pragmatic, often skeptical eye to the hype surrounding new development platforms, preferring to focus on the “integration tax” that often sinks ambitious projects. In this conversation, we explore the tactical realities of building Retrieval-Augmented Generation (RAG) agents within the Azure AI Foundry ecosystem, moving beyond the polished demos to address the granular challenges of data grounding, security boundaries, and the shift from coding infrastructure to configuring it.
The following discussion examines the evolution of RAG pipelines from week-long manual “plumbing” tasks to streamlined afternoon configurations. We delve into the critical technical choices—such as opting for newer project types over legacy hub-based models—and the necessity of hybrid search for handling enterprise-specific terms like error codes and product names. We also touch upon the sophisticated concept of agentic retrieval for managing compound queries and the heavy lifting required for production-grade security, including Entra ID integration and document-level access control.
Transitioning from manual “plumbing” to a managed configuration approach drastically changes the development cycle, but how does this shift impact the way an architect views the “seams” of a RAG pipeline?
In the traditional approach, an architect was essentially a master plumber, spending a full week or more just trying to make sure the embedding job didn’t choke on the vector store’s requirements. You would find yourself hand-rolling vector search and writing custom chunking scripts, only to have the orchestration layer break every time a minor dependency was updated. The “seams” were where we usually failed—specifically the gap between a chunking strategy and the embedding model’s context window, which often led to garbage results in production despite looking great in a notebook. Moving to Azure AI Foundry feels like those seams have been cauterized because the platform now owns the integration points that used to eat the first sprint of every project. Instead of fighting the SDK or writing glue code, I spent my time thinking about whether the data was actually any good, which is a much higher-value use of an engineer’s brain.
When you first began rebuilding your agent, you mentioned moving away from legacy hub-based projects in favor of the newer Foundry project type. What specific technical friction does this newer structure resolve for a developer?
The decision to skip hub-based projects is about avoiding technical debt before you even write a line of code, as many old tutorials still push legacy structures that add unnecessary layers of complexity. By spinning up the newer Foundry project type, the deployment of a chat model and an embedding model becomes a “click, click, done” experience that genuinely takes only about five minutes. This immediacy is the first signal that you aren’t building infrastructure anymore; you are configuring an environment where endpoints are ready to use immediately. It removes the friction of manual endpoint management, allowing you to move directly into the data grounding phase without the typical “infrastructure tax” that used to stall progress. If you see a tutorial suggesting the hub-based approach today, my advice is to skip it entirely to ensure your agent is built on the current, most efficient framework.
You’ve advocated for hybrid search over pure vector search when dealing with enterprise documents. Could you walk us through a scenario where a vector-only approach might fail a user?
Pure vector search is incredible for semantic meaning, but it is notoriously bad at identifying exact, “jagged” strings like specific product names, error codes, or unique policy identifiers that populate enterprise PDFs. For instance, if a user searches for a specific error code like “ERR-992-X,” a vector model might point them toward general troubleshooting because it understands the “vibe” of a problem, but it might miss the exact document containing that code. By turning on hybrid search—which combines keyword matching with vector search—you ensure that those exact terms aren’t lost in the mathematical “averaging” of embeddings. It was a critical step in my afternoon build because it prevented the retriever from returning great-sounding but ultimately irrelevant content, which is the fastest way to lose a user’s trust.
The concept of “agentic retrieval” seems to be a solution for the messy way humans actually ask questions. How does this feature change the interaction between the LLM and the underlying search index?
Classic RAG is essentially a one-shot deal where the system fires a single query per user turn, but real-world questions are often “two questions wearing a trench coat.” If a user asks to compare the Q3 and Q4 policies to identify renewal changes, a standard system might get overwhelmed trying to find both sets of data in one go. Agentic retrieval solves this by breaking that compound question into sub-queries, running them in parallel, and then merging those results before the generation phase even begins. It transforms the agent from a passive searcher into an active researcher that understands it needs to look in multiple places to provide a coherent answer. I’ve found that turning this on from day one is essential because retrofitting it later, after your users have already started asking messy, multi-part questions, is far more painful than setting it up at the start.
While the infrastructure setup has become faster, you warned that security and access control are still very much a manual burden. What are the specific risks of relying on prototype-level security like API keys in a production environment?
API keys are a dangerous crutch that many developers use to get a demo running, but they have no place in a customer-facing RAG system. The real work involves migrating to Microsoft Entra ID, which is a full afternoon of work in itself and not just a simple checkbox you can click at the last minute. Without document-level security filters configured in Azure AI Search, you run the risk of your agent “hallucinating” access, where it articulately leaks sensitive HR files to a sales team member simply because the retriever found the data. Furthermore, we have to treat every retrieved document as untrusted input; a poisoned PDF with a prompt injection attack is a legitimate threat that can manipulate the model’s behavior. Your system instructions must be robust enough to assume the data might be trying to trick the model, requiring a level of architectural skepticism that a “quick” setup often ignores.
Given the “compounding costs” of embeddings, storage, and tokens you mentioned, how should an organization model their expenses before committing to a large-scale RAG deployment?
The costs of RAG are deceptive because they stack up in ways that aren’t immediately obvious during a low-traffic demo. You aren’t just paying for the final chat completion; you are paying for the embedding generation, the ongoing storage of the index, and the massive amount of extra tokens consumed by stuffing retrieved passages into every single prompt. If you choose a chunking strategy that is too granular or too broad, those costs can compound faster than your budget can handle once you hit real production traffic. I recommend modeling these costs against your expected document volume and query frequency before you finalize your chunking strategy, rather than trying to fix an expensive architecture after it’s already live. It’s about being as efficient with your token “spend” as you are with your manual coding time.
What is your forecast for the future of agentic AI within these managed ecosystems?
I believe we are moving toward a “zero-glue” future where the orchestration of multi-agent systems becomes as native as a dropdown menu, but the true differentiator will remain the quality of the underlying data. As platforms like Azure AI Foundry continue to absorb the boring “integration tax” of RAG, the competitive advantage will shift away from who has the best “plumbing” and toward who has the most sensible chunking strategies and the tightest security boundaries. We will see agents that are not just retrieving information but are capable of performing complex reasoning across siloed enterprise datasets with minimal developer intervention. However, the human element—ensuring that the model is grounded in “truth” and that access controls are ironclad—will remain the most difficult and most important part of the job for any AI architect.
