In the rapidly evolving world of enterprise AI, few names carry as much weight in software architecture as Vijay Raina. As a specialist in enterprise SaaS and a seasoned architect, Vijay has spent the last eighteen months hands-on in the trenches, designing MoJoCo—an agentic modernization platform built to handle the complexities of regulated industries like banking and global supply chains. He brings a rare, “field manual” perspective to the table, arguing that the success of agentic systems isn’t determined by which glamorous large language model a team picks, but by the rigorous engineering of the data pipeline that feeds it. His philosophy centers on the idea that context is a managed asset, not just a retrieval step, and his insights challenge the common industry obsession with massive context windows as a silver bullet for reasoning failures.
The following discussion delves into the intricate architecture of context engineering, moving beyond simple prompts to explore a sophisticated five-stage runtime pipeline: Gather, Enrich, Verify, Compress, and Inject. Vijay explains how these stages map onto four critical architectural layers—Acquisition, Refinement, Distribution, and Evolution—to create a system that is both extensible and diagnosable. Throughout the conversation, he highlights the technical trade-offs of hybrid retrieval, the necessity of hierarchical context trees, and the importance of “landmark attention” for reducing token costs without sacrificing the reasoning signals that allow agents to make high-stakes decisions in production environments.
Many teams assume that upgrading to a larger context window is the definitive solution for reasoning failures, but you’ve observed that this often backfires in production. Why does a bigger window frequently lead to what you call ‘context-window collapse’ instead of better results?
The instinct to throw a bigger model at a reasoning failure is a trap I’ve seen teams fall into across banking and supply chain sectors more times than I can count. When an agent starts losing the thread at turn nine, the team sees a “noisy” output and immediately upgrades to a larger context window, only to find the exact same failure recurring at turn fourteen a few weeks later. The reality is that a bigger context window actually causes collapse because it lowers the cost of pushing “junk” into the prompt, forcing the agent to reason around irrelevant noise. You end up with the most critical piece of context buried in the middle of a massive prompt, and because of how models prioritize information, they often ignore that middle section entirely. True context engineering isn’t about volume; it’s about the upstream work of gathering, enriching, and verifying data so that the reasoning signal remains pure and focused, preventing the structural hallucinations that happen when a model is overwhelmed by 100k tokens of unrefined data.
You advocate for a dual view of the pipeline—one that is runtime-sequential and another that is architectural and concurrent. How does balancing these two perspectives prevent a system from becoming either a ‘fragile monolith’ or an over-engineered platform?
It is essential to maintain both views because they solve different problems for different stakeholders at different times. The runtime view—Gather, Enrich, Verify, Compress, Inject—is what the debugger uses when a specific request goes off the rails; it’s a sequential flow that allows us to see exactly what happened to a single payload as it moved from raw data to an injected prompt. On the other hand, the architectural view of Acquisition, Refinement, Distribution, and Evolution is what I use as an architect to ensure the system is extensible and has clear service boundaries. If you only build the runtime view, you end up with a single, massive Python function that might work for one month but breaks the moment you try to add a new domain or data source. Conversely, if you only focus on the beautiful four-layer architecture, you’ll find yourself unable to diagnose why a specific customer’s query produced bad context because you lack a trace of the individual payload’s journey through those layers.
When it comes to the ‘Gather’ stage, you’ve noted that relying solely on dense vector retrieval often fails for literal data like SKUs or regulatory citations. What are the specific mechanics of the hybrid retrieval and ‘alpha’ tuning you use to fix this?
The failure of pure dense retrieval is palpable when you’re looking for a literal match—like a specific container ID or a customer SKU—and the model confidently returns something that is “semantically” similar but factually useless. To solve this, we run dense and sparse retrievers, like BM25, in parallel and then fuse their results using a weighted alpha parameter, which is arguably the most important knob in the entire pipeline. In our implementation, we don’t just use a static alpha of 0.5; instead, we route queries through a lightweight classifier that adjusts the weight based on intent. For instance, if a query contains a regex match for an account number or a regulatory citation, we push the alpha down toward 0.2 to favor the literal precision of sparse retrieval. If the query is more exploratory, such as looking for “similar past exceptions in cold-chain logistics,” we shift the alpha toward 0.8 to prioritize semantic relevance, ensuring the agent gets the right balance of facts and concepts.
The ‘Enrich’ stage moves beyond a flat bag of documents toward a hierarchical context tree. How does this structure actually help an agent navigate complex information without getting lost in the details?
Treating retrieved documents as a flat list is a recipe for confusion, so we organize them into a recursive tree structure that includes root summaries, synthesis nodes, and raw fact leaves. This design allows the agent to navigate the tree by depth, pulling a high-level summary for a quick initial assessment and only descending into the leaves when a specific sub-decision requires granular facts. We use integer levels rather than categorical types for these nodes because levels are inherently composable, making it much easier to extend the system later without fighting a rigid taxonomy. Each node, even at the summary level, carries its own metadata regarding provenance, time windows, and confidence scores. This means the agent isn’t just seeing a summary; it’s seeing a summary and knowing whether it was derived from five reliable, recent sources or one single, stale data point, which is critical for making “reasoning-grade” decisions.
In regulated industries like banking or healthcare, you’ve mentioned that the ‘Verify’ stage is often the difference between a successful deployment and a hallucination post-mortem. How do you implement a policy-driven verification step that effectively handles ‘ground truth’?
The Verify stage is where we encode actual business policy into the pipeline, and it’s something I believe teams should implement on day one rather than waiting for their first major failure in month four. We use this stage to cross-reference facts against a ground-truth registry—for example, in a supply chain, a carrier’s tracking API is the ground truth for a container’s location, but our policy might dictate that if an update hasn’t been pushed in eight hours, a manual ops log takes precedence. We also apply contrastive ranking to demote documents that are conceptually similar to the query but factually divergent, which acts as a powerful shield against hallucinations. This stage is also the natural home for compliance, where context is scrubbed for PII or regulatory cleanliness before it ever touches a prompt template. By making these trust decisions in the pipeline, we ensure the agent doesn’t have to waste its reasoning budget trying to figure out which of two conflicting sources it should believe.
You’ve highlighted that naive truncation in the ‘Compress’ stage is dangerous because it removes contrast. How does ‘landmark attention’ preserve the reasoning signal while still achieving a 70% reduction in token costs?
Naive truncation is a silent killer of agentic accuracy because when you only keep the top-K most relevant documents, you throw away the “close-but-different” cases that keep the model from over-generalizing and reaching a confident but wrong answer. Instead of just cutting the text, we use landmark attention to identify high-information tokens—landmarks—distributed throughout the full context, preserving those at high fidelity while compressing the surrounding “filler” text. In my own production workloads, this approach has consistently delivered a roughly 70% reduction in tokens, taking a 16K-token payload down to about 4.8K without losing the structural shape of the data. This ratio is often the literal difference between a system that is economically viable at one cent per turn and one that is dead on arrival at three cents per turn. We use a selection function for these landmarks that considers named-entity boosting and TF-IDF weighting, ensuring that even in a compressed state, the “load-bearing” parts of the context remain intact for the model to attend to.
Placement matters immensely in the ‘Inject’ stage, yet many teams just append context to the end of a message. What are the specific strategies for interleaving context to ensure the model actually ‘sees’ the most important data?
The “lost in the middle” phenomenon is a very real architectural hurdle, which is why we treat injection as a deliberate placement exercise rather than a simple concatenation task. We use token-level fusion to interleave context tokens with query tokens at specific fusion points defined in the prompt template, ensuring that load-bearing information sits at the start or end of the prompt where the model’s attention is strongest. For example, a Logistics Analyst at an “Initial Assessment” step gets a completely different injection template than a Supply Chain Manager at “Resolution Planning,” even if the underlying context tree is the same. We also use a Function Identifier to inject context directly into the function-calling schemas, so the agent’s available tools are dynamically parameterized by the current situation. This prevents the agent from being overwhelmed by a static, massive catalog of tools and keeps it focused on the actions that are actually relevant to the verified context it has just received.
Finally, how do you close the loop between the runtime pipeline and the architectural evolution of the system to ensure it doesn’t just run, but actually improves over time?
The system is only “real” if it is observable, which is why the Evolution Layer is dedicated to capturing and feeding signals back into the upstream stages. We instrument three primary signals from the start: relevance feedback from the agent or human-in-the-loop, drift detection on the retrieval distribution, and compression fidelity checks. If we see relevance scores dropping, that signal is fed back to reweight the fusion alpha in the Gather stage; if we detect a shift in the documents being returned, we know it’s time to investigate upstream data changes or query-class drift. We even run periodic tests of the same query with and without compression to ensure our landmark selection isn’t causing regressions in the agent’s decision-making. By engineering this feedback ingestion directly into the architecture, we ensure the pipeline is a living system that adapts to the shifting realities of production data rather than a static piece of code that slowly degrades in accuracy.
What is your forecast for agentic systems?
I believe the next eighteen months will see a massive shift in focus away from the “model wars” and toward the “architecture wars,” where the winners will be the teams that treat context as a managed, industrial-grade pipeline. We are quickly approaching a ceiling where model size alone cannot solve the reliability issues inherent in complex, multi-turn reasoning for regulated industries. My forecast is that we will see the rise of highly specialized, deterministic context-engineering tools—like the ARC and MAM tools we use in MoJoCo—that act as the essential pre-processors for LLMs. The competitive moat for any company building agentic systems won’t be their choice of model, but their ability to verify, compress, and strategically inject context at scale. Eventually, the pipeline itself will be seen as the “product,” while the underlying large language models will be treated as interchangeable commodity components that provide the reasoning “engine” for a much larger, more sophisticated machine.
