In this discussion, we sit down with a veteran of the software and chip design world to explore the intricate dance between high-performance hardware and the rapidly evolving landscape of generative AI. With a background spanning from early computer engineering at the University of Pittsburgh to leading generative AI software at one of the world’s most influential technology companies, our expert provides a unique “full-stack” perspective. We delve into how the necessity of running difficult workloads—from computational fluid dynamics to large language models—has driven a unique era of hardware-software co-design, resulting in specialized engines and open-source model families like Nemotron.
The conversation covers the technical shifts in model precision, the transition from traditional transformers to hybrid architectures, and the move toward complex agentic systems that mimic object-oriented programming. Our expert also sheds light on the strategic importance of “true” open source, explaining why sharing training data and “gym environments” is crucial for enterprise trust and domain-specific specialization in fields like chip design and cybersecurity.
The “extreme co-design” process involves a daily feedback loop between model builders and hardware architects. How does this collaboration specifically influence the “plan of record” for next-generation silicon, and what metrics determine if a specific software workload justifies a new hardware engine or SKU?
Extreme co-design is a rapid-fire, engineer-to-engineer daily feedback loop that ensures hardware planning isn’t happening in a vacuum. During our Plan of Record (POR) process, we identify recurring bottlenecks in model training or inference; for instance, if we see a specific memory bottleneck consistently slowing down large-scale deployments, it informs the architectural requirements for the next generation of silicon. A prime example of this is our recently announced context memory engine, which was born directly from the need to handle massive sequences more efficiently. We justify a new engine or SKU when a workload becomes an industry standard or poses a significant hurdle that general-purpose compute can’t solve optimally, like the transition from early high-performance computing to the specific demands of deep learning and LLMs. It’s about ensuring that by the time the hardware is manufactured, it is already perfectly tuned for the software libraries and model architectures that have emerged in the interim.
Training natively in reduced precisions like FP8 or NVFP4 can reduce memory requirements by half compared to FP16. What are the accuracy trade-offs when training in low precision versus using post-training quantization, and how does this affect the scalability of multi-node inference?
The historical approach has been to train in high precision and then quantize down, but that often leads to a loss of about 1% to 2% in accuracy, which then requires post-training to recover. By training natively in reduced precisions like FP8 for Hopper or NVFP4 for Blackwell, we find that the model retains its entire accuracy target from the start while immediately benefiting from a 50% reduction in memory footprint. This memory efficiency is a game-changer for scalability, especially when moving from a single node of 8 GPUs to multi-node configurations of 16 or more. Lowering the precision allows us to fit larger, more robust models into smaller form factors or distribute them across nodes with much lower latency and higher compute efficiency. It essentially allows the hardware to breathe, providing more room for the KV cache and allowing for higher throughput during the decoding phase of inference.
The Nemotron family utilizes a hybrid architecture combining Mamba State Space models with traditional Transformers. Why is this combination more token-efficient for long-context recall, and how does it solve the quadratic scaling problems typically associated with dense transformer architectures during inference?
Standard dense transformers face a quadratic scaling challenge where every token must attend to every other token, which causes inference time and memory usage to explode as sequences grow. By integrating Mamba State Space models, which are inherently sequence-to-sequence models, we introduce a more linear scaling factor for context recall. This hybrid approach allows us to replace some of the traditional transformer heads with these highly efficient state-space modules, significantly reducing the computational overhead. In our Nemotron releases, we’ve found that this combination provides a “world model” view that is far more token-efficient during both training and inference. It allows the model to maintain high accuracy without the massive “attention tax” typically paid by pure transformer architectures when dealing with very long sequences.
Modern AI systems are shifting from simple Retrieval Augmented Generation toward complex agentic systems. How does treating agents like “object-oriented programming” change the way developers manage shared memory, and what hardware optimizations are necessary to handle the high token demand of these reasoning models?
We often joke that building AI agents feels like a speed-run through the history of networked software, landing us squarely in a new form of object-oriented programming. In this paradigm, an agent is like an autonomous object that you spin off to perform a task; it thinks, acts, and returns an answer, but it needs to manage its own state and “memory” of previous interactions. This shift requires sophisticated memory hierarchies that can traverse both hardware and software, moving parts of the context to disk and recalling them only when necessary. To support the high token demand of these reasoning models, we’ve developed tools like Dynamo for disaggregated serving, which allows us to split the “prefill” and “decode” stages across different GPU SKUs. This architectural flexibility ensures that the system can handle a million-token context—roughly a million words—without the agent “losing its mind” or stalling the entire pipeline.
Releasing open weights, training data, and reinforcement learning “gym environments” provides a complete recipe for model development. How does access to raw training data help enterprises mitigate liability, and what steps are involved in using these open recipes to build specialized models for chip design?
Many enterprises feel stuck between a rock and a hard place because they can’t audit the “black box” of third-party APIs, which creates significant liability concerns regarding data provenance and bias. By releasing the full recipe—the weights, the architecture, and the raw training data—we allow companies to interrogate the data and gain the confidence needed to govern their own AI deployments. For specialized tasks like chip design, an enterprise can take our trusted base model and use our open “reinforcement learning gym environments” to create their own verifiers. For example, in code generation, a gym can verify if the output compiles or passes a unit test; in chip design, partners can build similar automated environments to verify technical accuracy. This creates a bootstrap effect, where companies like ServiceNow can take our foundations and build domain-specific models, like their Apriel model, with full transparency and control.
Memory hierarchies must traverse both hardware and software to maintain context in million-token sequences. When an agent needs to store or recall vast amounts of data, what are the best practices for disaggregated serving, and how do tools like Dynamo optimize communication between GPUs?
Best practices for disaggregated serving involve separating the heavy lifting of the initial data “prefill” from the iterative “decoding” process, which allows for much higher GPU utilization. Our Dynamo framework manages this by distributing the workload across the network, while NIXL handles the high-speed communication between the GPUs to ensure there is no bottleneck as the model context grows. When an agent is dealing with a million tokens, it’s not just about raw capacity; it’s about how you move that data through the memory hierarchy without losing the “needle in the haystack.” We utilize specialized context memory engines in our hardware to handle these massive sequences, ensuring that the software can recall specific data points from the million-word context with minimal latency. This holistic approach ensures that the “context rot” often seen in poorly optimized systems is mitigated by tight integration between the storage layer and the compute engine.
What is your forecast for the evolution of open-source model architectures and their integration with specialized hardware?
I believe we are moving toward a future where AI models are treated exactly like standard software libraries, with regular update cycles, bug fixes, and feature requests. We will see a massive proliferation of “system-of-models” architectures where general-purpose GPUs remain the gold standard because they can flexibly run the wide variety of models—speech, vision, and text—required for a single agentic task. My forecast is that “true” open source, which includes the training data and environments, will become the industry requirement for enterprise trust, allowing for a worldwide R&D engine where developers can push pull requests directly to model architectures. As we move toward GTC and beyond, the integration will become so seamless that the distinction between the “chip” and the “model” will blur, creating a unified software development platform that is refreshed and optimized as frequently as any modern SaaS application.
