In the high-stakes world of enterprise data architecture, few things are as frustrating as a “blocked” release. Vijay Raina, an expert in enterprise SaaS technology and software design, has spent years navigating the complexities of cloud-based data integration. His work often centers on Azure Data Factory (ADF), a powerful tool that, while excellent for orchestration, frequently presents a significant bottleneck when teams attempt to scale. The primary challenge lies in its rigid deployment model—an “all or nothing” approach that can halt progress when multiple features are being developed in parallel. In this conversation, we explore how Raina and his team moved beyond the standard limitations of ADF to implement a selective deployment pattern, ensuring that ready-to-go features aren’t held hostage by those still undergoing testing.
We discuss the inherent risks of full-state deployment, the technical architecture required to filter artifacts during the build process, and the critical importance of separating infrastructure concerns from application logic.
How does the standard “full-state” deployment model in Azure Data Factory create friction for teams working on multiple features simultaneously?
The default behavior of Azure Data Factory is to promote the entire factory state rather than individual changes, which essentially forces a “lowest common denominator” approach to releases. Imagine a scenario where your team is working on Feature 100, which is still stuck in User Acceptance Testing (UAT) awaiting business approval, while Feature 200 is fully validated and ready for production. In the standard model, you simply cannot move Feature 200 without also dragging Feature 100 into the live environment, leading to a high-stress choice between delaying a critical release or risking the promotion of unapproved code. This creates a recurring pattern of accidental promotion of incomplete work and a massive increase in coordination overhead between parallel teams. We found that this wasn’t just an edge case; it became a systemic risk that made our production releases far more dangerous than they needed to be.
To solve this, you introduced a selective packaging layer. Could you walk us through how this architecture shifts the deployment from “state-based” to “intent-based”?
The shift required us to stop relying on the native ADF “Publish” button and instead insert a custom logic layer between the build and deployment phases. This architecture utilizes a workflow driven by a manifest file that explicitly defines the scope of what needs to be moved—essentially telling the system exactly which pipelines are ready for prime time. By introducing a staging folder, we can filter the ADF repository so that the ARM export only sees a subset of artifacts, such as Feature 200, while ignoring the work-in-progress on Feature 100. This process involves a sequence of validation, selective packaging via a custom script, and a sanitized ARM export, ensuring that we are deploying based on our specific intent rather than the entire current state of the development branch. It provides a level of precision that the default npm-based export tools simply don’t offer out of the box.
Dependency management is notoriously difficult in data workflows. How did you ensure that moving a single pipeline wouldn’t break the system because of a missing dataset or linked service?
Dependency awareness is arguably the most critical and difficult part of this entire implementation because missing even a single linked service or dataset will cause the deployment to fail immediately. We addressed this by writing a Python script, select_adf_subset.py, which is designed to programmatically identify and pull in every necessary dependency associated with the pipelines listed in our manifest. Pipelines often reference multiple layers of artifacts, so the script must be thorough enough to trace those connections across the repository’s structure, including credentials and managed virtual networks. We learned the hard way that “incremental” deployment is often misunderstood; it doesn’t act like a simple patch, so your generated templates must be absolutely complete for every artifact you’ve decided to include. If you don’t automate this discovery, you’re essentially playing a manual game of whack-a-mole with deployment errors.
You mentioned a specific CI/CD workflow involving Node.js 20 and OIDC. What does that technical execution look like on the ground?
Our CI/CD pipeline is built on a two-job structure that starts with early validation as a guardrail to ensure the JSON files are valid before we even attempt a build. We use Node.js 20 and the @microsoft/azure-data-factory-utilities package to validate the resources in the context of the target factory, using Azure login via OIDC for secure authentication. Once validated, the adf-build job kicks in, running our Python scripts to stage the subset and then exporting the ARM templates specifically from that filtered staging tree. This results in a “safe” template—like ARMTemplateForFactory.safe.json—which has been stripped of sensitive or infrastructure-heavy resources that shouldn’t be touched by application developers. Finally, the deployment job takes that sanitized artifact and uses an incremental ARM deployment to push the changes to the dev or production resource groups.
Why is the separation of “Infrastructure-owned” and “Application-owned” artifacts so vital to a stable selective deployment strategy?
In a mature enterprise environment, you have to draw a clear line in the sand between what a developer owns and what the platform team manages. Application-owned artifacts, such as pipelines, datasets, and triggers, are the pieces that change frequently as business requirements evolve, whereas infrastructure components like Integration Runtimes and Managed Private Endpoints are shared resources. If you don’t separate these, a simple pipeline update could accidentally overwrite a shared Linked Service or a Virtual Network configuration, causing a massive outage across multiple teams. We use a specific script, strip_arm_resources.py, to remove these infrastructure-dependent components from the generated subset to ensure they remain untouched during the incremental deployment. This separation of concerns allows application teams to move fast without the risk of breaking the underlying plumbing that everyone relies on.
Reflecting on this journey, what was the most significant “lesson learned” for your team regarding the intersection of tooling and process?
The most profound realization was that the problem was never the tooling itself, but rather the granularity of the deployment we were asking the tools to perform. Whether we used the UI’s manual publish button or the npm-based export, we kept getting the same “full factory” result because we hadn’t changed the underlying logic of what we were feeding into the exporter. We also had to rethink our definition of “incremental” deployment, realizing that while it sounds like it should only add new things, it actually reapplies the full configuration for everything included in the template. This added a layer of complexity to our scripts and manifests, but the trade-off was worth it because it gave us total control over our release cycles. Ultimately, we discovered that while selective deployment is technically demanding to set up, it’s the only way to achieve true agility in a large-scale ADF environment.
Do you have any advice for our readers?
My advice for anyone struggling with ADF deployments is to start by clearly defining your ownership boundaries before you write a single line of automation code. Sit down with your platform and application teams to decide who owns the Linked Services and Integration Runtimes versus who owns the Pipelines and Triggers, as this clarity will dictate your entire CI/CD strategy. Once those boundaries are set, don’t be afraid to add a custom scripting layer to your pipeline; the native tools are great for simple projects, but enterprise-scale work demands a manifest-driven approach to ensure safety and precision. Investing the time to build a dependency discovery engine will save you hundreds of hours in failed deployments and late-night production hotfixes down the road. Focus on intent-based releases, and you’ll find that your team’s velocity increases significantly because they are no longer afraid of breaking the entire factory with a single change.
