All posts

The Problem with Fabric Data Pipeline Schedules and How to Fix It

I just finished implementing Data Pipeline schedules in Microsoft Fabric, and I ran into enough friction that it warrants writing down. This article covers the problem, why the obvious workarounds don't work, and the fix I ended up building into my deployment pipeline.

The root cause

Data Pipeline schedules in Fabric are a first-class citizen that physically deploys with the pipeline. When you set a schedule, Fabric writes a .schedules file into the Data Pipeline’s folder, right next to the JSON pipeline definition. That file is part of the item, so it travels wherever the item travels: into git, through your PRs, and into every workspace you deploy to.

This is inconsistent with Semantic Model refreshes, which do not carry over on deployment. It is also a known issue in the Fabric community.

Why this becomes a problem

The design breaks down under a setup I would label as must-have for serious workloads:

  • Separate workspaces per environment
  • CI/CD promoting items between them
  • Different schedules for dev and prod

To promote items via CI/CD, you store your workspace item definitions in git and connect your workspaces to branches. And then the flow looks like this:

  1. You do some work in your development workspace. When you’re happy with it and had a few successful runs, you commit it to git through the Fabric UI.
  2. You switch to your git provider and create a PR to the production branch.
  3. You open the PR and see diffs in your definitions, but also in the .schedules files, carrying your development schedules.
  4. You get confused (like me), because there is nothing you can do about it, and there is no straightforward way around it.

The workaround that doesn’t work

My initial assumption: just exclude the .schedules files from the commit. If they are omitted from the deployment, the current production state should remain unaffected, right?

It does not work like that. As confirmed by a Microsoft employee in the community thread linked above, it is not possible to isolate or exclude the schedule when deploying a pipeline. Deployments are all or nothing. If the .schedules file is missing, your production schedule is wiped. If it contains your dev schedule, your production schedule is overwritten with it.

So the two “options” you are left with are equally bad:

  • Commit production schedules to dev before making a PR to main, polluting your dev workspace with prod configuration.
  • Let the deployment overwrite the schedules and patch them back afterwards, leaving you with an incorrect intermediate state where prod briefly runs on dev schedules, or no schedules at all.

Neither is acceptable when the whole point of a schedule is that it fires reliably.

The fix: snapshot and restore prod schedules in the deployment pipeline

There are probably multiple ways to solve this, but this is what I added to my deployment pipeline, which runs on every merge to the prod branch:

  1. Clone the repo at the dev branch.
  2. Stage a copy of the git directory where the items live (this can be the root or something like /fabric).
  3. Check out origin/<prod-branch>.
  4. Snapshot the existing prod *.schedules / .schedules files and store them in a temporary location.
  5. Replace the prod git directory with the staged dev tree.
  6. Strip all schedule files that came from dev.
  7. Restore the prod schedule snapshot, but only where the parent item still exists.
  8. Commit and push to <prod-branch>.

Visualized process

The effect: dev definitions flow to prod as intended, dev schedules never leave dev, and prod schedules survive every deployment untouched. Deleted pipelines don’t leave orphaned schedule files behind, because the restore step checks that the parent item still exists.

The first time you set a prod schedule, you set it once in the prod workspace and commit it. From that point on, the pipeline preserves it on every promotion.

Closing thoughts

Schedules are environment configuration, not item definition. Fabric treats them as the latter, and until Microsoft changes that, anyone running a multi-workspace CI/CD setup has to compensate for it in their deployment tooling. The snapshot-and-restore approach above is simple, reversible, and keeps both environments in the state you actually intended. If Microsoft ever ships a native way to exclude schedules from deployment, you can delete these steps and nothing else changes.