Author Fabric data pipelines in Python

In short

• A Fabric data pipeline is a JSON document. The portal and a hand edit of that JSON are the two ways to change it, and both get worse as dependencies grow.
• fabric-data-pipelines authors the same definition in typed Python: a loop for twelve tables, a refactor for a rename, a module for a query you want to share.
• The library checks the graph before export, then writes raw JSON or the *.DataPipeline/ folder Fabric syncs from Git.
• It produces definitions. Terraform, fabric-cli, and fabric-cicd still deploy them. An existing pipeline can be imported and round-tripped.

I have been using Microsoft Fabric for the past two years. In the beginning I was disappointed. The gap was infrastructure as code, which I wrote about in Microsoft Fabric: the good, the bad and the ugly. Shortly after that piece, the official Terraform provider was released. One year later the platform had become something workable. It is gradually getting better. It is still not where I need it to be.

One of the remaining gaps is data pipelines. A Fabric data pipeline is an Azure Data Factory pipeline that lives inside Fabric and integrates with the other items in a workspace. There are exactly two ways to build and change one: the GUI, which is ClickOps, or editing the underlying JSON, by hand or through an MCP. Both get worse as complexity and dependencies grow.

I would rather use code to create and structure these pipelines. So I built fabric-data-pipelines, a Python library that lets you author Fabric data pipelines as code.

01 / The object

What breaks once the pipeline grows?

A Fabric data pipeline is a JSON document. Activities are objects in a list. Execution order lives in nested dependsOn arrays. Every connector detail sits somewhere inside that tree.

The GUI hides this, and that is fine until the pipeline is real work:

1. The same landing pattern, twelve times. You need it for twelve tables, so you copy the activities twelve times. The one field that should have changed is the one that did not.
2. A rename breaks a dependency. dependsOn names another activity as a string. Rename the truncate step and forget the copy that waits for it, and the graph is wrong in a way the diff does not shout about.
3. The script is one line of JSON. A SQL change lives inside a string with no newlines. Review means copying it out, formatting it, and only then seeing what changed.
4. Half the definition is environment-specific. You promote from dev to prod and discover connection ids, workspace ids, and queries baked into the same tree.
5. There is no module boundary. A SQL query, a Teams message template, a shared policy: each one is inline. Nothing can be imported, because the only unit is the serialization tree.

02 / The fix

Write the pipeline in Python

Making Python the authoring interface is what removes those limits. Twelve landing tables become a loop or a factory function. Renames are refactors the IDE handles. Reviews diff readable Python, and the scripts inside it, instead of generated JSON. Environment-specific values become parameters. Shared queries and templates become imports, because there is finally a module boundary.

Install the current release. It needs Python 3.10 or newer.

pip install fabric-data-pipelines
# or
uv add fabric-data-pipelines

A pipeline that waits, then runs a notebook, is the whole pattern:

from fabric_data_pipelines import Notebook, Pipeline, Wait

wait = Wait(name="Wait_For_Upstream", wait_time_in_seconds=30)

transform = Notebook(
    name="Transform_Silver_Sales",
    notebook_id="00000000-0000-0000-0000-000000000002",
    workspace_id="00000000-0000-0000-0000-000000000001",
)
wait.then(transform)

pipeline = Pipeline(
    name="Daily_Silver_Sales_Transform",
    activities=[wait, transform],
)
pipeline.save("daily_silver_sales_transform.json")

That emits a valid Fabric pipeline definition. The same four steps cover everything after it:

1. Create typed activities. Data movement (Copy, Lookup), control flow (ForEach, IfCondition, Fail), or execution (Notebook, Script). The activity reference is the full list. An activity that is not modeled yet goes in as a RawActivity, so you are never blocked.
2. Declare execution order in code. More on that below.
3. Build a Pipeline. This is where defaults live.
4. Export JSON, or a Fabric item folder.

03 / Order

Dependencies that read like a workflow

Orchestration should be readable. This is the part that made me build the library: I wanted Airflow-style DAG code for Fabric data pipelines.

lookup.then(copy).then(notebook)

# linear flows
lookup >> copy >> notebook

.then() defaults to a Succeeded dependency. The other Fabric conditions are there when you need them:

copy.then(notify, on="Failed")
copy.then(cleanup, on="Completed")

Fan-in is the same idea, from the activity that waits:

join.after(copy_a, copy_b)

The dependency reference is the short version of those helpers.

04 / The check

Validation before a deploy

Python as the interface means the graph is checked before it is serialized. Duplicate activity names, unknown dependency targets, cycles, and cross-scope dependencies all fail in the terminal, or in CI, instead of during a Fabric deployment.

Control-flow activities such as IfCondition, ForEach, Switch, and Until have their own nested scopes. Wiring a dependency across those boundaries is a classic mistake. Fabric refuses it. The library refuses it first.

05 / Metadata

Twelve tables, one function

The claim above was that twelve landing tables become a loop. This is what that looks like:

from fabric_data_pipelines import (
    AzureSqlMITable, Copy, ExternalReferences, Pipeline,
    Script, ScriptBlock, SqlMISink, SqlMISource, expr,
)

LANDING = expr.library_variable("Demo_ETL_Library_Landing")
SOURCE = expr.library_variable("Demo_ETL_Library_SourceDb")

def land_table(schema: str, table: str) -> tuple[Script, Copy]:
    truncate = Script(
        name=f"Truncate_Landing_{table}",
        database="Landing",
        scripts=[ScriptBlock(
            text={"value": f"TRUNCATE TABLE {schema}.{table}", "type": "Expression"},
            type="Query",
        )],
        external_references=ExternalReferences(connection=LANDING),
    )
    copy = Copy(
        name=f"Copy_{table}_to_Landing",
        source=SqlMISource(
            sql_reader_query=f"SELECT * FROM [{schema}].[{table}];",
            dataset_settings=AzureSqlMITable(database="SourceDb", connection=SOURCE),
        ),
        sink=SqlMISink(
            write_behavior="insert",
            dataset_settings=AzureSqlMITable(
                database="Landing",
                schema_name=schema,
                table=table,
                connection=LANDING,
            ),
        ),
    )
    truncate.then(copy)
    return truncate, copy

TABLES = [
    "customers", "orders", "order_lines", "products",
    "invoices", "payments", "shipments", "returns",
    "suppliers", "inventory", "promotions", "stores",
]

pipeline = Pipeline(
    name="Landing_Daily_Load",
    activities=[
        step
        for table in TABLES
        for step in land_table("sales", table)
    ],
)
pipeline.save("landing_daily_load.json")

land_table returns the truncate and the copy, so the pipeline list flattens each pair. One function, twelve tables, and a readable diff when the pattern changes.

The table list is ordinary Python data, so it does not have to be hardcoded. Load it from a YAML file or a metadata table in the warehouse, and the same thirty lines are a metadata-driven ingestion framework.

The same pattern exists in the portal, and it behaves differently.

	In a data pipeline	In Python
Where the loop runs	At runtime. A Lookup reads the control table, a ForEach walks it, and expressions fill in the blanks.	Before deployment. The activities are already concrete when Fabric sees them.
How you test it	Deploy, then read the monitoring view.	Unit-test the code, and diff the definition it emits.
What Fabric receives	A template that still has to be interpreted.	A definition with nothing left to guess.

Runtime iteration is still there when you want it. Fabric's ForEach is fully supported. The parameterized ELT controller is the example: one parent pipeline, a table list in a parameter, a dataflow refresh per item. Watermark lookups and lock-guarded orchestration are in the same ETL patterns guide.

06 / Export

Two ways to export

There are two export modes, so the output can match the deployment you already have.

Use the raw definition when you want the JSON itself, for inspection, tests, or an API call:

json_text = pipeline.to_json()
pipeline.save("daily_load.json")

Use a Fabric Git item folder when the repository is the source of truth:

pipeline.save_item("out")

That writes the shape Fabric's Git integration expects:

Gold_Finance_Metrics_Refresh.DataPipeline/
  pipeline-content.json
  .platform
  .schedules

A whole workspace is the same call, over a list:

from fabric_data_pipelines import save_workspace

save_workspace([pipeline_a, pipeline_b], "workspace")

Schedules are part of the model. When a pipeline has them, .schedules is written with the item.

One detail is worth calling out. Fabric tracks item identity through logicalId in .platform. Set logical_id on the Pipeline and a rename keeps the same Fabric item. Omit it and the library derives one from the name. If a .platform file already exists, the existing logicalId is preserved. That is the kind of thing you only learn by breaking it once. The export notes cover both modes.

07 / The boundary

Where this library stops

This is an authoring and export library. It writes definitions. Terraform, fabric-cli, and fabric-cicd still deploy and promote them. Your existing deployment pipeline moves the files this library produces. The deploy notes walk through Git sync and fabric-cicd.

Tool	Job
fabric-data-pipelines	Author the pipeline and export the definition
Fabric portal	Learn an activity, debug a run
fabric-cicd	Promote items across environments
Terraform	Capacity, workspaces, the platform around the items
fabric-cli	Operate the workspace from the command line

Generate the *.DataPipeline/ folder here. Let the tool you already trust publish it.

08 / Round trip

Bring a pipeline you already have

Most people who want this library are not starting from an empty file. The library imports an existing pipeline JSON and turns it into Python.

The import is round-trip safe. Load a definition, export it again, and you get equivalent JSON back. Nothing is dropped or reinterpreted on the way through, so a production pipeline can be imported without a silent change behind it.

fabric-data-pipelines codegen path/to/Item.DataPipeline -o pipeline.py

The generated code is correct, and it is not pretty. Treat it as a working starting point, clean it up once, and from then on the Python is the source of truth. The migration guide is the full walkthrough, including a standalone pipeline-content.json, which needs --name because the content file does not store the pipeline's name.

If you are running Fabric pipelines at any scale, I want to know which activities and patterns you need next. Issues are welcome, and so is a straight opinion on whether this solves the problem.

09 / Questions

Questions people ask

Does this deploy the pipeline?

Deployment stays with the tools you already use. This library writes the definition. Generate the folder here, then promote it with Git sync, fabric-cicd, Terraform, or fabric-cli. The deploy notes are that split.

Do I have to stop using the Fabric portal?

Use the portal to learn an activity and to inspect a run. Use Python when the pipeline is product code: more than one author, a dependency that must hold, a pattern you will copy. The comparison is the short version of that split.

Can I migrate a pipeline that already exists?

Yes. fabric-data-pipelines codegen reads an item folder or a pipeline-content.json and writes a Python module. Load that module, export again, and the JSON is equivalent. Details are in Migrate from Fabric.

What if my activity is not in the library yet?

Put it in a RawActivity with the original type and properties. The pipeline still exports. You tighten the parts you change often and leave the rest until the library models them.

When should the loop stay inside Fabric?

When the iteration has to happen at runtime: a control table that changes between runs, a parameter the caller fills in. ForEach covers that. The parameterized ELT controller is the pattern. When the table list is known before deployment, generate the activities in Python and let Fabric run a concrete graph.

Where does the code live?

The docs are at fabric-data-pipelines.datalyft.io. The package is fabric-data-pipelines on PyPI. The source is datalyft/fabric-data-pipelines.

10 / References

References

1. Microsoft Fabric: the good, the bad and the ugly, Bastiaan, Medium, 20 February 2025 medium.com
2. Microsoft Fabric: one year later, Bastiaan, Medium, 21 May 2026 medium.com
3. fabric-data-pipelines, overview fabric-data-pipelines.datalyft.io
4. Getting started fabric-data-pipelines.datalyft.io
5. Activity reference fabric-data-pipelines.datalyft.io
6. Dependencies fabric-data-pipelines.datalyft.io
7. ETL patterns fabric-data-pipelines.datalyft.io
8. Exporting fabric-data-pipelines.datalyft.io
9. Migrate from Fabric fabric-data-pipelines.datalyft.io
10. Deploy to Fabric fabric-data-pipelines.datalyft.io
11. fabric-data-pipelines on PyPI pypi.org
12. datalyft/fabric-data-pipelines github.com