12 articles tagged Data Engineering.
Link to Fabric has two cost components: the Delta replica held in Dataverse database capacity, and the Fabric capacity that consumes it. This covers both, with measured replica-to-source size ratios from a D365 F&O sandbox, current per-GB and per-SKU rates, and the levers that reduce consumption.
The Delta Lake (Parquet) export option in Azure Synapse Link for Dataverse is deprecated: closed to new customers from October 2026 and removed in December 2027. This covers the exact scope of the deprecation, what remains supported, and how to choose between Synapse Link CSV output and Link to Microsoft Fabric.
A Data Pipeline references a connection, not a string — and that connection has a different ID in every workspace. Here's how the Connection Reference variable type swaps it per environment.
Deterministic hash surrogate keys or IDENTITY columns for your dimensions? A practitioner's pros and cons across load order, idempotency, join cost, and reprocessing.
Variable Libraries plus fabric-cicd's environment parameter give you a versioned, typed way to swap Key Vault names, storage accounts, and other per-environment constants automatically.
Copy pipeline via Virtual Network Data Gateway failing with LakehouseOperationFailed / InternalServerError at the _delta_log commit step? The fix is a single Azure subnet configuration change.
VACUUM keeps Delta tables healthy, but it can also break Change Data Feed replay windows. This walkthrough shows exactly when incremental pipelines fail and how to design a safe Fabric pattern with guardrails and fallback recovery.
Speed up your Fabric pipelines by running notebooks in parallel — without breaking your Spark cluster. Learn how High Concurrency Mode works and how to configure session tags.
When defining identity columns in Databricks tables, two common options for automatic identity value generation are GENERATED BY DEFAULT and GENERATED ALWAYS. This article explains the differences and when to use each.
How to perform an in-place Synapse Spark pool upgrade for Synapse Link for Dataverse. Originally written for the 3.3 to 3.4 move — updated for Spark 3.5 and Delta Lake 3.0, since Spark 3.4 is now retired.
Microsoft Dynamics is a widely used product for implementing ERP and CRM solutions. A comparison of Export to Data Lake and Synapse Link for Dataverse for D365 Finance and Operations — updated with the confirmed retirement dates.
Azure Data Factory can securely access credentials stored in Azure Key Vault. This article describes how to set up key vault secrets for Azure Data Factory linked services.