Dynamics 365 Data Refresh Alert: Failure and Skipped Dataflow Analysis

Table of Contents

This article addresses a specific scenario encountered in Microsoft Dynamics 365 Customer Insights - Data where a Power Query-based data source fails to refresh as expected during a scheduled system refresh cycle. It outlines the symptoms users observe, explains the underlying reason for this behavior, and provides a comprehensive resolution for ensuring timely and accurate data updates from Power Query sources. Understanding the distinction between different refresh mechanisms within the platform is key to resolving this issue.

Symptoms¶

Users observing this issue will typically find that their data source processing status within Dynamics 365 Customer Insights - Data indicates a state of Completed with errors. While the overall system refresh process might succeed in its other tasks, the specific Power Query data source appears not to have pulled the latest data or executed its transformation logic. This results in downstream processes, such as data unification or segment creation, potentially operating on stale or outdated information from that particular source.

The primary indicator of this specific problem is the accompanying error message received upon reviewing the data source refresh history or system notifications. The error message clearly articulates that although the system-triggered refresh process completed successfully from a system perspective, the actual dataflow refresh associated with the Power Query source was deliberately bypassed. This skipping occurs because the configuration of the system refresh is not applicable to the mechanism used by Power Query data sources. The message explicitly guides the user towards configuring scheduled refresh directly on the data source itself if periodic updates are desired, highlighting the need for a different approach than relying solely on the system refresh.

Cause¶

The behavior described stems from the way Dynamics 365 Customer Insights - Data manages data refresh operations for different types of data sources. The platform employs two primary refresh methods: the system refresh and the data source refresh. The system refresh is a high-level process, typically configured by a system administrator, that orchestrates the refresh of standard data sources and triggers subsequent processes like data unification, segment calculation, and measure computation. Its primary function is to ensure that the core customer profile and related insights are updated based on the available data.

In contrast, the data source refresh mechanism is managed at the individual data source level. For data sources created using Power Query (which involves defining specific data extraction, transformation, and loading logic), the refresh process is inherently tied to the execution of that Power Query script. The system refresh, by design, does not execute the Power Query logic for these specific data sources. Instead, when the system refresh runs, it primarily checks the state of data sources that are not managed by Power Query’s independent refresh schedule or simply copies the data state as it was after the last successful data source refresh for Power Query sources.

Therefore, the error message “System triggered refresh succeeded; however, dataflow refresh was skipped because the System refresh settings are not applicable for this datasource” precisely reflects this architectural distinction. It means the system refresh did its job but recognized that the Power Query data source’s refresh cycle is controlled independently. If the Power Query data source has not been configured with its own scheduled refresh, or if its last manual refresh was long ago, the system refresh will not force it to update. Consequently, the system process will work with potentially outdated data from that source, leading to the symptoms observed.

Resolution¶

To resolve the issue where your Power Query data source isn’t being refreshed automatically by the system refresh, you must configure a dedicated refresh schedule specifically for that data source. This ensures that the underlying Power Query logic is executed periodically, pulling the latest data and making it available for subsequent processing steps within Dynamics 365 Customer Insights - Data. The process involves navigating to the data source settings and defining the desired refresh frequency and time.

Here is a step-by-step guide to setting up the scheduled refresh for your Power Query data source:

1. Navigate to Data Sources: In Dynamics 365 Customer Insights - Data, access the Data area from the left-hand navigation menu. Select Data sources. This page lists all the data sources currently configured in your environment.
2. Identify the Power Query Source: Locate the specific data source that is experiencing the refresh issue. Power Query data sources are typically those where you defined transformation steps using the Power Query editor during the ingestion process.
3. Edit Data Source: Click on the data source you wish to configure. This will open the details pane for that source. Look for an option to Edit or manage the settings for the data source. This might be represented by a pencil icon or an “Edit” button.
4. Access Refresh Settings: Within the data source editing interface, find the section related to Refresh settings. This section controls how and when the data from this specific source is updated within Customer Insights.
5. Configure Scheduled Refresh: The refresh settings will likely present options for Manual refresh or Scheduled refresh. Select the Scheduled refresh option. This enables the scheduling controls.
6. Define Frequency and Time: Specify how often you want the data source to refresh. Common options include Daily, Weekly, or Monthly. Based on the chosen frequency, you will also need to set a specific time of day when the refresh should occur. Consider your data source’s update cycle and the timing of your system refresh when choosing the schedule to ensure data is fresh before downstream processes run.
7. Select Time Zone: Choose the appropriate time zone for your scheduled refresh. This ensures the refresh runs at the intended local time.
8. Save Settings: Once you have configured the schedule, save the changes to the data source settings. The platform will now initiate refreshes for this data source according to the schedule you’ve defined, independently of the system refresh trigger.

By configuring the scheduled refresh directly on the Power Query data source, you are instructing Dynamics 365 Customer Insights - Data to execute the dataflow logic on a recurring basis. This resolves the issue where the system refresh skips the Power Query execution, ensuring that your customer data ingested through Power Query is kept up-to-date and available for subsequent processing steps and analysis within the platform.

Troubleshooting and Considerations¶

Even after setting up a scheduled refresh for your Power Query data source, you might encounter other issues preventing a successful refresh. It is important to monitor the refresh history and troubleshoot potential problems.

• Credentials: Ensure the credentials used to connect to the underlying data source from Power Query are up-to-date and valid. Expired passwords or changes in permissions can cause the refresh to fail. Check the connection settings within the data source configuration.
• Source System Availability: Verify that the source system (database, API, file share, etc.) is accessible from Dynamics 365 Customer Insights - Data at the scheduled refresh time. Network issues, firewall rules, or downtime on the source system can interrupt the data flow.
• Power Query Errors: Complex Power Query transformations can sometimes fail due to data inconsistencies, changes in the source schema, or errors in the M code. Review the error details provided in the refresh history. You may need to edit the data source, open the Power Query editor, and troubleshoot the specific query steps. Look for data type errors, column name mismatches, or issues with filtering or merging steps.
• Data Volume and Performance: Very large datasets or inefficient Power Query transformations can cause refreshes to time out or fail due to resource limitations. Consider optimizing your Power Query script to pull only necessary data, reduce transformation complexity, or implement incremental refreshes if supported by the source.
• Dependencies: If your Power Query data source depends on other systems or processes to prepare the data, ensure those upstream dependencies are completed before the scheduled refresh time in Customer Insights. Scheduling conflicts can result in the Power Query process pulling incomplete or incorrect data.

Addressing these potential issues in conjunction with setting up the scheduled refresh will significantly improve the reliability of your Power Query data ingestion pipeline. Regular monitoring of refresh status and history is a critical practice for maintaining data freshness and integrity.

Best Practices for Data Refresh Management¶

Effective data refresh management is crucial for the accuracy and timeliness of insights generated by Dynamics 365 Customer Insights - Data. Implementing best practices can prevent issues like the one described and ensure smooth data operations.

Firstly, carefully plan your refresh schedules. Align the refresh frequency of your data sources, especially Power Query sources, with the update frequency of your source systems. If your source data changes daily, schedule a daily refresh. If it updates hourly, explore options for more frequent refreshes if supported and necessary. Coordinate the data source refresh schedules to complete before the system refresh is scheduled to run, ensuring the system process operates on the freshest possible data from all sources.

Secondly, utilize the monitoring tools available in Dynamics 365 Customer Insights - Data. Regularly check the refresh history for each data source and the overall system refresh status. Set up notifications or alerts for failed refreshes to be proactively informed of issues. This allows for timely intervention and minimizes the impact of data staleness on your customer insights.

Thirdly, optimize your Power Query transformations. Complex or inefficient queries can put a strain on resources and increase refresh times, potentially leading to failures. Aim to perform filtering and column selection early in the query steps to reduce the data volume being processed. Test your queries with representative data volumes to ensure they complete within acceptable timeframes. Consider breaking down complex data ingestion into multiple, simpler data sources if necessary.

Finally, maintain clear documentation of your data sources, their refresh schedules, dependencies, and any custom Power Query logic. This documentation is invaluable for troubleshooting, onboarding new team members, and ensuring continuity in data management practices. Understanding the flow of data from source systems through Customer Insights is fundamental to effective platform administration and insightful customer analysis.

Data Refresh Flow Visualization¶

To better illustrate the distinct paths of data refresh within Dynamics 365 Customer Insights - Data, particularly concerning Power Query sources, consider the following simplified flow diagram. This diagram highlights how the system refresh interacts differently with standard sources versus those managed by Power Query’s independent schedule.

mermaid graph TD A[External Data Source] --> B{Data Source Type?}; B -->|Standard Source| C[Standard Data Source Ingestion]; B -->|Power Query Source| D[Power Query Editor/Logic]; C --> E(Customer Insights Layer); D --> F(Power Query Data Source); F --> G(Customer Insights Layer); E --> H[System Refresh]; G --> I(System Refresh); H --> J[Unification, Segments, Measures]; I --> K[Unification, Segments, Measures]; G --Skips Execution--> I; F --Requires Independent Scheduled Refresh--> F;
Diagram: Simplified Data Refresh Paths

In this diagram, standard data sources (C) are typically refreshed as part of the System Refresh (H). However, Power Query Data Sources (F), after being configured through the Power Query Editor (D), must have their own scheduled refresh (arrow back to F) to update the data available in the Customer Insights Layer (G). The System Refresh (I) process then uses the data already available in G but does not re-execute the Power Query logic in F. Both paths converge, feeding data into the downstream insights processes (J, K). The key takeaway is that the link “G –Skips Execution→ I” for Power Query sources necessitates configuring the “F –Requires Independent Scheduled Refresh→ F” loop.

This visual representation helps reinforce why relying solely on the system refresh is insufficient for updating Power Query-based data sources and underscores the need for explicit scheduling on the source itself.

Implementing these practices ensures a robust and reliable data foundation for your customer insights, enabling better decision-making and more effective customer engagement strategies. Addressing the specific error by configuring the Power Query data source’s schedule is the first step towards achieving this reliability.

---

Do you have further questions about managing data refreshes in Dynamics 365 Customer Insights - Data or encounter other issues? Share your experiences and queries in the comments section below. Let’s discuss troubleshooting tips and best practices!