When you schedule a data graph to refresh more frequently than once per day, you can
experience data drift. Data drift occurs when rapid refresh cycles don’t fully capture or
reconcile updates from the source system, causing temporary inconsistencies in the data graph.
Required Editions
A full refresh automatically runs once every 24 hours and corrects any drift that occurs
during more frequent refresh cycles.
What Causes Data Drift
Data drift can occur when the data graph is refreshed multiple times before a full refresh
completes. Common causes include:
-
Partial Updates: If records in the source system change several times within a day, not
all updates appear in the data graph until the full refresh runs.
-
Deleted or Modified Records: Records that are deleted or changed in the source remain
in the data graph until the next full refresh reconciles them.
-
Complex Relationships:When related data or join keys are updated, mismatched or
duplicate nodes can appear in the graph until the full refresh reprocesses all data.
- Hierarchical Dependencies: Drift can also occur when updates arrive for deeper-level
nodes before their parent nodes exist in the system. In such cases, child records
temporarily appear as unmatched or missing relationships until the parent nodes are
processed during the full refresh.
How Data Drift Is Corrected
A scheduled daily full refresh reprocesses all data from the source and realigns the graph
with the most recent state of your data. This process removes any inconsistencies,
duplicates, or outdated records that appear during high-frequency refresh cycles. As a best
practice, use higher-frequency refreshes only when near-real-time data is required and
temporary inconsistencies are acceptable.
