--- schema_version: "1" id: 70a4456ad0b1dbd17affa145663bda9bf09996e30b4a6d7620544438dd7dd5e7 title: Workflow of Data Processing Engine Definition Runs collection: admin version: current locale: en-us portal_url: https://docs.salesforce.com/admin/current/en-us/ind/264-0-0/dpe_monitor_workflow.md canonical_url: https://help.salesforce.com/s/articleView?id=ind.dpe_monitor_workflow.htm&release=264.0.0&type=5 source_site: help.salesforce.com content_type: documentation content_hash: 55d89ff61871b23a8fa467000926b6dd39d6cde346df61abf58f759a2a016bb0 description: A definition's run consists of different stages. The stages include syncing data sources on the definition's runtime platform, transforming data, and writing back results to target objects. The overall workflow remains the same for definitions on CRM Analytics and Data Cloud. Though there are key differences in the resources used to run definitions of each platform. products: - Cross Cloud release: "264" --- # Workflow of Data Processing Engine Definition Runs A definition's run consists of different stages. The stages include syncing data sources on the definition's runtime platform, transforming data, and writing back results to target objects. The overall workflow remains the same for definitions on CRM Analytics and Data Cloud. Though there are key differences in the resources used to run definitions of each platform. ### Required Editions
View supported editions.
## Overview of a Definition's Run A Batch Job record is created for each run. The batch job consists of details of all the stages of the run and provides a unified summary of the runs after they complete. For each stage, one or more Batch Job Part records are created. The number of batch job parts that are created vary based on the definition's design and the runtime platform. Batch job parts are also known as tasks. The tasks are executed in a sequence and if one task fails, the subsequent tasks are automatically canceled. ## Workflow of a Definition's Run on CRM Analytics (CRMA) Here are the stages of a definition's run on CRMA:
Stage What Happens Tasks Created Resources
Data sync Data from each of the definition's data sources are synced in CRMA. A task is created to sync data from each of the definition's data sources. Each task has the same name as the Data Source node that it syncs data for.

To learn about how to track the data sync jobs in CRM Analytics, see

.
Data transformation

A recipe is created in CRMA to transform the data.

After the recipe completes the data transformation, the data is sent back to Data Processing Engine for write back.

 A task named Execution is created.

To learn about how recipes transform data, see
Data write back

Transformed data is written back to the target objects and CRMA datasets.

  • To write back results to objects, Bulk API 2.0 is used.
  • To write back results to CRMA datasets, CRMA's Dataflow is used.

A task is created to write back data to each of the definition's target writeback objects. Each task has the same name as the Writeback Object node that it writes back data for.

If a definition consists of a Composite Writeback node, a task is created for each of the Writeback Object nodes that are part of the Composite Writeback node and a task is created for the Composite Writeback node. All the tasks have the same name as the Composite Writeback node but the tasks for the individual Writeback Object nodes contain a numerical qualifier.

If the writeback target is a CRM Analytics dataset, a task isn't created for write back.

To learn about the resources that are used to write back results, see:
After a run completes, a task named Analysis is created. This task prepares the analysis of the completed run. The analysis of the run is available on the Analysis tab of the run's batch job. ## Workflow of a Definition's Run on Data 360 Here are the stages of a definition's run on Data 360:
Stage What Happens Tasks Created Resources
Data cleanup Transient (in-memory) data that has remained in Data 360 from the definition's previous runs is deleted in preparation for the current run. A task named Cleanup is created. N/A
Data sync

Data from each of the definition's data sources are synced in Data 360.

  • To sync data from standard and custom objects, Data 360's CRM Salesforce Connector is used.
  • To sync data from CSV files, Data 360's Ingestion API is used.

For Data 360-native data sources such as data model objects (DMOs), data lake objects (DLOs), and calculated insight objects (CIOs), data sync isn't required.
  • To sync data from standard and custom objects, a task is created to sync data from each of the data sources. Each task has the same name as the Data Source node that it syncs data for.
  • To sync data from CSV files, a common task is created for all the data sources and two tasks are created for each CSV data source. The common task named IngestionApiConnector creates a job for the Ingestion API. The second task named Ingestion represents the data sync phase where the Ingestion API validates the CSV file and prepares to upload the file data to Data 360. The third task named after the data source node represents the actual task of uploading the data to Data 360.

To learn about the resources that are used to sync data, see

.
Data transformation A batch data transform is created in Data 360 to transform the data. A task named Execution is created. To learn about batch data transforms in Data 360, see Batch Data Transforms.
Data write back

Transformed data is written back to the target objects.

  • To write back results to standard and custom objects, Data 360's Query API is used to read the transformed data from Data 360 and Bulk API 2.0 is used to write back the results.
  • To write back results to Data 360-native data sources such as DMOs, DLOs, and CIOs, a batch transform is used.

A task is created to write back data to each of the definition's target writeback objects.

  • Tasks that write back to standard and custom objects have the same name as the Writeback Object node that it writes back data for.
  • Tasks that write back to Data 360-native objects have the same name as the target object with _WRITEBACK as the suffix.

If the definition consists of a Composite Writeback node, a task is created for each of the Writeback Object nodes that are part of the Composite Writeback node and a task is created for the Composite Writeback node. All the tasks have the same name as the Composite Writeback node but the tasks for the individual Writeback Object nodes contain a numerical qualifier.

To learn about the resources used to write back results, see:
## Extra Tasks Created for a Run If you run a definition from the builder and use the Save Count of Node Records or Save Node Output Without Writing Back Results debug mode for the run, extra tasks are created. Each task is named after the node for which you wanted to view the output record count. For more information about how the debug mode, see [Debug Mode: Save Count of Node Records](https://help.salesforce.com/s/articleView?id=ind.dpe_run_builder_debug_modes.htm&language=en_US).