---
schema_version: "1"
id: b68c5244456cc95a94e140286c9e4cb5a124a7252307a5868db0b25db51f2518
title: Data Stream Settings and Refresh Modes
collection: admin
version: current
locale: en-us
portal_url: https://docs.salesforce.com/admin/current/en-us/data/264-0-0/c360_a_data_stream_edit_settings.md
canonical_url: https://help.salesforce.com/s/articleView?id=data.c360_a_data_stream_edit_settings.htm&release=264.0.0&type=5
source_site: help.salesforce.com
content_type: documentation
content_hash: c3ebce18443366602a7e79e333c2cc921c7cce13dc895e0d46fee8f40a51c84f
description: With data streams, you can schedule or run jobs to ingest data from a source system. These jobs pull data from the source system into the connected data stream. The method and frequency options for data ingestion depends on the type of connector. Use the data stream settings to create or change the execution parameters of a data stream.
products:
  - Data 360
release: "264"
---
# Data Stream Settings and Refresh Modes

With data streams, you can schedule or run jobs to ingest data from a source system. These jobs pull data from the source system into the connected data stream. The method and frequency options for data ingestion depends on the type of connector. Use the data stream settings to create or change the execution parameters of a data stream.

### Required Editions

<table class="slds-table slds-table_bordered slds-max-medium-table_stacked slds-m-bottom_small edition"><colgroup><col></colgroup><tbody><tr class=""><td class="slds-cell-wrap" style="vertical-align:top;">Available in: Lightning Edition</td></tr></tbody></table>

## Refresh Modes

The Refresh Mode settings determine how data is ingested into a data lake object (DLO). Efficient use of these modes ensures data freshness and cost reduction by improving data processing efficiency.

<table class="slds-table slds-table_bordered slds-max-medium-table_stacked slds-m-bottom_small ">
        <colgroup>
          <col style="width:50%">
          <col style="width:50%">
        </colgroup>
        <thead class="slds-text-title_caps ">
          <tr class="">
            <th scope="col" class="slds-cell-wrap" style="vertical-align:top;" id="d106226e60">Setting</th>
            <th scope="col" class="slds-cell-wrap" style="vertical-align:top;" id="d106226e63">Description</th>
          </tr>
        </thead>
        <tbody>
          <tr class="">
            <td data-label="Setting" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e60 ">Incremental</td>

            <td data-label="Description" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e63 ">
              <p>This mode is available for most non-file-based data streams that have a
                Datetime field. Select the field that indicates when the incoming record was last
                modified. For each refresh, the data stream is updated with data added or changed
                since the last refresh.</p>
              <p>To make sure that there’s no data loss, the last few
                records from the previous run are reprocessed during each job run.</p>
              <p>To remove
                records from an incrementally updated DLO, mark deleted records in your source
                system with a Delete Record Flag. For example, <code>record_delete_flag =
                    true</code>. The DLO will then remove these records during the next update.
                You can permanently delete them from your source system after a successful update
                run.</p>
              <p style="margin-bottom:0;">Incremental refresh is also supported in Zero Copy Data Federation, but
                deletion of records during incremental refresh operations isn't supported. For
                more information, refer <a href="https://help.salesforce.com/s/articleView?id=data.c360_a_accelerated_data_lake_objects.htm&amp;release=264.0.0&amp;type=5" title="Caching retrieves data from the external source and temporarily stores it in a data lake object in Data 360. When you enable acceleration while creating a data stream, you turn on caching. We recommend caching when accessing large data sets or when data changes frequently. You can refresh the cached data at your preferred frequency.">Caching or Acceleration in Data Federation</a>. </p>
            </td>

          </tr>
          <tr class="">
            <td data-label="Setting" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e60 "> Upsert</td>

            <td data-label="Description" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e63 ">
              <p>This mode is available for data streams created from file-based connectors. For
                each refresh, the data stream is updated with data added or changed since the last
                refresh. </p>
              <p style="margin-bottom:0;">When the refresh mode is set to upsert, you can efficiently delete
                records from a DLO using a file. See <a href="https://help.salesforce.com/s/articleView?id=data.c360_a_delete_data_stream_records_deletion_file.htm&amp;release=264.0.0&amp;type=5" title="To delete data records ingested by using the Amazon Simple Storage Service (Amazon S3), Google Cloud Storage (GCS), Microsoft Azure, Ingestion API, or Secure File Transfer Protocol (SFTP) connector, create a delete file. Create the file in CSV or Parquet file format. S3, GCS, and Azure data streams support Parquet file ingestion.">Delete Records by Using a Delete File</a>.</p>
            </td>

          </tr>
          <tr class="">
            <td data-label="Setting" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e60 ">Full Refresh</td>

            <td data-label="Description" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e63 ">During each refresh cycle, all existing data is deleted and replaced with the
              newly imported dataset. If no other refresh setting is available, full refresh is
              the default refresh mode.</td>

          </tr>
        </tbody>
      </table>

## Conditional Settings

<table class="slds-table slds-table_bordered slds-max-medium-table_stacked slds-m-bottom_small ">
        <colgroup>
          <col style="width:50%">
          <col style="width:50%">
        </colgroup>
        <thead class="slds-text-title_caps ">
          <tr class="">
            <th scope="col" class="slds-cell-wrap" style="vertical-align:top;" id="d106226e146">Setting</th>
            <th scope="col" class="slds-cell-wrap" style="vertical-align:top;" id="d106226e149">Description</th>
          </tr>
        </thead>
        <tbody>
          <tr class="">
            <td data-label="Setting" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e146 ">Filter</td>

            <td data-label="Description" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e149 ">
              <p>Use a filter to extract only the records that fulfill the specified condition.
                The syntax of the condition depends on the source system. For example, on a source
                that relies on SQL92 syntax, the filter could be, <code>s_nationkey = 10 and
                  s_acctbal &gt; 2000</code>.</p>
              <p>When filtering on a DateTime field, the filter
                value must include the time zone. Use either UTC format (for example,
                <code>2026-03-30 08:00:00Z</code>) or local time with a time zone offset
                (for example, <code>2026-03-30 15:00:00+07:00</code>). This requirement
                applies to all Data 360 connectors.
              </p>
              <p><strong>Data Explorer</strong> displays timestamps
                in your user profile's time zone, or your org's default time zone (from
                <strong>Company Information</strong> in Setup), without the time zone offset.
              </p>
              <p style="margin-bottom:0;">To get
                the correct format for filters, use the <strong>Query Editor</strong>, which displays
                timestamps in UTC.</p>
            </td>

          </tr>
          <tr class="">
            <td data-label="Setting" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e146 ">Supported Operators</td>

            <td data-label="Description" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e149 "> For database sources, see the list in the section entitled "Supported Operators" below for more details.</td>

          </tr>
        </tbody>
      </table>

## File Settings

These settings are specific to data streams that use file-based connectors.

<table class="slds-table slds-table_bordered slds-max-medium-table_stacked slds-m-bottom_small ">
        <colgroup>
          <col style="width:50%">
          <col style="width:50%">
        </colgroup>
        <thead class="slds-text-title_caps ">
          <tr class="">
            <th scope="col" class="slds-cell-wrap" style="vertical-align:top;" id="d106226e217">Setting</th>
            <th scope="col" class="slds-cell-wrap" style="vertical-align:top;" id="d106226e220">Description</th>
          </tr>
        </thead>
        <tbody>
          <tr class="">
            <td data-label="Setting" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e217 ">Log an error if no file is found</td>

            <td data-label="Description" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e220 ">
              <p>When enabled and a file isn’t found during the data stream run, the line item
                in the refresh history for that run shows a failure status. A notification is
                triggered if you subscribe to data stream failure notifications. For more
                information, see <a href="https://help.salesforce.com/s/articleView?id=000389674&amp;type=1&amp;language=en_US" target="_blank">Data Cloud: Control Event Notifications Using
                  Flows</a>.</p>
              <p style="margin-bottom:0;">When this setting isn’t enabled and a file isn’t found during the
                data stream run, the refresh history shows 0 records but with a success status.
              </p>
            </td>

          </tr>
          <tr class="">
            <td data-label="Setting" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e217 ">Is headerless file retrieval enabled</td>

            <td data-label="Description" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e220 ">
              <p>When set, <span>Data 360</span> accepts the data records without headers.
                Don’t turn on this setting while you’re creating the data stream because the header
                information is necessary for the initial run when the schema is being generated. You
                can enable the setting after the initial run. </p>
              <p style="margin-bottom:0;">To avoid data corruption, maintain
                the order of fields in the incoming data records.</p>
            </td>

          </tr>
          <tr class="">
            <td data-label="Setting" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e217 ">Refresh only new files</td>

            <td data-label="Description" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e220 ">When set, all files are retrieved when the data stream is first run. For
              subsequent runs, only new files are retrieved.</td>

          </tr>
          <tr class="">
            <td data-label="Setting" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e217 ">Refresh initial file immediately</td>

            <td data-label="Description" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e220 ">
              <p style="margin-bottom:0;">When set, <span>Data 360</span> doesn't wait until the scheduled run
                and retrieves the initial file immediately.</p>
            </td>

          </tr>
        </tbody>
      </table>

## Supported Operators, Functions, and Keywords

For database sources,the `WHERE` clause supports the following:

Supported Operators

- Comparison: <, >, <=, >=, =, <>
- Pattern matching: LIKE, RLIKE
- Null checks: IS NULL, IS NOT NULL
- Set membership: IN, NOT IN
- Range evaluation: BETWEEN
- Uniqueness: DISTINCT

Supported Functions and Keywords

The following SQL functions and expressions are allowed within the `WHERE` clause:

- `CEIL`
- `CFLOOR`
- `CAST`
- `TRIM`
- `TIMESTAMP`
- `DATE`

Not Supported in the `WHERE` Clause

The following constructs are not supported and will result in an error if used:

- Data Manipulation Language (DML) operations: `INSERT`, `UPDATE`, `DELETE`
- Complex query structures: `JOIN,` `UNION`, `INTERSECT`, and subqueries
- Control characters: semicolons (;)
- Any other SQL functions or keywords not explicitly listed in the supported sections above

## Schedule

The frequency setting describes the time interval between executions of your data stream job. Based on connector capabilities, supported schedule intervals can vary.

<table class="slds-table slds-table_bordered slds-max-medium-table_stacked slds-m-bottom_small ">
        <colgroup>
          <col style="width:50%">
          <col style="width:50%">
        </colgroup>
        <thead class="slds-text-title_caps ">
          <tr class="">
            <th scope="col" class="slds-cell-wrap" style="vertical-align:top;" id="d106226e419">Setting</th>
            <th scope="col" class="slds-cell-wrap" style="vertical-align:top;" id="d106226e422">Description</th>
          </tr>
        </thead>
        <tbody>
          <tr class="">
            <td data-label="Setting" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e419 ">Frequency</td>

            <td data-label="Description" class="slds-cell-wrap" style="vertical-align:top;" headers="d106226e422 ">
              <div>
                <p>Set the frequency of data refresh by choosing the relevant frequency. The
                  available refresh frequencies are: </p>
                <ul>
                  <li>Hourly </li>

                  <li>Daily at a specified hour </li>

                  <li>Weekly on a specified day and time</li>

                  <li>Monthly on a specified day and time</li>

                </ul>
                <p>All frequency times are in UTC and are applicable for both full and
                  incremental refreshes.</p>
              </div>
              <p style="margin-bottom:0;">Amazon Simple Storage Service (Amazon S3), Azure Blob Storage, and Google Cloud Storage data streams have 5-, 15-,
                or 30-minute intervals. These connectors can synchronize data and deduplicate jobs
                to provide these narrow intervals.</p>
            </td>

          </tr>
        </tbody>
      </table>