Data Stream and Data Lake Object Refresh History

Data 360 provides record-level statistics of the data ingested by data streams and records processed into a data lake object (DLO).

Data Stream Refresh History

View the Refresh History tab on the data stream record home for the record statistics. After the initial ingestion of data into Data 360, a data stream pulls new data based on specific schedules and refreshes the data. The data refresh information can vary based on the connector that the data stream ingests the data from.

Field Name	Description
Refresh Date	The date and time of the last data refresh, displayed in your org’s time zone.
Refresh Mode	The way data is refreshed during each cycle, such as a full or incremental refresh.
Duration	Time taken for the data refresh.
Status	The status of the last refresh.
Records Processed	The number of records processed during the refresh cycle.
Records Added	The number of records ingested or added during the refresh cycle.
Records Removed	The number of records deleted from Data Cloud during a refresh cycle.
Problem Records	The number of problem records not ingested due to data quality issues. If a row encounters partial failures, the system counts the row as one problem record. Similarly, if a row encounters a total failure, the system counts the row as one problem record. The system focuses on the number of rows affected, not the individual failures within a row.

Data Lake Object Refresh History

View the Refresh History tab on the DLO record home for the record-level statistics. The statistics are aggregated in hourly time windows, and the refresh history is kept for up to 20 days. The information is available for DLOs that are used by data streams and data transforms. If there are no changes to data records in a DLO during the time window, no details are listed on the Refresh History tab.

Field Name	Description
Start Time	The beginning time of an hourly time window for aggregating the record-level statistics. It’s displayed in your org time zone.
End Time	The end time of an hourly time window for aggregating the record-level statistics. It’s displayed in your org time zone.
Total Records	The total number of records in the DLO at the end of the time window.
Records Added	The number of records that are added to the DLO during the time window.
Records Updated	The number of records that are updated in the DLO during the time window.
Records Removed	The number of records that are deleted from the DLO during the time window.

Note The field mapping counts displayed on the DLO record home don't match the fields that show as mapped on the data stream field mapping page. This is because mappings for certain system fields are system managed and don't appear on the field mapping page.

Understanding the Difference Between Data Stream and DLO Metrics

Data Stream Refresh History and DLO Refresh History provide different views of your data.

Feature	Data Stream History	DLO Refresh History
Perspective	Business Logic View	Storage View
Focus	Records processed from the source (for example, Snowflake)	File-level operations in the Data Lake storage
Records Added	New records from the source	Records written to new storage files
Records Removed	Records deleted from the source	Records from storage files being rewritten or replaced

Note Data 360 uses Apache Iceberg to manage data storage in Parquet files. Apache Iceberg is a table format that tracks which files contain your data. When data changes, Apache Iceberg often rewrites entire files. To update even one record, Data 360 marks the old file as deleted (removing all its records) and creates a new file with both the old records and the changes.

Example

You process 13,169 new rows into a table that already has 7,083,765 rows.

Data Stream shows: 13,169 Records Added
DLO Refresh shows: 7,083,765 Records Removed and 7,096,934 Records Added

Data 360 rewrote the entire 7M+ row table to safely commit the 13K changes. Total Table Count = Records Added - Records Removed.

For billing and credit consumption, always refer to Data Stream Refresh History metrics, which reflect the actual business data processed.