Records that can’t be ingested into Data 360 due to data quality issues are logged as problem records and stored in a problem record data lake object (DLO). Problem records are created when data is ingested through: all structured ingest connectors, Web and Mobile App connectors, and Ingestion APIs.
Problem Record Considerations
- The problem record DLO is named
PR_DLO_Name. The name indicates the data stream that the record is associated with. - Data 360 stores problem records for 30 days. However, if you delete a data stream with problem records before the 30 days, its problem record DLOs are also deleted.
- You can’t package problem record DLOs.
- You can use a problem record DLO as an input to a data transform but not as an output.
- The data stream Refresh History tab lists all problem records related to data stream ingestion from a data source.
- You can query the DLO using Data Explorer, the Query Editor, or Query API.
Example of a Problem Record DLO Payload
"error_context__c": "{"dataStreamJobId":"ec579524-993f-433f-b3bd-48114a9f4ac5","sourceFile":"bad_row.csv"}",
"error_code__c": "TRANSFORM_EVALUATION_ERROR",
"error_timestamp__c": "2024-02-23T17:50:33.547Z",
"detailed_error__c": "{"errorMessage":"Transformation error","additionalInfo":{"transformationErrors":[{"fieldName":"datetime__c","errorMessage":"INVALID_DATE_FORMAT - Cannot parse source field DateTime with value \\"abc-ef-gh 03:03:03\\" using common date patterns"}]}}",
"error_row"__c: "{"id__c":"3","firstname__c":"Loree","datasourceobject__c":"s3://dev1-uswest2-cdp001-test-data/jack/person/person_with_invalid_date_time.csv","internalorganization__c":null,"lastname__c":"Drewski","datasource__c":"jack_person_with_invalid_date_time_csv_2","datetime__c":null}",
"error_fields__c": "datetime__c"
Schema Attributes of a Problem Record DLO
The problem record DLO has these schema attributes.
| Field Name | Field API Name | Field Type | Description |
|---|---|---|---|
| Id |
id__c
|
Text | The auto-generated unique ID used for the primary key. |
| Error Context |
error_context__c
|
Text | The JSON string along with the job ID. |
| Error Code |
error_code__c
|
Text | Error code string that describes the reason for a record drop. |
| Error Timestamp |
error_timestamp__c
|
DateTime | Timestamp of when the error record was ingested in Data 360. |
| Detailed Error |
detailed_error__c
|
Text | Additional details about the record error. |
| Error Row |
error_row__c
|
Text | Displays the content in the problem record. |
| Error Fields |
error_fields__c
|
Text | Displays a concatenated string of all error fields for record rejection. It applies only to fields with error_code =TRANSFORM_EVALUATION_ERROR. |
These are the different error types that result in a problem record.
| Error Code | Description |
|---|---|
NOT_NULL_CONSTRAINT_VIOLATION
|
The record is dropped due to null value in a required field. For example, the primary key value is null. |
| TRANSFORM_EVALUATION_ERROR | A partially rejected record. An error occurs during the transformation process of a field. There can be multiple errors in one record processing multiple fields. |
| UNHANDLED_ERROR | The record is dropped due to an error during row processing. |
| DUPLICATE_ROW | The record is dropped because multiple records with the same primary key are in the batch. |