Recover for ServiceNow Synthetic Backups

Using Auditing

Note Enabled by default.

ServiceNow tracks creation, updates, and deletions for out-of-the-box tables like incident, change, and problem tables with auditing. However, auditing is available for a small subset of audit-enabled tables because it is not scalable to configure auditing on all tables due to the potential volume of retained audited data.

Synthetic Backups with Auditing uses the following logic:

• For new records and updated records, we query the table and sort by the sys_updated_on field to identify what has been created or updated since the last backup.
  • Operating with the caveat that past/future timestamps are possible and these field values can be changed via script.
• For deleted records, we query sys_audit_delete to identify what was deleted since the last backup.
  • These tables are excluded from our backups due to volume.
• For tracking changes in reference tables, we query sys_rollback_incremental.

Using Business Rules

Note Enabled by default.

A business rule (BR) is a “server-side script that runs when a record is displayed, inserted, updated, or deleted, or when a table is queried”. Utilizing this platform ability, we track record creations, updates, and deletions on a selected set of tables - ideally those which have a large data size / volume (100k+), but low data change rate (less than 30%) - and record these changes in our custom change log table. We pull from this changelog table when we run our synthetic backups.

As business rules add resource overhead on the ServiceNow instance, a delicate balance must be achieved in-order to justify the efficiency gains without negatively impacting platform performance. Thus, we can’t apply business rules to ALL tables, just a select subset that meets the criteria mentioned above. For now, we need to manually identify which tables are candidates for Synthetics. This involves a combination of asking customers which tables are mission-critical and comparing them to the data we collect in our OpenSearch dashboards to see which tables meet our conditions.

Once tables are identified, Own will configure and create the Synthetic BRs via the Table API. BRs will be enabled for each table specified and all references to that table. BRs will not be created for parent or child tables. If a table is audited enabled, we will not create a BR (see Auditing above).

Installation

• Download the Global scoped Update Set via the Service Options.
• Import and commit the Update Set in ServiceNow.

The Update Set contains the following:

• Custom Tables
  • Own Recover BR Changes: Stores info on our business rules (created in sys_script table), their actions (create, update, or delete) and the tables they are configured for. This table is primarily used to check if our BRs were disabled between backups. If so, we fall back to a Full backup for the table because we risk missing changes.
  • Own Recover Changelog: Stores records that were created, updated, or deleted for our Synthetic tables. Contains values 0 (create), 1 (update) or 2 (delete).
  • Own Recover Config: Tracks our high water mark (HWM) which is the ID of the latest record in the changelog fetched.
• Table Cleaner
  • Own Recover BR Changes: Remove records that are more than 1 month old.
• Scheduled Job
  • Own Recover Changelog Cleaner: After a Synthetic Backup, remove all records up to the latest HWM. This will run on a set schedule (ex: every 12 hours).

Example order of operations:

Note At least one full backup is required before running synthetic backups.

• Full Backup
• Synthetic Backup: 10 records were captured in the changelog table. Our Synthetic Backup retrieves the 10 records and sets the 10th record value as the HWM in the Own Recover Config table. The Own Recover Changelog Cleaner removes the 10 records we backed up.
• Synthetic Backup: 10 more records are captured in the changelog table. Our Synthetic Backup retrieves records 11-20 and sets the 20th record value as the HWM in the Own Recover Config table. The Own Recover Changelog Cleaner removes the 10 records we backed up.
• Synthetic Backup: Repeat the above until scheduled weekly Full backup.

Note

• Both approaches above require 1 Full backup before running Synthetics.
• Importing data via XML will not trigger business rules.
• A table can only support either Synthetics with Business Rules or Synthetic with Auditing.
• Table Cleaner (sys_auto_flush) jobs delete data from tables on a set schedule. These jobs do not trigger our business rules and the deletes are not captured in sys_audit_delete. Thus, these deletions will not be captured in both Synthetic Backups options.
• Using setWorkflow(false) to update records in a script will not trigger business rules or be captured via Auditing.