Corrupt data can often be restored. Before doing so, it's helpful have an understanding of the data stores in the system.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIT.htmlUsing PIT, you can restore your database to a state ranging from the last database snapshot to a few minutes before the current time.
For detailed steps about restoring corrupt data, see Restore Corrupt Data.
The diagram below provides an overview of all the data stores in the system:
Order Management Plus treats Salesforce, Kubernetes, all Apache Kafka queues and external systems as data stores.
|
Data Store |
Contains |
Data Characteristics |
Comments |
|---|---|---|---|
|
PSQL |
Downstream Orders/Order Items Orch Plans/Orch Items Inventory Items |
Source of truth Updated frequently |
Ensure Inventory Items are linked to Assets. Inventory Items also store unique information about the service to recover manually if lost. |
|
Riak |
Session storage |
Disposable |
|
|
Riak |
Grants cache |
Disposable |
Grants cache is fetched from a persistent storage i.e. Postgres Database and automatically reconstructed from persistent storage if data is not found in cache (i.e. Riak). You can trigger the cache reload operation manually to resync the grants cache data if persistent storage state is rolled back. |
|
Kubernetes |
Deployments for workers |
Derived Updated extremely infrequently |
Automatically reconstructed when worker-manager restarts |
|
PSQL |
Catalog |
Derived from SFDC Updated infrequently |
|
|
PSQL |
Orders/Order Items Assets |
Derived from SFDC Updated frequently |
Some of the order information is derived from Order Management Plus data as a result of integration |
|
S3 |
Catalog |
Derived from SFDC data Updated infrequently |
Can re-sync with SFDC |
|
Completer queue (Kafka) |
Orch Items |
Derived from PSQL Updated frequently |
After the rollback, these queues are in inconsistent state. Therefore, they are drained during restoration. Any items streamed for completion are marked as Fatally failed using product-supplied tooling. |
|
Kafka2SFDC queue (Kafka) |
Orders/Order Items |
Derived from PSQL Updated frequently |
To check the impact of the data rollback on the assets, the retention time for the order_orderitems queue is 30 minutes. A REST service and a tool allows you to export asset changes from this queue |
|
Kafka2Schedule queue (Kafka) |
Orch Items |
Derived from PSQL Updated frequently |
|
|
Salesforce |
Catalog |
Critical Updated infrequently |
|
|
Salesforce |
Orders (structure, state information) |
Critical Updated frequently |
When the PSQL instance rolls back to a previous point in time, there are orders that were submitted between the point to which we restore the PSQL and the current clock time. The state of provisioning of these orders is unknown and requires an implementation-specific tool and manual activities for restoration. The Products team will provide tooling to identify those orders. |
|
Salesforce |
Assets |
Critical Updated frequently |
As with orders, some assets can be created or updated during the lost data window. The Products team will provide tools to identify the changes made to the assets |
|
External system |
Critical Updated frequently |
Restore strategy is on a case by case basis |
|
|
Incoming queue (Kafka) |
Orders/Order Items |
Critical |
This queue should have retention configuration large enough to hold requests for 30 minutes. |