Records in the SOR are assigned a persistent identifier called the record ID (rec_id). Source record and Tamr ID updates and merges are tracked in the record history for every record in the SOR.

You can publish the following datasets from the SOR to a configured S3, ADLS2, GCS, BigQuery or Snowflake publish destination:

• RealTime (golden) records
• Tamr ID to record ID mappings
• Relationships
• Source records

If you are publishing to a BigQuery or Snowflake connection, the dataset publishes as a table.

If you are publishing to an S3, ADLS2, or GCS connection, you can choose the file format:

• Delta
• Iceberg
• Avro
• CSV
• NDJSON
• Parquet

Configuring System of Record Publish Destinations

Important for Snowflake Destinations: If you are publishing RealTime Records (golden records) and choose to publish incremental changes, publishing will overwrite the existing records in the table. We recommend publishing incremental results to a staging table, and then merging the results to a separate Snowflake table storing all published data. See Snowflake's MERGE documentation.

To configure destinations:

1. Navigate to Destinations, select New Destination.
2. Depending on the dataset you want to publish, choose one of the following destination types:
   1. RealTime Records
   2. RealTime Tamr ID Mapping
   3. RealTime Relationships
   4. Source Records
3. Name the destination and select the RealTime table.
4. Optional. If you are publishing results of a limited release RealTime data product, select a configured view for this destination. See Adding and Configuring a RealTime Data Product for more information on configuring data views.
5. If you are publishing Source Records, choose whether to publish pre or post-standardized records from the RealTime data product (default) or by the search then create or update record API. If you selected a view, this option is set automatically by the view settings.
6. Select the connection to use to publish the dataset.
7. If you selected a BigQuery or Snowflake connection, in the Address field, enter the Table to which to publish the output.
8. If you are publishing to an ADLS2, OneLake, S3, or GCS connection:
   1. In the Address field, enter the file path to which to publish. This is the path for the published output within the configured bucket and prefix of the connection. After entering the path, the full URI to the published output displays. Do not include trailing or leading / unless they are intended to be part of the path.
   2. Choose the file format for the the published file.
   3. Specify the delimiter, quote, and escape characters for the published dataset.
9. If you are publishing RealTime Records (golden records), choose whether to enable Incremental Publish. If enabled, choose whether to:
   • Publish the records that were updated since the last time the dataset was published.
   • Publish records that were updated within a specified time window.
10. In the System Metadata Configuration section, review the metadata fields that will be included in the published output. You can choose to exclude optional fields and edit the field name values for the output.
    Note: After creating the destination, you can not edit the metadata configuration.
11. Select Create Destination.

The new destination appears on the Destinations page. You can publish the dataset by choosing the Play icon or by using the Jobs API (see Jobs Overview).

If you are running the job via the API, this job requires the destinationId for the destination, which is available on the Destinations page.

Published Dataset Schemas

Schema for RealTime Records

The default output for RealTime Records includes the following fields:

• data (as JSON)
• Optional metadata fields:
  • recordId
  • tableId
  • versionId
  • data (as JSON)
  • createdMs (time the record was created in milliseconds since unix epoch)
  • updatedMs (time the record was updated in milliseconds since unix epoch)

Schema for RealTime Tamr ID Mapping

This mapping table contains the Tamr ID from the data product outputs and its associated rec_id in the SOR.

The default output for RealTime Tamr ID Mapping includes the following fields:

• Required metadata fields:
  • tamrID
  • recordId
• Optional metadata fields:
  • tableId
  • versionId
  • createdMs (time the record was created in milliseconds since unix epoch)
  • updatedMs (time the record was updated in milliseconds since unix epoch)

Below is a sample of the mapping table output:

Schema for RealTime Relationships

The default output for RealTime Relationships includes the following fields:

• relationshipDetails
• Required metadata fields:
  • relationshipId
  • fromTableId
  • fromRecordId
  • toTableId
  • toRecordId
  • relationshipTypeId
• Optional metadata fields:
  • versionId
  • createdMs (time the record was created in milliseconds since unix epoch)
  • updatedMs (time the record was updated in milliseconds since unix epoch)

Schema for Source Records

The default output for Source Records include the following fields:

• data (as JSON)
• Optional metadata fields:
  • recordId
  • sourceID
  • sourceName
  • tableId
  • updatedMs (time the record was updated in milliseconds since unix epoch)