Records in the SOR are assigned a persistent identifier called the record ID (rec_id). Source record and Tamr ID updates and merges are tracked in the record history for every record in the SOR.

Available Datasets to Publish

You can publish the following datasets from the SOR to a configured S3, ADLS2, GCS, BigQuery or Snowflake publish destination:

• RealTime (golden) records
• Tamr ID to record ID mappings
• Relationships
• Source records

Published Dataset Formats

If you are publishing to a BigQuery or Snowflake connection, the dataset publishes as a table.

If you are publishing to an S3, ADLS2, or GCS connection, you can choose the file format:

• Delta
• Iceberg
• Avro
• CSV
• NDJSON
• Parquet

Details for CSV Files

Tamr Cloud publishes CSV files to the specified directory in multiple parts. The name of your files will be output-<number file>.csv. Once publish is done, there will be an additional file in your directory named _SUCCESS; it is used to indicate that your publish job is complete and that data is ready to be consumed. This file is empty and does not contain any data.

Example directory after multipart file publish:

<provided_path>
    ├── output-00000.csv
    ├── output-00001.csv
    ├── output-00002.csv
    ├── output-00003.csv
    ├── output-00004.csv
    └── _SUCCESS

The following are details applicable for published CSV files:

• Delimiter: Comma (,)
• Encoding: UTF-8
• Header row: The first row is treated as a header.
• Trailing and leading spaces: Trimmed in both headers and values.
• Quote characters within values: The quote " character in values are escaped with a quote ".
  For example, Hello my name is "John Doe" is changed to Hello my name is ""John Doe"" in the published file.
• Commas within values: Values that contain commas are enclosed in quotes (" ").
  For example:
  • Company, Inc is converted to "Company, Inc"
  • Hello, my name is "John Doe" is converted to "Hello, my name is ""John Doe"""
• Multi-line values: Multi-line values are converted to single-line values.

Configuring System of Record Publish Destinations

Important Notes

• Important Note for Snowflake Destinations: If you are publishing RealTime Records (golden records) and choose to publish incremental changes, publishing will overwrite the existing records in the table. We recommend publishing incremental results to a staging table, and then merging the results to a separate Snowflake table storing all published data. See Snowflake's MERGE documentation.
• Known Issue for Publishing Array Values: This issue is relevant when using views in RealTime data products. When configuring record consolidation rules to determine values in your golden records, one option is to include all distinct values for the attribute (Collect Distinct). Currently, if you include golden record attributes with this consolidation setting in a view, the values are not included in the published golden record dataset.

To configure destinations:

1. Navigate to Destinations, select New Destination.
2. Depending on the dataset you want to publish, choose one of the following destination types:
   1. RealTime Records
   2. RealTime Tamr ID Mapping
   3. RealTime Relationships
   4. Source Records
3. Name the destination and select Next.
4. Select the data product or table ID for the RealTime table.
5. Optional. If you are publishing results of a limited release RealTime data product, select a configured view for this destination. See Adding and Configuring a RealTime Data Product for more information on configuring data views.
6. If you are publishing Source Records, choose whether to publish pre or post-standardized records from the RealTime data product (default) or by the search then create or update record API. If you selected a view, this option is set automatically by the view settings.
7. Select Next.
8. Select the connection to use to publish the dataset.
9. If you selected a BigQuery or Snowflake connection, in the Address field, enter the Table to which to publish the output. Select Next.
10. If you are publishing to an ADLS2, OneLake, S3, or GCS connection complete the following and then select Next.
    1. In the Address field, enter the file path to which to publish. This is the path for the published output within the configured bucket and prefix of the connection. After entering the path, the full URI to the published output displays. Do not include trailing or leading / unless they are intended to be part of the path.
    2. Choose the file format for the the published file.
    3. Specify the delimiter, quote, and escape characters for the published dataset.
11. If you are publishing RealTime Records (golden records), choose whether to enable Incremental Publish. If enabled, choose whether to:
    • Publish the records that were updated since the last time the dataset was published.
    • Publish records that were updated within a specified time window.
12. In the System Metadata section, review the metadata fields that will be included in the published output. You can choose to exclude optional fields and edit the field name values for the output.
    Note: After creating the destination, you can not edit the metadata configuration.
13. Select Next.
14. Select Create Destination.

The new destination appears on the Destinations page. You can publish the dataset by choosing the Play icon or by using the Jobs API (see Jobs Overview).

If you are running the job via the API, this job requires the destinationId for the destination, which is available on the Destinations page.

Published Dataset Schemas

Schema for RealTime Records

The default output for RealTime Records includes the following fields:

• data (as JSON)
• Optional metadata fields:
  • recordId
  • tableId
  • versionId
  • data (as JSON)
  • createdMs (time the record was created in milliseconds since unix epoch)
  • updatedMs (time the record was updated in milliseconds since unix epoch)

Schema for RealTime Tamr ID Mapping

This mapping table contains the Tamr ID from the data product outputs and its associated rec_id in the SOR.

The default output for RealTime Tamr ID Mapping includes the following fields:

• Required metadata fields:
  • tamrID
  • recordId
• Optional metadata fields:
  • tableId
  • versionId
  • createdMs (time the record was created in milliseconds since unix epoch)
  • updatedMs (time the record was updated in milliseconds since unix epoch)

Below is a sample of the mapping table output:

Schema for RealTime Relationships

The default output for RealTime Relationships includes the following fields:

• relationshipDetails
• Required metadata fields:
  • relationshipId
  • fromTableId
  • fromRecordId
  • toTableId
  • toRecordId
  • relationshipTypeId
• Optional metadata fields:
  • versionId
  • createdMs (time the record was created in milliseconds since unix epoch)
  • updatedMs (time the record was updated in milliseconds since unix epoch)

Schema for Source Records

The default output for Source Records include the following fields:

• data (as JSON)
• Optional metadata fields:
  • recordId
  • sourceID
  • sourceName
  • tableId
  • updatedMs (time the record was updated in milliseconds since unix epoch)