This topic describes how to set-up and operationally run an end-to-end data pipeline using a data product with the RealTime datastore. In this use case, the RealTime datastore is used as a System of Record (SOR). This deployment supports RealTime-powered Search APIs and 360 View pages.

Note that diagram above includes a connection to Snowflake as an example, but you can connect to any supported cloud storage application.

Setting up and running the data pipeline includes:

1. Configuring the batch data product.
2. Creating the Tamr RealTime SOR table.
3. Searching for records in the SOR.
4. (Optional) Orchestrating the pipeline.
5. (Optional) Configuring a webhook to monitor authoring events.

Once the data product is configured for real-time search, the RealTime datastore is refreshed each time you run the data product. You can use the search API operations to search for records in the RealTime datastore via exact or fuzzy match.

Step 1: Configure the Data Product

Configure and run the data product, following the instructions for the type of data product you are using. If necessary, perform curation on the results.

Next, follow the instructions to setup publish configurations instructions to setup publish configurations to publish the data product outputs to your downstream applications.

Step 2: Create the SOR Table

On the Configure Data Product page for the data product, select the RealTime Settings button and then enable the Enable RealTime data refresh option.

Now, each time your data product refreshes the outputs are written to and update records in the SOR table. The table id (tbl_) of the SOR is displayed on this page.

Step 3: Search for Records in the SOR

You can search for records in the SOR either in the Tamr Cloud UI or by using the Realtime Search API endpoint.

If search does not return a matching record, you add new records to the SOR by updating the source datasets and refreshing the data product.

Searching in the Tamr Cloud UI

To search for records in the SOR in the UI, select Data in the navigation menu and then choose the data product. You can then search for a record.

Searching Using the API

Use the Realtime Search API endpoint to search for and return records in the SOR with similar attribute values (fuzzy match). The search endpoint requires the table id (tbl_) of the SOR which is displayed on the API Settings page in Step 2. If you have questions about configuring your search queries for optimal performance, reach out to [email protected] for assistance.

Step 4: Orchestrate the Pipeline (Optional)

Orchestrate the pipeline to perform the following tasks in order. Each of these tasks can be run using the Job API create operation. See the Jobs API documentation for more information on running each of these jobs. Additionally, see Coordinating Data Product Refresh via the Jobs API for a sample script to perform these tasks end-to-end.

1. Refresh the sources in the data product. This job requires the sourceId for the source, which is available on the Configurations > Sources page.
2. Update (refresh) the data product. When this job completes, Tamr automatically applies the updates to the SOR. This job requires the dataProductId for the data product, which is available on the Configure Data Product page for that data product.
3. Publish the data product outputs. This job requires both the dataProductId and the destinationId. The destinationId is available on the Publish page for that data product.

You can also run each of these jobs manually in the Tamr Cloud UI.

Step 5: Configure a Webhook (Optional)

Tamr’s event-driven notification system enables you to keep all of your systems up to date with the golden records in the RealTime SOR. This system can publish Realtime notifications for authoring events to a user-specified endpoint (a webhook).

See Using Tamr RealTime Event Notifications for more information on configuring and using a webhook to monitor these events.