The Jobs endpoint provides a programmatic interface to orchestrate jobs, including listing jobs, getting status for specific jobs, creating jobs, and stopping jobs.
Important: You cannot use the Jobs endpoint to interact with jobs that are scheduled in the UI Scheduler page.
Many job requests require resource IDs. See Obtaining Resource IDs.
Job States
Jobs run via the Jobs API can have the following states:
- PENDING: The job has been submitted but is not yet running.
- RUNNING: The job is running.
- STOPPING: The job has been stopped before finishing, and is in the process of stopping.
- DONE: The job is finished. This can indicate that the job finished successfully, was stopped, or failed.
Creating Jobs
You can create the following types of jobs with the Jobs create operation:
| Job Type | Description | Requirements |
|---|---|---|
| load | Refreshes (loads) a source dataset. | The source being loaded (refreshed) must have been previously added in the Tamr Cloud UI.
You must provide the |
| update (data product) | Runs a configured data product and updates the data product results. | The data product must have been added and configured in the Tamr Cloud UI. The data product does not have to be run in the UI before creating this job via API.
You must provide the |
| update (workflow) | Updates the Tamr RealTime system of record with either the output of a data product run or a source dataset, depending on the workflow configuration. |
The worklow must be configured by Tamr before this job can be run. You must provide the |
| publish | Publishes datasets to configured publish destinations.
Create a publish job to export data product datasets for golden records, source records, enhanced source records, and so on. For Tamr Realtime users, create a publish job to publish the RealTime datastore dataset to a configured S3 destination. |
The publish destination must be configured in the Tamr Cloud UI. Before publishing the Tamr RealTime datastore, contact Tamr ([email protected]) to configure the destination.
You must provide the
Additionally, if you are publishing a non-legacy data product, you must also supply the |
Running the Data Pipeline via API
You can run all tasks in your data pipeline by making multiple, sequential calls to the Jobs create endpoint. These tasks include refreshing source datasets, running the data product, and publishing the resulting datasets to configured destinations.
Important: For best results, leave a buffer of several hours between jobs for the same data product. The duration of each job depends on the size of the dataset and complexity of the mastering flow. Monitor how long jobs take to complete, and use that information to ensure you have an adequate buffer between jobs.
Concurrent Job Limits
Tamr Cloud supports 5 concurrent running jobs per tenant by default. Additional job requests return the following error: 429 Too Many Requests.