HomeGuidesAPI ReferenceChangelog
HomeGuidesTamr API ReferenceTutorialsEnrichment API ReferenceSupport Help CenterLog In


Transformations are tools that affect the final output of a unified dataset, replacing the effort of individually transforming each input dataset before you load them into Tamr.

You can apply transformations to all records in the unified dataset, or to records from specific input datasets. As a best practice, Tamr recommends that you apply transformations to the unified dataset whenever possible.

  • Applying transformations to the unified dataset is, in most cases, more efficient than applying them to the input datasets, so processing completes faster.
  • You can use the Tamr-created origin_source_name attribute to apply transformations to specific input datasets, even when they are run on the unified dataset. For example, you can use a case expression that checks the value of origin_source_name before applying a data cleaning transformation. Alternatively, you can use origin_source_name as part of the condition of a JOIN statement to apply the JOIN to a subset of input datasets. See Managing Primary Keys and Join.

Transformations operate at a project level and produce a single output dataset based on one or more input datasets. Transformations never change input datasets.

Accessing Transformations

You can add transformations to schema mapping, categorization, and mastering projects. See Enabling Transformations in Categorization and Mastering projects.

To access transformations:

  1. On the Unified Dataset page, select Show Transformations. The transformation panel appears with drop-down menus for adding transformations.
  1. Choose Add Transformation.
    If you add a transformation to the Input Datasets section, the note "For records from datasets" appears in the bottom left corner of the transformation panel. Select this note to specify the datasets affected by this transformation. By default, the transformation is applied to all input datasets.

You can also add a transformation between other transformations. Move the mouse cursor between two existing transformation editor boxes and select Add transformation.

  1. To change the scope of a transformation from the input datasets to the unified dataset or vice versa, open both sections in the transformation editor and drag the transformation between the sections.

Displaying and Previewing Transformations

Once you write your transformation, you can use Preview all in the transformations panel to preview a transformation. If the transformation violates any rules, Tamr issues an error message explaining why it failed. This preview allows you to test transformations without affecting the entire dataset. You can iterate and improve your transformations quickly, viewing the results as you go before saving any changes.

Once you are satisfied with the transformation results, save the changes. The number on the Save changes button indicates how many changes have been made since the last time you saved changes. If you don't save transformations, they will not persist if you navigate away from the page. The Save changes button is disabled if you have any transformations with errors and when no transformations have changed.

Save changes control with change countSave changes control with change count

Save changes control with change count

To revert transformations and go back to the last time they were saved, choose Cancel Changes. This reverts all changes that weren't saved.

You can also preview a the effect of a transformation by clicking Preview on each individual transformation. Subsequent transformations are then grayed out to signify that they are not included in the preview, but you can still edit and reorder them. To save changes, select Save changes. This saves all changes, and not only those made to the previewed set of transformations.

To see a preview of your data before any transformations are applied, choose Preview at the top of the Input Datasets section.

Reordering Transformations

Transformations only have local effects and you can reorder them at any time. Reordering may change the output.

To reorder transformations, select and hold the icon with two horizontal lines at the top left of the transformations panel and then drag it to the desired location within the transformations script.

Reordering transformations with drag and dropReordering transformations with drag and drop

Reordering transformations with drag and drop

Saving and Applying Transformations


Applying Transformations

To apply transformations so that they become part of the data pipeline for your unified dataset, choose Update Unified Dataset on the Unified Dataset and Schema Mapping pages. This applies transformations to the unified dataset.

The Save changes button on the transformation panel keeps your work in case you navigate away or want to come back to it later. To apply this saved work to all records in the unified dataset, export the transformed unified dataset. To include the transformed unified dataset in your data pipeline, select Update Unified Dataset.

Removing a Transformation

To remove a transformation, select the Trash icon in the transformation editor, then select Remove in the confirmation popup. Then, Save Changes to commit your action.

Making System-Wide Changes to a Transformation

If a change, such as a company-wide policy or Tamr software release, affects a transformation that is used in multiple projects, a system administrator can use a utility script to update that transformation across all projects. See Transformation Tools.

Data Types

Tamr transformations support multiple data types. See Data Types and Transformations.

Enabling Transformations in Categorization and Mastering Projects

To add transformations to a mastering or categorization project you must indicate that you want the project to include this feature. You can enable transformations in Categorization and Mastering projects during project creation or after a project is already created.

In this example for a new mastering project, check **Enable Transformations** to include this feature.In this example for a new mastering project, check **Enable Transformations** to include this feature.

In this example for a new mastering project, check Enable Transformations to include this feature.

If you decide to include transformations in a mastering or categorization project after an admin creates it, locate the project on the Tamr dashboard and move your cursor over the top right corner to access Edit (pencil icon). The same dialog opens so that you can check Enable Transformations.

To change an existing project, select **Edit**.To change an existing project, select **Edit**.

To change an existing project, select Edit.

Once enabled, transformations cannot be disabled in a project.

If you are writing transformations in a Categorization or Mastering project, or plan to use a unified dataset containing transformations in a second project, it is important that the Tamr-generated attributes origin_source_name and origin_entity_id meet certain conditions. Transformations can be used to maintain these conditions:

  • origin_source_name must be a string type. Each string should be a name of one of the input datasets.
  • origin_entity_id must be a string type.

Additionally, the tamr_id attribute generated by Tamr must be a unique string type, since it is a primary key that Tamr manages for you.

Additional Information

Functions List

For a full list of all supported functions, see Functions. You can also get transformation help in-product.

Metadata for Input Datasets

You can set metadata values for an input dataset or any of its attributes in a mastering, categorization, or schema mapping project. You can then use the metadata in transformations.

For more information, see Using Metadata in Transformations.

Referencing Attributes

To reference attributes in a transformation script, wrap them in double quotes, although this is not required (attribute and "attribute" both work). You may reference an attribute without using any quotes, however, any attribute containing spaces or escaped characters must be wrapped in double quotes. An attribute name containing double quotes itself can be referenced by escaping the double quotes. For example, this is an "attribute name" becomes "this is an ""attribute name""".

Attributes in transformations are case sensitive.

Referencing Datasets

Dataset names follow the same pattern as attributes. Wrap dataset names in double quotes if they include spaces or escaped characters, such as USE "myData.csv"; or USE my_data;. See Referencing Other Datasets, or see join for an example referencing an input dataset.

Using Single Quotes

Single quotes are interpreted as string literals 'string'.

Tab Autocomplete

For transformations such as Script and Formula, pressing the tab key provides a list of suggested inputs, including functions and attributes.
Hints autocomplete with tab in the transformation editor.

Did this page help you?