User GuidesAPI ReferenceRelease NotesEnrichment APIs
Doc HomeSupportLog In

Defining the Blocking Model

Create a blocking model so that Tamr Core can generate matching and non-matching record pairs.

The purpose of the blocking model is to identify attributes that can help Tamr Core identify duplicate records. The blocking model helps make Tamr Core efficient by including all possible matching pairs of records (maximizing recall), while allowing some non-matching pairs to be included for experts to label.

You build a blocking model by adding one or more blocking clauses and configuring their terms:

  • Each clause in your model is separated by an OR
  • Each term within a clause is connected by an AND

Adding a Blocking Clause

A blocking clause consists of one or more terms which are logically connected by AND. Each complete clause is connected by OR.

To add a blocking clause:

  1. Navigate to the Pairs page.
  2. Choose Manage pair generation. A template for the first term appears. See Configuring a Blocking Term.
  3. Select Add another row to add another term to the clause.
  4. To create a different clause, move your cursor over the area between two existing terms and select the "OR" separator that appears.
Separating two terms in a single clause to make two clauses.Separating two terms in a single clause to make two clauses.

Separating two terms in a single clause to make two clauses.

Configuring a Blocking Term

Note: As a best practice, limit the number of terms per clause to two or three. Adding more than four terms results in slower pair generation.

To configure a blocking term:

  1. Navigate to the Pairs page.
  2. Select Manage pair generation.
  3. Select the unified attribute name.
  4. Select a similarity threshold (%) and similarity function.
    Tip: Avoid low similarity thresholds (that is, less than 0.5 for text similarity).
  5. For text attributes, select a tokenizer.
    Note: Checking the Is Null box generates pairs where at least one attribute is null, meaning this filters out all respective non-null:non-null value pairs for pair generation.

See Tokenizers and Similarity Functions.

Estimating Record Pair Counts and Blocks

Tamr Core uses the blocking model to arrange records into blocks so that duplicate records belong to the same block. A record can belong to more than one block. To help determine the efficiency of your model, you can estimate the number of records per block and the number of blocks per record. By default, the estimate uses a sample of 20M pairs or about 6,300 records.

The efficiency of a model is affected by the following:

  • Number of entries per clause in the blocking model: blocks per record increase with the number of terms.
  • Tokenizer type: trigram and bigram increase the number of blocks per record.
  • Token weighting type: no weighting increases the number of blocks per record compared to IDF weighting.
  • Similarity function: For longer text attributes, Jaccard can be a better option than cosine when there are more than 10 tokens per value.
  • Similarity thresholds (%): lower thresholds increase the number of blocks per record.

To estimate record pair counts and blocks:

  1. Navigate to the Pairs page.
  2. Select Manage pair generation.
  3. Configure the blocking terms in one or more clauses.
  4. Select Estimate Counts. Estimate counts display for each clause. Overall estimates for the blocking model display at the bottom of the screen.

The format of the estimate counts is:
<x> from <y>
<n> blocks per record
where
<x> is the estimated number of pairs that are expected to meet the clause or blocking model. These pairs will be written to the Pairs page for subject matter review and verification.
<y> is the estimated number of pairs that Tamr Core will evaluate in order to find the <x> pairs that meet the clause or blocking model. In other words, <y> represents the amount of compute work that Tamr Core will perform in order to identify <x> pairs.
<n> is the average number of blocks per record. Tamr Core uses the blocking model to group records into “blocks”, so that duplicate records belong to the same block. When evaluating record pairs, Tamr Core compares only records within the same block. A record can belong to more than one block; the amount of computation required increases when records belong to more than one block.

Guidelines for Interpreting Estimates

  • An efficient model should result in no more than 100 blocks per record.
  • For maximum efficiency, the model should result in no more than 10 blocks per record.
  • The number of comparisons should be less than the number of records multiplied by 100.

Guidelines for Improving Estimates

If an Estimate Counts job takes a long time to run, review your blocking model for these possible causes and solutions. Making any of these changes can reduce the estimated number of pairs or improve system efficiency without an unacceptable decrease in recall.

Data Quality Issues

Cause 1: An attribute that includes frequently-occurring default values or mock data increases estimates.

For example, 1 million records have an identical address value of “123 My Address St” in your 10 million record dataset. If your blocking model creates a pair when two records match on “address”, the system attempts to create 1 trillion pairs.

Solution 1: Use your data knowledge to review the blocking model and replace any attributes that rely on default filler values.

Cause 2: Generating record pairs within a source dataset that is already free of duplicates affects efficiency.

Solution 2: By default, Tamr Core generates pairs within each source dataset as well as across the different source datasets. Use your data knowledge to decide whether you can exclude one or more source datasets from intra-dataset pair generation. See Excluding Pair Generation within a Dataset.

Similarity Threshold Setting

Cause: Low thresholds for determining similarity increase estimates.

Solution: For your first Estimate Pairs job, set a high similarity threshold for each term to get a baseline of how long the job takes to run and the resulting estimate counts. Then, incrementally lower the similarity thresholds in one or more terms to include more pairs.

Tokenizer Selection

Cause 1: Selecting a tokenizer other than Default increases processing time.

Solution 1: Limit use of Bigram or Trigram to one term in a blocking model and to attributes with a limited maximum length. You can use a small data subset to evaluate the results of using Default compared to Bigram or Trigram before proceeding to the full dataset.

Cause 2: Selecting Bigram or Trigram on both the Schema Mapping and Pairs pages affects efficiency.

Solution 2: The machine learning tokenizer you specify for an attribute on the Schema Mapping page affects system training after pairs are generated (that is, after experts identify matching and non-matching pairs). It does not affect how the system generates pairs. Try using Bigram or Trigram for machine learning and Default on the Pairs page for the same attribute, and then adjust the similarity threshold to maximize for speed and recall.

Excluding Pair Generation within a Dataset

You can choose to exclude certain source datasets from being searched for matching and non-matching pairs. When you exclude a dataset, record pairs are not generated from within that source, only between other sources. For example, if a dataset is known to be free of duplicate records, you can indicate that the blocking model should exclude that dataset.

To exclude pair generation within a dataset:

  1. Navigate to the Pairs page.
  2. Select Manage pair generation.
  3. Select Open exclusions.
  4. Select + add source and choose a dataset to exclude.

Generating Record Pairs

Note: After making any changes to your blocking model, be sure to re-estimate record pair counts before you generate record pairs.

To generate record pairs:

  1. Navigate to the Pairs page.
  2. Select Manage pair generation.
  3. Select Generate Pairs. See Monitoring Job Status.

Both curators and verifiers assign and verify record pairs in the mastering project workflow.


Did this page help you?