Note: The ending slash is optional for all JDBC parquet exports across all cloud providers.

Describes the target Tamr datasets and their primary keys in order to ingest multiple query results at a time. Useful in cases where you have hundreds of tables to ingest, for example, research assays.

Descriptions of this object's keys follow.

Name of the dataset in Tamr Core.

Note: If present, this list does not update the policyIds of a dataset that already exists and is being updated.

The individual field that uniquely identifies each record in the source. Optional.

If left blank, Tamr Core uses the `TAMRSEQ` column as the primary key. See Data Import in Using the Core Connect API.

SQL query used to retrieve data from a JDBC source.

When set to true, Core Connect imports services metadata. Defaults to false when left blank. See Core Connect API Example Requests for an example.

When set to true, Core Connect retrieves the metadata for the dataset stored in the Snowflake source. Defaults to false when left blank. See Core Connect API Example Requests for an example.

INSERT INTO <final tablename> FROM (SELECT * FROM <tamr staging table> WHERE <condition>)

Optional. For a data source in Snowflake, retrieves metadata. See Adding a Metadata Property. For example:

“metadataConfig” : {
 “query”: “select COLUMN_NAME, TAG_NAME, TAG_VALUE from snowflake.account_usage.tag_references where OBJECT_NAME = '<import table>'”,
 “attributeColumn”:”COLUMN_NAME”,
 “keyColumn”:”TAG_NAME”,
 “valueColumn”:”TAG_VALUE”,
}

Contains the following keys:

• query: SQL query identifying the attribute, key, and value columns in the dataset to import.
• attributeColumn: the name of the column in the query that indicates the ingested dataset attribute.
• keyColumn: the name of the column in the query for the metadata property’s key.
• valueColumn: the name of the column in the query for the metadata property’s value.

INSERT INTO <final tablename> FROM (SELECT * FROM <tamr staging table> WHERE <condition>)

• When set to false (default), records from the imported file are added to the target dataset.
• When set to true, all records are deleted (truncated) from the target dataset before the file is imported.

If S3 Access Key / Secret Keys are not provided, or role assumption via the awsRoleARN key is not used, Core Connect will fall back to the AWS Default Credentials lookup chain as described in the Working with AWS Credentials documentation.

Role session name is fixed as “tamr.connect” and the session duration is fixed to 1 hour.

See encryptionConfig Object for descriptions of its keys.

The number of parallel threads running the export job.

datasetName

Name of the dataset in Tamr Core.

string

policyIds

Optional. An array of the authorization policies that will include the new Tamr Core dataset as a resource.

Note: If present, this list does not update the policyIds of a dataset that already exists and is being updated.

array

primaryKey

Set of fields uniquely identifying each record in the source.

string

profile

If set to true, Tamr Core adds a job to profile the new dataset after ingest to the queue. Set to false to profile the dataset at another time. Defaults to false.

Boolean

recipeId

Optional. To add a dataset to a specific project, you can supply the recipeId to associate the dataset with.

integer

resourceConfig

See resourceConfig Object for descriptions of its keys.

Object

truncateTamrDataset

Allows you to include imported data additively or destructively.

• When set to false (default), records from the imported file are added to the target dataset.
• When set to true, all records are deleted (truncated) from the target dataset before the file is imported.

Name of the character set used in the file. Defaults to UTF-8. See Character Sets table, Name column for more information.

List of characters to use to automatically determine the column delimiter when the column delimiter is not known.

Specifies a custom header line.

When the column delimiter is not known, set this key to true so that Core Connect will analyze the dataset to detect the delimiter. The charactersForDetection key provides a hint for what characters to consider.

Limit of characters to read in a single field. Used to avoid out of memory errors in case of invalid delimiters. Defaults to 4096, with a maximum of 40Mb.

Value used to separate values in a field into an array. For example, the following color values are ingested as an array of strings when you set the multiValueDelimiter to a pipe (|) character.

sku, colors
"1001232", "blue|red|green"

• columnDelimiter
• escapeCharacter
• quoteCharacter
• recordSeparator

For more information, see resourceConfig Object.

In the deltaConfig object, you specify values for the following keys:

• fromVersion (int) The starting version of the Tamr Core dataset used in the delta export.
• toVersion (int)
  The ending version of Tamr Core dataset used in the delta export.

Example:

{"Addr":"Address"}

In this example, the Addr field in the Tamr Core dataset will be exported as the Address field.

datasetName

Name of the dataset in Tamr Core.

string

policyIds

Optional. An array of the authorization policies that will include the new Tamr Core dataset as a resource.
Note: If present, this list does not update the policyIds of a dataset that already exists and is being updated.

array

primaryKey

Set of fields uniquely identifying each record in the source.

string

profile

If set to true, Tamr Core adds a job to profile the new dataset after ingest to the queue.
Set to false to profile the dataset at another time. Defaults to false.

Boolean

recipeId

Optional. To add a dataset to a specific project, you can supply the recipeId to associate the dataset with.

integer

resourceConfig

The resourceConfig object contains the following key for Avro files:

• ignoreComplexTypes (Boolean)

Avro supports six complex data types: Records, Enums, Arrays, Maps, Unions, and Fixed.

• When set to true, Core Connect ignores any complex types detected in Avro.
• When set to false, Core Connect returns an error when complex types are detected in the Avro file.

object

truncateTamrDataset

Allows you to include imported data additively or destructively.

• When set to false (default), records from the imported file are added to the target dataset.
• When set to true, all records are deleted (truncated) from the target dataset before the file is imported.

flattenEnable

Defines the type applied to imported values: array of values or values.

• When set to false (default), the type for every attribute is an array of values. For example: {"TAMRSEQ":["1"],"PK":["305d411f"],"surname":["matei"]}.
• When set to true, the type for every attribute is a value. For example: {"TAMRSEQ":"1","PK":"305d411f","surname":"matei"}.