Data Integration Jobs

The following sections provide information about the parameters and attributes of jobs that work with Data Integration platforms and services.

AWS Glue Job

The following table describes AWS Glue job attributes.

Attribute

Description

Connection profile

Defines the connection profile for the job.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Spaces

Glue Job Name

Defines the AWS Glue job name that you want to run. After you create an AWS Glue pipeline, it is saved as a jobname and you can run it.

Glue Job Arguments

Determines whether to add arguments to the AWS Glue job.

Arguments

Defines the AWS Glue job runtime parameters

Format: {"--myArg1": "myVal1", "--myArg2": "myVal2"}

Status Polling Frequency

(Optional) Determines the number of seconds to wait before checking the status of the job between intervals.

Default: 15

Azure Data Factory Job

Azure Data Factory is a cloud-based ETL and data integration service that allows you to create data-driven workflows to automate the movement and transformation of data.

The following table describes the Azure Data Factory job type attributes.

Attribute

Description

Connection profile

Defines the connection profile for the job.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Spaces

  • Variable Name: %%AZURE-ACCOUNT

Resource Group Name

Determines the Azure Resource Group that is associated with a specific data factory. A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group.

Data Factory Name

Determines the name of the Azure Data Factory that contains the pipeline you want to run.

Pipeline Name

Determines which data pipeline runs when the Control-M job is executed.

Parameters

Defines specific parameters in JSON format that are passed when the Data Pipeline runs.

Format: {"var1":"value1", "var2":"value2"}

Status Polling Frequency

(Optional) Determines the number of seconds to wait before checking the status of the job between intervals.

Default: 15

Azure Synapse Job

Azure Synapse Analytics enables you to perform data integration and big data analytics.

The following table describes Azure Synapse job parameters:

Attribute

Description

Connection profile

Determines which connection profile is used to connect to the Azure Synapse workspace.

Pipeline Name

Defines the name of a pipeline that you defined in your Azure Synapse workspace.

Parameters

Defines pipeline parameters to override when the job runs, defined in JSON format as pairs of name and value.

Format: {"param1":"value1", "param2":"value2"}

For no parameters, specify {}.

Status Polling Interval

(Optional) Defines the number of seconds to wait before checking the status of the job.

Default: 20 seconds

Boomi Job

The Boomi job enables you to integrate Boomi processes with your existing Control-M workflows. The following table describes Boomi job parameters:

Parameter

Description

Connection profile

Determines which connection profile is used to connect to the Boomi endpoint.

Atom Name

Defines the name of a Boomi Atom associated with the Boomi process.

Process Name

Defines the name of a Boomi process associated with the Boomi Atom.

Status Polling Frequency

Determines the number of seconds to wait before checking the status of the job between intervals.

Default: 20 seconds

Tolerance

Defines the number of API call retries during the status check phase. If the API call that checks the status fails due to the Boomi limitation of a maximum of 5 calls per second, it will retry again according to the number in the Tolerance field.

Default: 3 times

Informatica Job

The Informatica job enables you to automate an Informatica workflow or tasks within the workflow, and define the parameters to pass to the workflow.

The following table describes the Informatica job type attributes.

Attribute

Description

Connection Profile

Defines the connection profile for the job.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Spaces

Variable Name: %%INF-ACCOUNT

Repository Folder

Defines the Repository Folder containing the workflow that you want to run.

Variable Name: %%INF-REP_FOLDER

Workflow

Defines the workflow that you want to run in Control-M for Informatica.

Variable Name: %%INF-WORKFLOW

Instance Name

Defines the specific instance of the workflow that you want to run.

Variable Name: %%INF-INSTANCE_NAME

OS profile

Enables you to specify an OS profile when running or rerunning an Informatica job.

Run Options

Defines options regarding the workflow task hierarchy.

Depth

Determines the number of levels within the workflow task hierarchy that is used to select workflow tasks.

Default: 10

Variable Name: %%INF- DEPTH

Run

Determines whether to run the whole workflow, start from a specific task, or run a single task as follows:

  • Run the whole workflow: Runs the whole workflow.

  • Start from task: Starts the workflow from the task that you specify.

    Variable Name: %%INF- START_FROM_TASK

  • Run single task: Runs the task that you specify.

    Variable Name: %%INF-RUN_SINGLE_TASK

Parameters

Determines an array of parameters that is passed to the workflow.

Each parameter is comprised of the following:

  • Scope: Defines the scope of the parameter in an array definition.

  • Name: Defines the parameter name in an array definition.

  • Value: Defines the parameter value in an array definition.

Include workflow events log in Job output

Determines whether to include the workflow event log in the job outputClosedA tab in the job properties pane in the Monitoring domain that shows the output of a job, which indicates whether a job ended OK, and used, for example, with jobs that check file location.

Include detailed error log for failed sessions

Determines whether to include a detailed error log for a workflow that failed.

Get session statistics and log

Determines whether to retrieve session statistics and log messages.

Action on rerun

Determines which operation is executed when the workflow is in a suspended status, as follows:

  • Recover: Restarts suspended workflow from point of failure

  • Force restart: Restarts a suspended workflow from the beginning.

  • Force restart from a specific task: Restarts the suspended workflow that you define

    Variable Name: %%INF- RESTART_FROM_TASK

Workflow parameters file

Defines the path and name of the workflow parameters file.

Variable Name: %%INF-WORKFLOW_PARAMETERS_FILE

Informatica Cloud Services Job

Informatica Cloud Services jobs enable you to automate your Informatica workflows for multi-cloud and on-premises data integration through the Control-M environment.

The following table describes Informatica Cloud Services Job attributes.

Attribute

Description

Connection Profile

Defines the connection profile for the job.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Spaces

Variable Name: %%INF-ACCOUNT

Task Type

Determines one of the following task types to run on Informatica Cloud:

  • Mapping Task

  • Masking Task

  • PowerCenter Task

  • Replication Task

  • Synchronization Task

  • Linear Taskflow

  • Taskflow

Task Name

Defines the name of the task to execute on Informatica Cloud.

This parameter is not relevant for a taskflow.

TaskFlow URL

Defines the Service URL of the taskflow to execute on Informatica Cloud.

In Informatica Data Integration, you can obtain this Service URL through the Properties Detail option of the taskflow.

Input Fields

Defines input fields for a taskflow.

Format: input1=value1&input2=value2&input3=value3

Call Back URL

(Optional) Defines a publicly available URL to which to post the job status.

Verification Poll Interval

Defines the number of seconds between polls for job status verification.

IBM InfoSphere DataStage Job

The following table describes the IBM InfoSphere DataStage job type attributes.

Attribute

Description

Connection Profile

Defines the connection profile for the job.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Spaces

Variable Name: %%DataStage-ACCOUNT

For more information about creating a local connection profile for this job, see Creating a connection profile.

Project

Defines the Control-M for IBM InfoSphere DataStage project name.

Variable Name: %%DataStage-PROJECT

DataStage Job

Defines the Control-M for IBM InfoSphere DataStage job name.

Variable Name: %%DataStage-JOB_NAME

Job Invocation ID

Defines the Control-M for IBM InfoSphere DataStage job invocation ID.

Variable Name: %%DataStage-JOB_INVOCATION_ID

Parameters Type

Determines from where Control-M retrieves the parameters, with options as follows:

  • None: Does not retrieve parameters.

  • Server: Retrieves the parameters from the DataStage Job, and displays them in the Parameters list.

  • File: Retrieves the parameters from the file that you define in Parameters File.

  • Server and File: Retrieves the parameters from the file that you define in Parameters File and from the DataStage Job, and displays those from the job in the Parameters list.

Parameters

Displays the parameters and their values from the DataStage Job when Server or Server and File is selected from Parameters Type.

Parameters File

Defines the IBM InfoSphere DataStage parameter file.

Variable Name: %%DataStage-PARAMS_FILE

More Options

Opens more options.

Limits

Defines limits on the job.

Stop stages after <value> Rows

Defines the maximum number of rows that the job can contain. Control-M stops the stages after the maximum is reached.

Variable Name: %%DataStage-MAX_ROWS

Abort job after <value> Warnings

Defines the maximum number of warnings about the job. Control-M aborts the job after the maximum is reached.

Variable Name: %%DataStage-MAX_WARNINGS

Job Output

Determines the type of information that goes in the output as follows:

  • Info: Writes Info log entries to Control-M for IBM InfoSphere DataStage job Sysout.

    Variable Name: %%DataStage-OUTPUT_INFO

  • Warnings: Writes Warning log entries to Control-M for IBM InfoSphere DataStage job Sysout.

    Variable Name: %%DataStage-OUTPUT_WARNING

  • Fatal: Writes Fatal log entries to Control-M for IBM InfoSphere DataStage job Sysout.

    Variable Name: %%DataStage-OUTPUT_FATAL

  • Reject: Writes Reject log entries to Control-M for IBM InfoSphere DataStage job Sysout.

    Variable Name: %%DataStage-OUTPUT_REJECT

  • Started: Writes Started log entries to Control-M for IBM InfoSphere DataStage job Sysout.

    Variable Name: %%DataStage-OUTPUT_STARTED

  • Reset: Writes Reset log entries to Control-M for IBM InfoSphere DataStage job Sysout.

    Variable Name: %%DataStage-OUTPUT_RESET

  • Batch: Writes Batch log entries to Control-M for IBM InfoSphere DataStage job Sysout.

    Variable Name: %%DataStage-OUTPUT_BATCH

Run Options

Defines run options for the job.

Run in restart mode

Runs the Control-M for IBM InfoSphere DataStage job in restart mode.

Variable Name: %%DataStage-RESTART_SEQUENCE

Reset job before run

Resets the Control-M for IBM InfoSphere DataStage job before the job runs.

Variable Name: %%DataStage-RESET_JOB

Talend Data Management Job

The Talend Data Management Job enables the integration of data management and data integration tasks or plans from Talend with your existing Control-M workflows.

The following table describes Talend Data Management job attributes.

Attribute

Description

Connection profile

Defines the connection profile for the job.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Spaces

Task/Plan Execution

Determines one of the following options for execution in Talend:

  • Execute Task

  • Execute Plan

Task Name /
Plan Name

Defines the name of the Talend task or plan to execute, as defined in the Tasks and Plans page in the Talend Management Console.

Parameters

(For a task) Defines specific parameters to pass when the Talend job runs, defined as JSON pairs of parameter name and value. All parameter names must contain the parameter_ prefix.

Format: {"parameter_param1":"value1", "parameter_param2":"value2"}

For no parameters, specify {}.

Log Level

(For a task) Determines one of the following levels of detail in log messages for the triggered task in the Talend Management Console:

  • Information — All logs available

  • Warning — Only warning logs

  • Error — Only Error logs

  • Off — No logs

Bring logs to output

(For a task) Determines whether to show Talend log messages in the job output.

Values: checked|unchecked

Default: unchecked

Task Polling Intervals /

Plan Polling Intervals

Determines the number of seconds to wait before checking the status of the triggered task or plan.

Default: 10 second