File Watcher (ctmfw)
The ctmfw (Control-M File Watcher) utility, enables you to detect the successful completion of a file transfer activity or a file creation or file deletion. You can use it before activating a job or before performing a task, such as adding or deleting eventsAn entity that creates a sequence relationship between jobs by enabling the successor job to execute after the predecessor job has executed, which is dependent upon a created file.
The Control-M Fire Watcher utility does not detect files with a modify time older than 50 years.
The ctmfw utility runs as a process on the Agent host computer. The process waits for the creation or deletion of a specified file. The ctmfw utility does the following:
-
File transfer activity: When the file is detected, the job continues to monitor the size of the file. When the file reaches a specified minimum size and does not increase in size for a specified period of time, the File Watcher utility either finishes with a status of OK or executes a specified DO action. DO actions can consist of adding or deleting events or executing a command.
-
File creation: The file size is ignored if a wildcard is specified as part of the file name unless the mon_size_wildcard parameter is set to Y.
-
File deletion: ctmfw must first detect the existence of the file before it can detect its deletion.
The ctmfw utility runs from the command line to detect a single file or multiple files. For more information, see Watching a Single File and Watching Multiple Files.
There are two usages for this utility:
-
Service: (Windows only) As a service, ctmfw takes its parameters (rules) during startup from the rull.dat file whose full path name is specified in <Agent>\data\ctmfw.cfg. For more information, see Usage as a Service.
-
Running as a utility: ctmfw is invoked from the command line. Rules can be provided in the command line or by a rule file. You can watch a single or multiple files. For more information, see Watching a Single File or Watching Multiple Files.
To allow other users to run ctmfw, they must have write permissions to $CONTROLM and the CONTROLM environment variable must be set. For more information, see Running Control-M/Agent utilities as other users. For details about ctmfw processing and return codes issued after detecting if a file is created or deleted in the specified time frame, see ctmfw utility processing and Return codes.
Usage as a Service
As a service, ctmfw takes its parameters (rules) during startup from the rull.dat file whose full path name is specified in <Agent>\data\ctmfw.cfg on Windows.
To change one or more rules, change the contents of the rull.dat file. The rull.dat file provided with Control-M/Agent is a sample file and must be changed to reflect your requirements. The full path name to the ctmfw.cfg configuration file must be specified under the following Windows registry key that is generated automatically by the installation script:
HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\CONTROL-M/FileWatcher\SYSPRM\File Watcher Configuration File
The default value for this key is <CONTROL-M/Agent_install_directory>\DATA\ctmfw.cfg
You need administrative permissions to update the registry key. It is recommended that this default value not be changed.
The configuration file must contain the following line:
-input <ruleFileName>
The variable <ruleFileName> is the full path name of a rule file containing the File Watcher rules.
The following example shows a sample rull.dat file:
INTERVAL 5
FROM_TIME 0001
MIN_SIZE 50
MIN_DETECT 5
WAIT_TIME 2
ON_FILEWATCH NONEXIST CREATE 10 3 1
THEN
DO_CMD "BAD FILE WAS CREATED IN 1 MINUTE"
ELSE
DO_CMD "GOOD FILE WAS NOT CREATED IN 1 MINUTE"
END_ON
The FileWatcher service running under the local system account cannot detect network resources (files located on remote systems). If you want the FileWatcher to detect network resources, configure the FileWatcher service to run under a regular user account instead of Local System Account.
When running as a service, ctmfw generates an execution log file. This file is saves in the Agent proclog directory under the following name:
U_CTMFW_<process_id>.log
The following example shows a Trace file output:
2017/03/10 13:04:24 182 FW:set INTERVAL=3
2017/03/10 13:04:24 182 FW:set MIN_SIZE=4
2017/03/10 13:04:24 182 FW:set FROM_TIME=0909
2017/03/10 13:04:24 182 FW:ctmfw:command line 'ON_FILEWATCH tst CREATE'(arg#=3).
id=1.
2017/03/10 13:04:24 182 FW:ctmfw:command line 'ON_FILEWATCH prd CREATE 0 0 1 NOW
10'(arg#=8)id=2
2017/03/10 13:04:24 182 FW:ctmfw:command line 'ON_FILEWATCH abc DELETE 0 0 1 NOW
10'(arg#=8)id=3
2017/03/10 13:04:24 182 FW:File 'test' exists, its current size is 265.
id=1.
2017/03/10 13:04:24 182 FW:File 'test' has reached the minimum size of 4.
size=265 bytes id=1.
2017/03/10 13:04:24 182 FW:File 'abc' does not exist. id=3.
2017/03/10 13:04:36 182 FW:File transfer was completed. The size of file 'test'
is 265. id=1.
2017/03/10 13:05:09 182 FW:Executing:< dir >
2017/03/10 13:05:27 182 FW:File 'prd' was not CREATED within the time limit.
id=2.
2017/03/10 13:05:27 182 FW:File prd will be scanned at 1315. id=2.
2017/03/10 13:05:27 182 FW:File 'abc' was not DELETED within the time limit.
id=3.
2017/03/10 13:05:27 182 FW:File abc will be scanned at 1315. id=3.
2017/03/10 13:05:30 182 FW:File prd, is out of time window. next time:1315, id=2.
2017/03/10 13:05:30 182 FW:File abc, is out of time window. next time:1315, id=3.
2017/03/10 13:15:01 182 FW:File prd, entered the time window from '1315' for
monitoring, id=2.
2017/03/10 13:15:01 182 FW:File abc, entered the time window from '1315' for
monitoring, id=3.
By default, logs in the proclog directory are retained for 1 day. If the maximum days to retain output files parameter is set to a number higher than 1, logs are retained for the number of days specified in that parameter.
Watching a Single File
This procedure describes how to watch a single file with the ctmfw File Watcher utility. Rules can be provided in the command line or by a Rule file.
Begin
-
From a command line, type the following:
ctmfw FILE (absolute path)
-
Enter the required parameters.
Copy< mode (CREATE|DELETE)> Default: CREATE
< minimum detected size <number>
[' '|Bytes(B)|Kilo(K)|Mega(M)|Giga(G)] >Default:0
< interval between file search (seconds) > Default: 60sec
< interval between filesize comparison iterations (seconds) > Default: 10sec
< number of iterations while the size is static > Default: 3 iterations
< time limit for the process (minutes).> Default: 0 (no time limit)
Effective while the file does not exists or,
the file size is static and the minimum size
was not reached >
< monitor file size , minimal and maximal age, when wildcard is used > Default: N
< starting time for detecting files (HHMM or YYYYMMDDHHMM > Default: NOW
< absolute stop time (HHMM or YYYYMMDDHHMM > Default: +0000 ( No stop time )
< minimal age of file (modified time)
format:xxxxYxxxxMxxxxDxxxxHxxxxMin > Default: NO_MIN_AGE
< maximal age of file ( timestamp monitoring )
format:xxxxYxxxxMxxxxDxxxxHxxxxMin > Default: NO_MAX_AGEAll parameters must be assigned a value, even if that value is zero. If only six values are specified, the default value for mon_size_wildcard is used. If five parameters are specified, default values for wait_time and mon_size_wildcard are used.
ctmfw /home/watchedfile.txt CREATE 100 10
is resolved by using default values for mon_int, min_detect, wait_time, and mon_size_wildcard as follows:
ctmfw /home/samplefile.txt CREATE 100 10 10 3 0 N
ctmfw Parameters
The following table lists the ctmfw utility rule file parameters:
Parameter |
Description |
---|---|
FILE |
Determines the path of the file to be detected. The file name can include wildcard character * to represent any number of characters (including no characters) or ? to represent any one character. The path and file name must not exceed 214 characters. |
mode |
|
minimum detected size |
Determines the minimum file size in bytes. This parameter is ignored if the FILE parameter contains wildcards (unless the monitor file size when wildcard is used parameter is set to Y) or if the mode parameter is set to DELETE. Default: 0 (any size detected). |
interval between file searches |
Determines, in seconds, the interval between successive attempts to detect the existence/deletion of a file. Default: 60 |
interval between filesize comparison iterations |
Determines, in seconds, the interval between attempts to monitor the size of a file after it is detected. This parameter is ignored when using wildcards in FILE or when using DELETE mode. Default: 10 |
number of iterations while size is static |
Determines the number of attempts at monitoring the file size, where the size remains static and greater than or equal to minimum detected size, (indicating successful creation of the file). This parameter is ignored when using wildcards in FILE or when using DELETE mode. Default: 3 |
time limit for the process |
Determines, in minutes, the maximum amount of time taken to run the process without detecting the file at its minimum size (CREATE), or detecting its deletion (DELETE). If the file is not detected/deleted in this specified time frame, the process terminates with an error return code, as described in Return codes. Default: 0 (no time limit). |
monitor file size when wildcard is used |
Determines whether the file size should be monitored if the filename contains wildcards. This parameter is ignored if the filename does not contain a wildcard. Valid values:
If this parameter is set to Y and more than one file matches the specified mask, the ctmfw utility selects the first file that is detected, monitors its file size, and ignores all other matching files. |
starting time for detecting files |
Determines an absolute time at which the utility starts monitoring the file. For example, 200712061400, means that at 2 PM on December 6th, 2007 the File Watcher utility will start watching the file. Alternatively, you can use the HHMM format, in which case the current date is used. |
absolute stop time |
Determines an absolute time at which the file is no longer watched. For example, 200702061400, would mean that at 2 PM on February 6th, 2007 the File Watcher utility will stop watching the file. Alternatively, you can use the HHMM format, in which case the current date is used. |
maximal age of file |
Determines the maximum amount of time that can pass since the file you want to watch was last modified. For example, 2y3d5h means that after 2 years, 3 days, and 5 hours has passed, the file will no longer be watched. Entering a value of 2H10Min, means that after 2 hours and 10 minutes has passed, the file will no longer be detected. This parameter is ignored if the mode parameter is set to DELETE. Default: 0 If using the default value, files older than 50 years cannot be detected by the utility. If you want to detect files older than 50 years, you must define the number of years, such as 51y. |
minimal age of file |
Determines the minimum amount of time that must have passed since the file you want to watch was last modified. For example, 2y3d5h means that 2years, 3 days, and 5 hours must pass before the file will be watched. Entering a value of 2H10Min, means that 2 hours and 10 minutes must pass before the file will be detected. This parameter is ignored if the mode parameter is set to DELETE. Default: 0 |
Watching Multiple Files
This procedure describes how to watch multiple files with the ctmfw File Watcher utility.
Begin
-
Type the following command to invoke the ctmfw utility for multiple files:
ctmfw -input <ruleFileName>
The variable <ruleFileName> is the complete path name of the file containing the definitions for each file to be detected. The .Rule file contains the following sections:
-
Global parameters, whose default values apply to all the files in the rule file. For more information, see ctmfw Rule File Global Parameters.
-
ON_FILEWATCH statements identifying which files to detect, specific criteria for each file, and the action to take upon detection or non-detection. Any number of ON_FILEWATCH statements can appear in a Rule file. For more information about ON_FILEWATCH parameters, see File Watcher (ctmfw).
If any mandatory parameter is omitted from a Rule file, the default value for that parameter is used. Parameters entered for ON_FILEWATCH statements override the default values.
-
ctmfw Rule File Global Parameters
The following table lists the ctmfw Rule file global parameters:
Parameter |
Description |
---|---|
CYCLIC_INTERVAL |
Determines, in minutes, the interval between multiple operations of detecting the file. This interval must be greater than the value for WAIT_TIME. If the cyclic_interval is 0, only one attempt to detect the file will be performed. Default: 0 |
FROM_TIME |
Determines the starting time for detecting all the files (default FROM_TIME). Used with WAIT_TIME to identify the time frame for detecting and monitoring the files. This parameter is expressed in 24-hour, hhmm format. Default: 0000 or Now |
INTERVAL |
Determines, in seconds, the Sleep interval between successive scans for all the files. This parameter replaces individual sleep_int and mon_int parameters for each file. Default: 10 |
MAX_AGE |
Determines the maximum amount of time that can pass since the file you want to watch was last modified.
This parameter is ignored if the mode parameter is set to DELETE. Default: 0 |
MIN_AGE |
Determines the minimum amount of time that must have passed since the file you want to watch was last modified. For example, 2y3d5h means that 2years, 3 days, and 5 hours must pass before the file will be watched. This parameter is ignored if the mode parameter is set to DELETE. Default: 0 |
MIN_SIZE |
Determines whether the file size should be monitored if the filename contains wildcards. This parameter is ignored if the filename does not contain a wildcard. Valid values:
If this parameter is set to Y and more than one file matches the specified mask, the ctmfw utility randomly selects one matching file, monitors its file size, and ignores all other matching files. |
STOP_TIME |
Determines the absolute time at which the file is no longer watched. For example, 200702061400, means that at 2 PM on February 6th, 2007 the FileWatcher utility will stop watching the file. You can also use the HHMM format, which uses the current date, plus the HHMM entered. Default: 0 (meaning, no stop time) STOP_TIME can only be used as a global parameter. |
WAIT_TIME |
Determines, in minutes, the maximum time to run the process without detecting the file at its minimum size (CREATE), or detecting its deletion (DELETE). If the file is not detected/deleted in this specified time frame, the process terminates with an error return code, as described in ctmfw- return codes table below. Default: 0 (no time limit). |
ctmfw Utility Processing
If the file is detected and the size remains static within the time frame (CREATE) or the file has been deleted (DELETE), the DO commands in the THEN block are executed. If the file is not detected/deleted within the time frame, the statements following the ELSE block are executed.
CTMFW terminates when either all the files in the Rule file have been processed or a DO_OK/DO-NOTOK action is executed. If any ON_FILEWATCH statement contains a cyclic_interval, CTMFW only terminates on a DO_OK/DO-NOTOK action. If the cyclic_interval Global parameter is specified and not 0, CTMFW only terminates on a DO_OK/DO-NOTOK action or when the STOP_TIME was reached.
-
The sleep interval between succeeding scans must be 10 seconds.
-
If the ctmfw utility detects that the datafile.txt file in the /home/controlm directory is created in the specified time interval, then the command interpreter must execute the command to move the contents of the ~<controlm_owner>/ctm_server/datafile.txt file to~<controlm_owner>/ctm_server/workfile.txt.
-
If the ctmfw utility detects that the datafile.txt file in the ~<controlm_owner>/controlm directory is not created in the specified time interval then command that executed is "FILE WAS NOT CREATED".
-
When the ctmfw utility detects that the ~<controlm_owner>/ctm_server/tempfile.txt file is deleted, then the following command is executed "FILE WAS DELETED".
#******************************************************************
INTERVAL 10
ON_FILEWATCH ~<controlm_owner>\ctm_server\datafile.txt CREATE
THEN
DO_CMD move \ctm\datafile.txt \ctm\workfile.txt
ELSE
DO_CMD "FILE WAS NOT CREATED"
END_ON
ON_FILEWATCH \ctm\tempfile.txt DELETE
THEN
DO_CMD "FILE WAS DELETED"
END_ON
#*******************************************************************
Valid Actions
The following table lists the ctmfw valid actions:
If the file is detected and the size remains static within the time frame (CREATE) or the file has been deleted (DELETE), the DO commands in the THEN block are executed. If the file is not detected or deleted within the time frame, the statements following the ELSE block are executed. ctmfw terminates when all the files in the Rule file have been processed.
If an ON_FILEWATCH statement contains a cyclic_interval parameter, ctmfw will only stop monitoring a file on a DO_OK or DO_NOTOK action.
Action |
Description |
---|---|
DO_CMD <command> |
Execute a valid command under the command interpreter. Full path names are required for files. |
DO_EXIT [exit code] |
Terminate ctmfw with the user-defined exit code |
DO_NOTOK [exit code] |
Terminate an ON_FILEWATCH statement with status NOTOK. Exit code is optional and replaces the standard return code, as described in the ctmfw-return codes table below. |
DO_OK |
Terminate an ON_FILEWATCH statement with status OK. If there is more than one file in the Rule file, the result displayed is that of an AND algorithm. |
Return Codes
The return codes listed in the following table are issued by the ctmfw utility after detecting if a file is created or deleted in the specified time frame.
Return Code |
Description |
---|---|
0 |
File was successfully created or deleted (file arrived in the specified time frame and file size is above or equal to the minimum specified size). |
1 |
|
2 |
Indicates that the ctmfw request timed out. That is, the file was not detected in the specified time frame. |