Task and Transformation
The Data processing is composed of two components which can be used either separately or seamlessly reference each others capabilities:
- Task Hierarchy - TH
The aim of the task launching mechanism is to divide long-lasting processes into smaller, more manageable functional blocks. It means, that the User may divide the usually complicated and labor-consuming executing process into several smaller tasks. Such an attitude makes an executing process organized, clearer and easier to be controlled. If any of child tasks fails, it is possible to restore the process operation starting directly from the point of failure where a problem occurred. It results in increased efficiency and reduced execution time as it does not require returning to the starting point. See also: Tasks
- Data transformation - DT
Data transformation component enables designing record flows and data modifying tasks (so called "blocks"). A single flow is called a "processing package” which is a graphic description of particular data transformation steps. A package complexity level may take the form of both a very simple directed graph and a complex, pipeline-oriented, parallelized transformation schema comprising data from relative data base management systems (RDBMS), Web services or in-memory indexed files.
For each package, it is possible to define a set of parameters: paths to I/O data files, logical data base names or user-defined execution arguments.
To build a complete and operating solution, an external triggering element, which will launch tasks / packages at the appropriate moment and in the appropriate way, is also necessary. Various hosting options may be used:
- console programs: TransformationRunner.exe and TransformationService.exe, called e.g. from the Windows Scheduler,
- in a separate application: VSoft Scheduler,
- custom control module used in the particular deployment in the backend system.
The following figure shows an exemplary processing package, which prepares a simple report file on the basis of two relative data-bases.
The following figure shows an exemplary (Task Hierarchy TH) task dependency diagram, having two independent processes, each having two starting points (each launched as a result of detecting two cohesive groups of input files).