What is the Data processing
The Data processing enables integration with other systems through data exchange and mass processing of information sets.
The Data processing consists of 2 main components, one for the flow of records (Data Transformation) and the other for the flow, control and management of tasks (Task Hierarchy).
Using a graphic interface, the designer creates and modifies packages that transform large data sets from multiple sources, and also builds a hierarchy of tasks that control the subsequent steps of the data processing.
The Data processing allows you to:
- integrate with other systems by receiving and sending data, both through various file formats and integration with the database,
- build data processing packages in a graphical way, divide complex packages into subpackages,
- run packages in the graphical environment, keep up with the processing steps of the package and identify bottlenecks of processing on a current basis,
- design logic of packages containing, among others:
- change of the structure of the processed data in the subsequent steps of processing,
- combining data from different data sources,
- operations on hierarchical data structures containing sub-structure collections,
- integrate systems on both sides during data processing by calling external systems web service as one of the steps of processing packages or publishing packages as a web service for their invocation by external systems,
- handle erroneous data with the determination of acceptable levels of error, which may suspend further processing or not,
- automatically handle technical errors (e.g. automatic resuming of processing in the event of short-term loss of connections with data sources),
- manually resume the further processing in the event of occurrence and removal of a technical error which cannot be handled automatically (e.g. lack of disk space),
- build complex processing networks, within which individual processing steps can be started in parallel or started after occurrence of specific events,
- profile the processed data at any stage of the processing and build automatic statistics of the amount of data flowing through the individual steps of the package.