ETL and beyond
Data processing is a platform that has been created in consequence of filling many practical gaps of typical ETL (Extract, Transform, Load) class packages and adding several quite unique elements, which simplify and thus enable certain tasks to be understood and even modified directly by "business" analysts, task that, in the traditional approach, required a programmer to do the translation.
Therefore, it may be stated that Data processing exceeds the ETL class tool definition by:
- readability of defined processes - which is expressed in:
- a data structure used: a tree with one-to-many relations,
- a possibility of embedding/autogenerating/downloading documentation almost in each element,
- adjustable and readable so called "autodocumentation" generated for the vast transformation logic,
- a possibility of searching definitions in terms of e.g. transformation operations, database objects,
- a possibility of spreading computational and I/O load, and shaping efficiency quite freely:
- a natural approach to concurrent tasks of the control module - forks and joins in a dependency net,
- a possibility of linking Data processing blocks using synchronous "hard" links or asynchronous queues,
- manually assigning worker threads to fragments of processing pipelines,
- detailed, live reports of particular elements' efficiency, database session statistics, etc.
- numerous facilities for a designer:
- auto-completing syntax and semantics of "data" calculation expressions, SQL scripts, regular expressions,
- handy browsing through big trees of data structures - by names and comments,
- statistical reports of elements' performance, database operations times, similar errors and causes of rejecting data records,
- numerous drag-drop operations between data schema tree, plain text, and external editors (e.g. from MS notepad and excel),
- recording definition change history in the central repository (in the so called Version server) together with multi-access control,
- less typical but very useful functionalities:
- loading data from any text files according to a specification in the form of trees made up of regular expressions,
- resuming an aborted task hierarchy starting from the moment of aborting, with a possibility of definition and/or dependency change in the course of action,
- live graphic analysis of logical dependencies in someone else's SQL scripts,
- automated profiling of unknown data sets (RDBMSs, file sets),
- generating random values on the basis of regular expressions, given or sampled distributions,
- configurability, including:
- adjusting and even replacing almost any platform element for particular deployment needs,
- placing large fragments of the platform in another tool (host application) together with extending/limiting available functions,
- price:
- a possibility of fairly unrestricted adjusting a licensed functionality to client's needs.
Threads – Data processing
- By means of a package, data is retrieved from one or many sources.
- Each record is appropriately transformed, joined, divided, etc. and it finally gets to one (or more) of target destinations.
- Most transformations can be done by a freely defined number of threads.
Tree structures
One of the main advantages which make Data processing different from the other tools available on the market, is operating on data trees (as opposed to simple, flat structures).
- A basic advantage of a tree structure is an unusual easiness of handling individual data records with apparent division into parent and child elements.
- A tree node is a logical record composed of fields and child records which may be lists (represent a one-to-many relationship). Such structures enable independent transformation of logical record trees without the need to split them and to lose bindings to their parents.
- It also facilitates natural integration with other "tree-oriented" data formats (e.g. XML).
For more about the data schema, see Data schema in the Mappings.