Dictionary as a cache
A function of storing elements in server memory (the elements are also usually saved in a database) is often very useful for performance reasons when processing large data sets. Such solution leads to accelerating the operation of a transformation package many times, however it should be well thought out and several elements should be applied properly.
- A dictionary is any table, not necessarily a database table (with columns of a string type) which may be both modified and read while a package is running. If you make sure that a dictionary content verification is located in the same processing thread in phase in which a dictionary is completed, you will get an effective cache mechanism.
It can be seen in the following example:
- Let's assume we want to differentiate elements which have already been transformed (e.g. saved to a database) from the ones which have not been transformed yet. We need it for directing elements-duplicates to another path (e.g. for report only) with higher transformation performance (e.g. without saving to a database).
- A minimal package executing such task may look like the one shown in the figure below.
A block cache marking declares a dictionary named e.g. "cache1", which key and value is "key". It is also the name of the only field generated by "Some data source" which is copied to a new record and cut in the pair of blocks Prepare record for cache and Split record for cache.
Expressing classification is built as follows:
dic.cache1__key( ToString(key)) is_empty
10 000 elements which are integer keys of the range e.g 0-1000 are generated here. On average, 10 times more records should be directed to the path "Quick path - mark in cache" than to a path long transformation. It is important that a synchronous connected subgraph declares only one transformation thread, otherwise duplicates of keys may appear in the "slow" stream.