Fabasoft object address reservation during COLD data imports

Last update: 4 August 2017

Information

Fabasoft COLD is framework to import data to a Fabasoft Folio domain from a datasource like SQL databases or CSV text files. Data from different data tables can be combined by key attributes and objects can be linked together. The data imports are defined in the object class "Data Import" (FSCCOLD@1.1001:DataImport).

Fabasoft COLD is designed to bulk import big amounts of data. The Data Import object class uses two properties to improve import performance:

• Number Of Records for Commit
• Number Of Threads

To optimize the time to create new objects, COLD preallocates COO addresses for new objects. The preallocation size of object addresses uses this formula:

Preallocated object addresses = Number Of Records for Commit * Number Of Threads * 2

For example, if you set the COLD import to 150 records to commit (bulk size) and 10 threads, COLD will preallocate blocks of 3000 object addresses. If the Fabasoft Kernel does not need all the object addresses it preallocated for running the data import, it will use these addresses for other new objects of that COO Store. It will also reuse the left preallocated addresses for the next COLD data import.

Special attention using COLD on a scheduled and/or reoccurring basis

For reoccurring (e.g. scheduled) COLD data imports, please consider the following special behaviours:

• COLD uses the object placement of the imported object classes. For performance reasons, if a preallocated address block is full during the import, COLD will preallocate a new address block in the same COO-Store as the last block was assigned. This may lead to uneven allocation if more than one COO Store is used in the placement. Please regularly check free object addresses in the COO Stores.
• On shutdown of the Fabasoft Kernel, the Kernel saves the preallocated address block in the COMMITDIR cache directory. If you delete the COMMITDIR directory, the Kernel will forget about preallocated addresses and will start a new block after the forgotten block. This may fill up COO Stores earlier than expected. Do not delete the COMMITDIR directory on a server using COLD data imports.
• In Fabasoft Folio versions below Fabasoft Folio 2014, backlink objects do not observe object placement, but will be created in the same COO Store as the dependend object. Therefore, consider an object address buffer in the COO Stores you import data. We recommend an address buffer of 1 to 2 million object addresses.
• If a COO Store reaches the address limit you considered to use as buffer, create new COO Stores and assign the object placement in the object classes.