Incremental Loads

When a data source does not support change data capture views from which incremental changes may be loaded, Ajilius supports the nomination of ascending value columns to control extracts.

These columns may be an ID field for capturing new rows, or perhaps a date-last-modified column where edits are to be captured.

This feature is available from the Column Editor using the Change Data Management block, highlighted in the diagram above.

Change Data Columns are only available for data sources which natively support a WHERE clause. These are mostly relational databases. Change Data Columns are not supported for file-based data sources such as delimited files and spreadsheets.

When you check the box marked Change Data Column, Ajilius will hold the most recent value of this column that has been loaded from the source. The next time an Extract is run, a WHERE clause will be appended to the query for values that are greater than the retained value.

Should you ever need to roll back this value, use the Reset Controls option from the Schedule menu:

On choosing this option, you will be shown a list of all tables for which control values are held by Ajilius. Select one or more tables from this list, and their control values will be rolled back to the previous value, allowing an entire extract to be re-run if necessary.

Note that only the most recent value is held, you can only roll-back for one iteration.