The new Ajilius scheduler uses metadata to figure out the dependencies and run sequence of your ELT jobs.
These dependencies can currently break through custom code.
Here’s an example, contrived from a customer query earlier today:
In this query there is a dependency between the tables load.load_from_table and load.load_earlier_table. Unless load.load_earlier_table has been populated before processing load.load_from_table, then duplicate rows may be selected.
In the long term we will use a SQL parser to extract these dependencies from custom queries, but we’re still working on the evaluation of parsers.
Meanwhile, these dependencies can be resolved through multiple steps in the scheduler. Here is an example of scheduler parameters that would ensure the correct sequence of loads in a full data warehouse refresh:
-w <dwname> -b reset -w <dwname> -l load_earlier_table -w <dwname> -f all
This could also be written as:
-w <dwname> -b reset -l load_earlier_table -f all
These commands will ensure that load_earlier_table is the first table processed by the scheduler, and it will be available when needed by load_from_table.
Remember, any time you’re not sure how to do something using Ajilius, you are most welcome to contact us and discuss.