#CloudCred

I saw a great tweet from Snowflake today:

I laughed when I read it, it exactly mirrors the way I feel about Oracle, IBM and Teradata (among others) with their cloudwashing of legacy DW platforms.

We’re cloud-first. Cross-cloud. Multi-cloud. Even hybrid-cloud. We’ve been working with cloud data warehouses since they were born.

Ajilius. CLOUD data warehousing.

Why Data Warehouse Automation kicks the crap out of SSIS

Here’s a great explanation of how to do incremental loads in SSIS. It uses third-party components from Pragmatic Works, and a combination of hand coding and SSIS tasks.

http://blog.pragmaticworks.com/incrementally-loading-data-from-salesforce.com

See how complex it is? How long it takes?

Here’s how to do the same thing in Ajilius:

Simply check the box on the column/s that govern incremental change.

Under the covers we do almost exactly the same tasks as described in the blog post, but the difference is that we do them, not you. You make a decision in the UI, and we instantly generate all the code and logic that implements that decision.

That’s it … a perfect example of why Data Warehouse Automation kicks the crap out of  great ETL tools like SSIS, and why we deliver data warehouses in a fraction of the time spent in those products.

Ajilius. Data warehouses. Faster.

Ajilius 2.4.0

We’re a month late, and we’re sorry. But Ajilius 2.4.0 has finally been delivered.

We’ll blog more about specific features, but here’s what you have to look forward to in this release:

  • BI Accelerators for Yellowfin, Tableau and Qlik. Full business-friendly metadata generation at the click (Qlik?) of a mouse. We’re the only data warehouse automation platform to support all three platforms.
  • BI Views for a unified query experience across any BI tool, including PowerBI and Excel. Now your PowerPivot users can share the same business-friendly table and column names as their Yellowfin and Tableau colleagues.
  • A revised, high-performance CTAS engine for MPP platforms, including Microsoft APS/PDW, Azure SQL Data Warehouse, Redshift and Snowflake. And we’re still the only data warehouse automation platform to support three click migration between supported platforms.
  • Full support for Microsoft APS/PDW. That makes us the only data warehouse automation platform that supports EVERY Microsoft RDBMS, SMP and MPP, on-premise and cloud.
  • A new Inference engine for managing early arriving facts. You have the one-click choice of automatically inferring dimension rows when new values are found in fact processing, or simply assigning those rows to the “Unknown” value.
  • Integrated Authentication for all Microsoft sources and targets.

Now the work begins on the Version 3.0 series of releases!

 

Is your metadata secure?

Are your databases at risk from your data warehouse automation product?

We’ve seen DWA where user credentials for production systems are held in clear text!

If you’re using a data warehouse automation product other than Ajilius, use a query tool to open your metadata database. For example, if your product uses SQL Server, use Management Studio to open the metadata. Now look at the metadata tables which store data warehouse and data source credentials. Can you read them? If you can, so can anyone who reads that database.

All Ajilius user passwords – that is, the passwords that you use to access Ajilius – are one-way hashed. That means you can’t read the password in our database, and we can’t reverse or recover your password, even if we have your database. Our hashing is based on the SHA256 algorithm for world-class data protection.

All credentials, to your data warehouses and your data sources, are fully encrypted. The user-id and password for each connection are encrypted using the AES256 algorithm, an incredibly strong algorithm used by many government security organisations.

Database query tools with authorised access to the metadata database only see a long string of characters like “4e3d3cc78416…” and not your real credentials.

Even if someone gets a physical copy of your metadata, and browses the disk file, they can’t read your database credentials.

Ajilius. We won’t risk your database security.

New competitor: BI Builders

A new week, and a new competitor. Norwegian company BI builders has popped into view with a very pretty looking dimensional warehouse solution for SQL Server users.

It is a desktop solution, generating SSIS, which puts it in the same category as Dimodelo, TimeXtender and (perhaps) WhereScape. LeapfrogBI is a little different, being a web-based solution.

This is becoming a very competitive section of the market!

New competitor: Optimal BI

It is great to see new entrants, bringing new approaches, to the data warehouse automation market. The days when one or two players had the market to themselves are drawing to a close …

This time it is Optimal BI, from New Zealand, with a product named Optimal Data Engine (ODE). It is a data vault product, evolving from consulting assignments, but not much more detail available at this time.

Follow their blog for new developments. http://optimalbi.com/blog/2015/07/03/ode-the-start-of-a-journey/

The old dinosaurs better get started on evolution!

Measuring load speeds

We recently had a situation where our load speeds were reported as being much slower than a competitor. This surprised me, because I knew that our loader could saturate the network from the source server, and I wondered how our competitor could be faster.

Luckily, the evaluator liked Ajilius, and did a little digging on our behalf. It turned out that the culprit was not our performance, but the competitor’s measurement technique.

When we load data into the warehouse from a source database, there are basically four steps that we need to perform:

  • Query
  • Extract
  • Load
  • Commit

The Query step is where we execute a query on the remote data source, such as “select c1,c2,c3 from t1 where c1 > 9467843”. The extract step is where we transfer the results of that query to the loader. The Load step moves those rows into the warehouse. Finally, the Commit step commits the load transaction/s. Depending on the source and warehouse, Ajilius may overlap one or more of those steps.

When we measure load performance we put a timer call before the Query, and again after the Commit. The elapsed time is the total time taken to extract and load the required data from the source system to the warehouse. This represents real-world performance, the type you need to measure if batch windows are important to you.

Our competitor had a different view of the world. Their measurement of performance was to take the time immediately before the Load step, and immediately after it. They claimed that this was a measurement of “load” performance. I guess they’re technically correct, but knowing just that one part of the job doesn’t help you to assess performance against real-world requirements.

When the customer repeated the tests, this time measuring the elapsed time for the whole job, the results were virtually neck and neck. I’m not surprised because, as I said earlier, I knew we were capable of saturating the relatively slow network in the customer’s development lab.

Ajilius performance tests? Always welcome.

WhereScape faking Data Vault?

Update (2016-07-09): It was true then, but it isn’t true now. WhereScape is now working with Dan Linstedt, the creator of the Data Vault methodology, to deliver automated Data Vault 2.0 solutions.

https://www.wherescape.com/blog/blog-posts/2016/july/wherescape-partners-with-data-vault-inventor-dan-linstedt/

 

I’m a little confused by recent claims about Data Vault and WhereScape Red.

Every WhereScape demonstration I have seen has been Load, Stage, Dimension, Fact, Cube. The tutorial is Load, Stage, Dimension, Fact, Cube. The training is Load, Stage, Dimension, Fact, Cube. The UI boils down to Load, Stage, Dimension, Fact, Cube.

Does that look like Data Vault to you?

Nowhere do I see Hub, Link and Satellite. Just Load, Stage, Dimension, Fact, Cube.

If it looks like Dimensional, presents like Dimensional, trains like Dimensional, and is documented like Dimensional, then there is a pretty good chance it IS Dimensional, and any claim to the contrary is just faking it.

If you want Data Vault, you probably need to be looking at BIReady or Quipu.

We’re proud of the fact that Ajilius is a data warehouse automation company based on Dimensional Modelling. We believe in doing one thing well, not trying to be anything to everyone. If you want a Dimensional data warehouse, buy the product that is firmly committed to this technique.

Ajilius. Keeping it real in data warehouse automation.