Data Profiling Patterns

Patterns in data are a great way to summarise the type and content of a column.

Our data profiling feature creates patterns in the following manner:

  • Values are converted to a string representation
  • Alphabetic characters are converted to an ‘x’.
  • Characters in the range 0-9 are converted to a ‘9’.
  • All other characters are left as-is.

Here are some examples of pattern conversions:

  • 2010-09-23 to ‘9999-99-99’
  • 09/07/2016 to ’99/99/9999′
  • Ajilius to ‘xxxxxxx’
  • (03) 5432 9876 to ‘(99) 9999 9999’

We find that the frequency of pattern occurrence is more valuable than the frequency of value occurrence. For example, it is more useful to know that a column contains only values with a pattern ‘9999-99-99’ than to see a long list of dates, each occurring only a handful of times.

Ajilius. Better profiling for better data warehouses.