Create Data “Thumbnail” in Azure Data Lake Store

Recently, a colleague could not create a model in Power BI Desktop because the dataset files were too large. The smallest file was 45GB. He asked if we could rerun the Azure Data Lake Analytics job (U-SQL) to create a smaller dataset. We could, however one of the jobs that creates the datasets runs for about four hours. The solution I came up with was to create a dataset “thumbnail” which would provide a small […]

Finding Column Definitions in Oracle, SQL Server and DB2

In order to speed up the development of Azure Data Factory pipelines, I created a table containing definitions of various databases tables. Each of the big three databases provide a method for querying table and column definitions. SQL Server select table_catalog, table_schema, table_name, column_name, ordinal_position, data_type, coalesce(character_maximum_length,numeric_precision) as column_length from information_schema.columns Oracle select ‘myOracleDatabase’ as databasename, owner, table_name, column_name, column_id, data_type, data_length from all_tab_columns DB2 select ‘myDB2Database’ as databasename, tabschema, tabname, colname, colno, typename, length […]