Approaches to running Databricks ETL code from Azure ADF
Databricks is fantastic, but there is a small issue with how people use it. The problem is that Databricks is all things to all people. Data scientists and data...
2020-06-03
1 reads
Ed Elliott's Sql Developer Blog
The four tenets of ETL testing
Every ETL pipeline is only ever as reliable as the data that the upstream system provides. It is inevitable that assumptions you make about the data you are provided...
2020-03-06
12 reads
The four tenets of ETL testing
Every ETL pipeline is only ever as reliable as the data that the upstream system provides. It is inevitable that assumptions you make about the data you are provided...
2020-03-06
2 reads
2020-02-21
3 reads
Passing status messages and results back from Databricks to ADF
When we use ADF to call Databricks we can pass parameters, nice. When we finish running the Databricks notebook we often want to return something back to ADF so...
2020-02-05
48 reads
Passing status messages and results back from Databricks to ADF
When we use ADF to call Databricks we can pass parameters, nice. When we finish running the Databricks notebook we often want to return something back to ADF so...
2020-02-05
2 reads
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
10 reads
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
1 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
163 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
2 reads
Blogs
Spark Connect Dotnet November 2024 Where are we?
By Ed Elliott
All Spark Connect Posts Introduction There have been quite a few changes in the last...
A New Word: Ochisia
By Steve Jones
ochisia – n. the fear that the role you once occupied in someone’s life...
Create a Numbers Table in Power Query
This is a quick blog post, mainly so I have the code available if...
Forums
Writer methods in PowerQuery (LOL)
I'm using PowerQuery in Excel on my desktop. (Yeah, I know you can do...
Need help with GROUP BY...
Here's my data... I am trying to return total days in status for each...
Need help with GROUP BY...
Here's my data... I am trying to return total days in status for each...
Question of the Day
The Density Vector
When building statistics, there is the concept of density that refers to the duplicates in a table. How is this calculated?
See possible answers