Approaches to running Databricks ETL code from Azure ADF
Databricks is fantastic, but there is a small issue with how people use it. The problem is that Databricks is all things to all people. Data scientists and data...
2020-06-03
1 reads
Ed Elliott
The four tenets of ETL testing
Every ETL pipeline is only ever as reliable as the data that the upstream system provides. It is inevitable that assumptions you make about the data you are provided...
2020-03-06
13 reads
The four tenets of ETL testing
Every ETL pipeline is only ever as reliable as the data that the upstream system provides. It is inevitable that assumptions you make about the data you are provided...
2020-03-06
2 reads
2020-02-21
3 reads
Passing status messages and results back from Databricks to ADF
When we use ADF to call Databricks we can pass parameters, nice. When we finish running the Databricks notebook we often want to return something back to ADF so...
2020-02-05
48 reads
Passing status messages and results back from Databricks to ADF
When we use ADF to call Databricks we can pass parameters, nice. When we finish running the Databricks notebook we often want to return something back to ADF so...
2020-02-05
2 reads
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
11 reads
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
1 reads
SQLCover 0.5 - Fixes, smaller features and an exciting surprise
It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...
2019-10-30 (first published: 2019-10-16)
403 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
164 reads
Blogs
Azure SQL offerings
By James Serra
There are three Azure SQL products with so many different deployment options, service tiers,...
T-SQL Tuesday #183 Roundup
By Steve Jones
I hosted this month’s T-SQL Tuesday party with my invitation asking about tracking permissions....
A Little Brainstorming with an AI
By Steve Jones
I was asked to do some a little thinking and brainstorming recently. Rather than...
Forums
Moving Database Files
I just learned that my database was created on my C:\ drive in the...
Migrate MSSQL to MYSQL - Different Servers
I am needing to migrate a MSSQL db to MySQL, on a different server...
how do you manage new columns in a tabular table
Hi we run 2019 ssas std. Yesterday i imported my tabular project to vs...
Question of the Day
Mixed Backups
I have a complex database with a few filegroups and files. Can I run a backup command like this? (assume file/filegroup names are valid).
BACKUP DATABASE [complex]
FILE = N'thirdone'
, FILE = N'thirdtwo'
, FILEGROUP = N'second'
TO DISK = N'C:\Program Files\Microsoft SQL Server\MSSQL16.SQL2022\MSSQL\Backup\complex.bak'
WITH NOFORMAT, NOINIT, NAME = N'complex-Full Database Backup', SKIP, NOREWIND, NOUNLOAD, STATS = 10
GO
See possible answers