Passing status messages and results back from Databricks to ADF
When we use ADF to call Databricks we can pass parameters, nice. When we finish running the Databricks notebook we often want to return something back to ADF so...
2020-02-05
3 reads
Ed Elliott
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
13 reads
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
2 reads
SQLCover 0.5 - Fixes, smaller features and an exciting surprise
It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...
2019-10-30 (first published: 2019-10-16)
408 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
169 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
7 reads
SQLCover 0.5 - Fixes, smaller features and an exciting surprise
It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...
2019-10-16
6 reads
How to test ETL Processes in production
This is the final part in the four-part series into testing ETL pipelines, how exciting!
Part 1 - Unit Testing https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ Part 2 - Integration Testing https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/ Part 3...
2019-10-02
122 reads
How to test ETL Processes in production
This is the final part in the four-part series into testing ETL pipelines, how exciting!
Part 1 - Unit Testing https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ Part 2 - Integration Testing https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/ Part 3...
2019-10-02
8 reads
How to connect spark to ms sql server without '[Error] [JvmBridge] java.sql.SQLException: No suitable driver'?
“[Error] [JvmBridge] java.sql.SQLException: No suitable driver” - unable to connect spark to Microsoft SQL Server.
In spark when you want to connect to a database you use Read() passing in...
2019-10-01
424 reads
Blogs
TempDB Internals – What’s New (SQL Server 2016 to 2022)
By Vinay Thakur
I wrote about TempDB Internals and understand that Tempdb plays very important role on...
AI: Blog a Day – Day 2: Generative AI, Multimodal Systems, and Agent AI
By Vinay Thakur
continuing from Day 1 where we covered the history of AI and GPT family,...
A Wellbeing Day at Redgate
By Steve Jones
It’s a day off for Redgate today. This is our annual wellbeing day, where...
Forums
A Quick Restore
Comments posted to this topic are about the item A Quick Restore
Guarding Against SQL Injection at the Database Layer (SQL Server)
Comments posted to this topic are about the item Guarding Against SQL Injection at...
Ola Hallengren Index Optimize Maintenance can we have data compression = page
I have a quick question on Ola Hallengren Index Optimize Maintenance . Do we...
Question of the Day
A Quick Restore
While doing some testing of an application, I wanted to reset my environment after doing some testing with this code:
USE DNRTest BACKUP DATABASE DNRTest TO DISK = 'dnrtest.bak' GO /* Bunch of stuff tested here */RESTORE DATABASE DNRTest FROM DISK = 'dnrtest.bak' WITH REPLACEWhat happens if this runs, assuming the "bunch of stuff" isn't anything affecting the instance. See possible answers