Scripts

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

You rated this post out of 5. Change rating

2024-10-24

20 reads

Technical Article

While Loop in T-SQL

Often, we encounter situations where we need to loop through a dataset to process or update records iteratively. In such cases, I use WHILE loop like below. The example below demonstrates how a WHILE loop can be used to iterate Sales records. This approach is particularly effective when working with a numerical or date column, […]

3 (2)

You rated this post out of 5. Change rating

2024-10-15

90 reads

Technical Article

Comprehensive SQL Performance Monitoring and Optimisation Script

This script monitors and reports the execution statistics of SQL statements over a specified period, capturing metrics such as execution counts, CPU time, I/O operations, and elapsed time. It provides a comprehensive view of query performance, aiding in the diagnosis of performance issues and identifying problematic SQL queries in the database.

5 (1)

You rated this post out of 5. Change rating

2024-07-29 (first published: )

1,122 reads

Technical Article

Creating a CSV Parser in Apex

Salesforce supports the conversion of only csv files utilizing Apex. However if you use their LWC - js platform that provides wide options to bring in libraries to parse all kinds of file types. Here in this script i would be providing basic idea of functions utilizing the parsing logic along with options to iterate […]

5 (1)

You rated this post out of 5. Change rating

2024-05-21

8,077 reads

Blogs

Counting Groups with Window Functions: #SQLNewBlogger

By

I looked at row_number() in a previous post. Now I want to build on...

Extending a SQL Server Failover Cluster Across Regions in  Google Cloud Platform (GCP)

By

I was the principal author of this SIOS whitepaper, which describes how to build...

Read the latest Blogs

Forums

How to determine if a DB is currently being used

By dax.latchford

Hi - I'm looking for advice regarding the best & quickest way to establish...

Data cleansing/conversion tool HPE

By juliava

Hello.   I am looking for a tool Data cleansing/conversion, was recommended HPE any...

System views in a contained availability group

By mark4data

I have a query that runs in a job to check on orphaned users....

Visit the forum

Question of the Day

The LAGging NULL

I have this data in a SQL Server 2022 table:

player         yearid team HR
Alex Rodriguez 2012   NYY  18
Alex Rodriguez 2013   NYY  7
Alex Rodriguez 2014   NYY  NULL
Alex Rodriguez 2015   NYY  12
Alex Rodriguez 2016   NYY  9
If I run this code, what are the results returned in the hrgrowth column?
SELECT
  player
, yearid
, hr
, hr - LAG (hr, 1, 0) IGNORE NULLS OVER (ORDER BY yearid) AS hrgrowth
FROM dbo.playerstats;

See possible answers