Scripts

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

5 (1)

You rated this post out of 5. Change rating

2024-11-13 (first published: )

425 reads

Technical Article

Concatenating Multiple Row Values into a Single Comma-Separated List

In scenarios where you need to consolidate multiple rows into a single, comma-separated value, you can achieve this using FOR XML PATH. This script demonstrates how to retrieve volunteer data and display the days they have selected for participation.

You rated this post out of 5. Change rating

2024-11-11 (first published: )

758 reads

Technical Article

While Loop in T-SQL

Often, we encounter situations where we need to loop through a dataset to process or update records iteratively. In such cases, I use WHILE loop like below. The example below demonstrates how a WHILE loop can be used to iterate Sales records. This approach is particularly effective when working with a numerical or date column, […]

3 (2)

You rated this post out of 5. Change rating

2024-10-15

353 reads

Technical Article

Comprehensive SQL Performance Monitoring and Optimisation Script

This script monitors and reports the execution statistics of SQL statements over a specified period, capturing metrics such as execution counts, CPU time, I/O operations, and elapsed time. It provides a comprehensive view of query performance, aiding in the diagnosis of performance issues and identifying problematic SQL queries in the database.

5 (1)

You rated this post out of 5. Change rating

2024-07-29 (first published: )

1,821 reads

Blogs

Comparing My Current Schema with a Backup with SQL Compare

By

A customer asked if they needed to restore a database from backup to compare...

Improving Our Conversations

By

If we aren't having the same conversation, we aren't going to be heard. That's...

How Do You Pick Events in Extended Events?

By

A while back I wrote about using AI to explore why people are not...

Read the latest Blogs

Forums

SSAS Cube Migration to Azure cloud

By bala2014

Dear All, I am currently working on a Proof of Concept (POC) where I...

create a table with data capture for next 3 years

By anug

Hello, I would like to build a table capturing data starting from 1st day...

Is GenAI Coming Faster Than You Think?

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Is GenAI Coming Faster Than...

Visit the forum

Question of the Day

Local Storage Options

I'm setting up a SQL Server 2019 instance and we are planning on using SMB storage for our database files. However, the file share isn't ready, so the idea is to use the \127.0.0.1dbfile as the location to start and then move these files to the remote server. Can I do this?

See possible answers