Picture this, your data ingestion team has created a table that has the sales for each month year split into different columns. At first glance, you may think “what’s the big deal? Should be pretty easy, right? All I need to do is unpivot these columns in Power BI and I’m good to go.” So you go that route, and the report works for one month. Next month, you get an urgent email from your stakeholders saying they can’t see this month’s numbers. That’s when you realize that this table will grow with new columns every month. That means that any report you make needs a schema refresh every single month. Unfortunately, Power BI will not grab new columns from a table once it’s published into the online service. The only way for the Power Query to pivot the new columns is for you to open the report in your desktop, go to Power Query, and refresh the preview to get all the columns in that table.
If you’re like me, you may not have time (or the will) to manually download, fix, and reupload this report each month. But have no fear! SQL is here! The unpivot function in SQL can feel like a black box, designed for only black belts in SQL to use, but we will break it down together and create a dynamic unpivot script.
What does unpivot in SQL accomplish? Unpivot takes multiple rows and converts them to columns. The general syntax is as follows:
SELECT
originalColumn1, originalColumnHeaders, originalValues
FROM
(SELECT originalColumn1, originalColumn2, originialColumn3, originalColumn4 FROM dbo.Table) T
UNPIVOT (originalColumnHeaders FOR originalValues in (originalColumn2, originialColumn3, originalColumn4)) as UP
Let’s break this down.
The first SELECT statement contains the column we don’t want to unpivot plus the two names of what we want our final columns to be. For example, using our table below, we want to keep the country column as is, but we want to unpivot those sales months and the sales values. So, we may name the originalColumnHeaders as SalesMonth and originalValues as Sales. You can use whatever you’d like, but it has to match what’s in the UNPIVOT statement.
The SELECT after FROM tells the query where to get data from. By default, SQL requires this to be hard-coded, but don’t worry, we have a way for that to be dynamic.
The UNPIVOT statement requires us to provide two new column names, the first will be what you want the column full of the old column headers to be called. In our case, this will be SalesMonth. The second will be the column full of the old values within those columns (aka Sales).
The hard-coded version of our example would look like this:
SELECT
country, SalesMonth, Sales
FROM
(SELECT country, sales_jan_2024, sales_feb_2024, sales_mar_2024, sales_apr_2024, sales_may_2024 FROM dbo.Table) T
UNPIVOT (SalesMonth FOR Sales in (sales_jan_2024, sales_feb_2024, sales_mar_2024, sales_apr_2024, sales_may_2024)) as UP
The result of this code looks pretty good! Now we can easily pull the month and year out from SalesMonth and have some data we can trend. But we still have that pesky hard coded problem.
To get around the hard coding, we have to do some SQL jujitsu. We will use a couple of variables that can be dynamically populated then execute the SQL as a variable using sp_ExecuteQueries. This is a bit easier to explain with in-code notes, so feel free to look through this query and read the notes that go along with it for details on how it works.
--Use this script to create the table for the demo
/* --Comment this line out to run the create table code
DROP TABLE IF EXISTS dbo.PivotedSales
CREATE TABLE dbo.PivotedSales (
country VARCHAR(250),
sales_jan_2024 DECIMAL(17,3),
sales_feb_2024 DECIMAL(17,3),
sales_mar_2024 DECIMAL(17,3),
sales_apr_2024 DECIMAL(17,3),
sales_may_2024 DECIMAL(17,3)
);
INSERT INTO dbo.PivotedSales (country, sales_jan_2024, sales_feb_2024, sales_mar_2024, sales_apr_2024, sales_may_2024)
VALUES
('South Africa', 111.22, 222.33, 333.44, 444.55, 555.66),
('Canada', 112.22, 227.33, 332.44, 400.55, 500.66),
('United States', 113.22, 228.33, 330.44, 401.55, 501.66),
('Mexico', 114.22, 229.33, 334.44, 404.55, 504.66),
('Ireland', 115.22, 230.33, 335.44, 409.55, 509.66),
('Germany', 116.22, 231.33, 336.44, 499.55, 599.66),
('South Africa', 1011.22, 2022.33, 3303.44, 4044.55, 5505.66),
('Canada', 1102.22, 2027.33, 3302.44, 4000.55, 5000.66),
('United States', 1103.22, 2280.33, 3030.44, 4001.55, 5001.66),
('Mexico', 1104.22, 2209.33, 3034.44, 4004.55, 5004.66),
('Ireland', 1105.22, 2300.33, 3305.44, 4009.55, 5009.66),
('Germany', 1106.22, 2310.33, 3036.44, 4909.55, 5909.66);
--*/--Original table
SELECT * FROM PivotedSales
--Set up parameters
;DECLARE @UnpivotColumns NVARCHAR(MAX), @FilterPefix NVARCHAR(MAX)
-- FilterPrefix is the prefix that all the columns we want to pivot have. This will allow us to dynamically grab any new columns as they get created
SET @FilterPefix = 'sales%' --Note, you can adjust this to be a suffix or a middle string if needed by moving the % (wildcard)
--This section sets our @Unpivot column variable to be a comma separated list of the columns in our table with the FilterPrefix
SELECT @UnpivotColumns = STRING_AGG(CONVERT(NVARCHAR(MAX), C.name),',')
FROM sys.columns C
INNER JOIN sys.types T ON T.user_type_id = C.user_type_id
WHERE C.object_id = object_id('dbo.PivotedSales') --this ensures we only get columns from our table
AND C.name LIKE ''+@FilterPefix+'' --this makes sure only columns with the filter prefix are returned
AND (T.name = 'decimal' OR T.name LIKE '%int%') --this ensures we only grab columns with a decimal or int type. You can adjust this if needed
SELECT @UnpivotColumns AS 'Unpivot Columns'
--This section creates a dynamic SQL statement using the comma separated list we just generated
DECLARE @ExecuteSQL NVARCHAR(MAX)
SET @ExecuteSQL = '
SELECT Country, SalesMonth, Sales
FROM
(SELECT Country, ' + @UnpivotColumns + ' FROM dbo.PivotedSales) P
UNPIVOT
(Sales FOR SalesMonth IN (' + @UnpivotColumns + ')) as S'
SELECT @ExecuteSQL AS 'Dynamic SQL Script' --this will show you the SQL statement we've generated
--Finally, we will use the system stored proc sp_executesql to execute our dynamic sql script
EXECUTE sp_executesql @ExecuteSQL
Now that is a lot of code, so be sure to look at each chunk and make sure you adjust it for your use case. Happy coding folks!
Huge credit for this solution goes Colin Fitzgerald and Miranda Lochner! Thank you both for sharing this with me!