Constant Azure Improvement

This week I noticed a number of changes in the Azure platform from their change feed. Actually, every week has a crazy number of changes across the entire platform, but this week a number of these were related to data and databases. That isn't always the case, but I saw some interesting items.

We have zone redundancy for the general purpose tier of Azure SQL database, as well as AAD security principals in the master database and a new DMV for backups. I think Azure SQL Database is a great product, though one that has felt quite limited at times. The use of DTUs at first, the high pause period for Serverless and limited options at lower tiers were always annoying.

The point at times seemed to be to get customers to place a big financial bet that a higher tier would meet their needs. I had hoped that some of that was just a blip in time as technology improved and more features would be deployed to all tiers. After all, the original Azure SQL database was a key-value store.

That seems to be coming true as I continue to see improvements to the offerings. Azure SQL Serverless now has a 1 hour pause period minimum instead of the original 6 hour time. Now Cosmos DB has also lowered their autoscale offering. Cosmos DB can be provisioned at a lower entry point for autoscale, at 100RU/s instead of 400RU/s, which might get more people to experiment with a PoC over a bit of time and use the database. I know I'm thinking of trying to mess with some data on the platform. I don't think I'd need 100 RU/s, much less 400, but I can better live the cost for a few months.

Hyperscale has been an interesting version of Azure SQL Database. I know quite a few customers that aren't sure if they need it, but they considered experimenting with it. The problem was that once you choose Hyperscale, you can't go back to another tier. That's quite a disincentive to experimentation, which seems to be part of the philosophy of choosing the cloud. There is now a preview of the migration of Hyperscale back to the general purpose tier. I'm guessing it took a little time for the Azure team to implement this, which is why there was a restriction, but it looks like that limit will go away.

The cloud is an amazing place, as I've written before. It enables things to happen at a pace, and with ease that I've never seen in a data center without a lot of prep work. This week it seems that the Azure cloud advanced quite a bit in the area that I care about, the data platform space.

There was also the general availability release of the Azure SQL Migration extension for ADS.

Steve Jones - SSC Editor

Join the debate, and respond to the editorial on the forums

Flyway provides a database-independent way for a team to track, manage and apply database changes, while maintaining strict control of database versions. It updates a database by running a series of versioned migration scripts, in order, and keeps track of all the changes in a special "schema history" table. It sounds simple, but it is easy to derail this team discipline if you don't find the right answers to the following questions…

Around this time last year, the 2021 AI in Healthc...

I had an email discussion with someone last week who was wondering about log backups and minimally-logged operations. Specifically they were confused as to how the log backup works if no checkpoint has occurred since the operation completed. Surely the data pages are still dirty in memory?

Did You Forget? This is another one of those sett...

CXCONSUMER and CXPACKET are associated with parall...

Everywhere You Look Finding out about this was pr...

The Max For The Minimum I’ve written several hundred thousand times about how important memory is to SQL Server, and to most databases in general. But once you have it,...

In this article, we will explore Microsoft SQL Server trace flags with all aspects, and we will learn also how to use them. Introduction Trace flags are the advanced...

I was always wondering what the KEEP PLAN hint does. The documentation isn’t very specific (emphasis mine):

When recovering a certificate created during a CREATE CERTIFICATE I get a "Warning: The certificate you created is expired" message , although the certificate is added to the server. Is this an issue? how can I fix?

Learn about how Delta Sharing works along with the benefits as well as creating, altering, describing, and sharing Delta Lakehouse data using Databricks. Also learn about consuming Delta Lakehouse...

This article explores the database-level roles in the Azure SQL Database. Introduction to Azure SQL Database security management Database security is critical for an organization to protect unauthorized access...

  In the previous blog, we covered how to configure server level audit and database level audit for Azure SQL database with default settings. In this blog we will see how you can...

Azure SQL Database takes regular automated backups for user databases and stores them in Azure Storage to enable point-in-time-restore to any desired point within configured retention. Automated backups include...

Unfortunately, as of April 2022 there is not an option to parameterise or add dynamic content to an “Execute Pipeline” activity to invoke a pipeline run.

This article reviews a common pattern of streaming data (i.e. real-time message ingestion) in Synapse dedicated pool. It opens a discussion on the simple standard way to implement this, as well as the challenges and drawbacks.

In this article we discuss the incremental process...

The Azure Synapse Analyzer Report was created to help you identify common issues that may be present in your database that can lead to performance issues. This report focuses on known best practices that Microsoft has identified with SQL Dedicated Pools. Following these best practices will help to get the best performance out of your solution.

Azure Synapse users are looking to unlock access to on-premises, open source, and hybrid cloud systems to extend advanced analytics capabilities for their organizations. Building connectivity between all your distributed data, apps, and data warehouses is not a trivial task.

Reading Time: 2 minutes For this months T-SQL Tuesday contribution I want to share best T-SQL advice I would give younger self. This months T-SQL Tuesday is hosted...

It’s that time of the month again, when we have ...

Be mindful of advice. Give an ear to those offering it. Just be mindful, there is no need to accept the advice, but it is wise to at least... The...

Words: 459 Time to read: ~ 3 mins Welcome to TSQL Tuesday 149! The monthly blogging party where we are given a topic to write a post around. This...

This month’s T-SQL Tuesday invitation comes from Camila Henrique who asks us to go back in time and give some T-SQL advice to our younger selves. It’s not hard...

The next career step for many IT engineers is IT manager. But they need more than technical prowess to make that jump. Here are five ancillary IT management skills...

This article will explain the steps of upgrading t...

Now that we have created our Redis Cache lets conn...

Reference architectures are great! You’ve got all of the key components in there, nice and clear. Colourful lines showing how data moves through each stage, product, or service. Great for a slide deck or a proposal to get rid of that old creaking data warehouse and into a shiny new Data Lakehouse.

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. With Amazon Redshift, you can analyze all your data to derive holistic insights about your business and your customers. One of the challenges with security is that enterprises don’t want to have a concentration of superuser privileges amongst a handful of users.

This article will explain Azure storage explorer, its installations, and details of how to start working with this application to access Azure storage services. Azure storage provides a flexible solution to store various types of data at a massive scale in the cloud environment.

Introduction to KQL - the Kusto Query Language

Join Redgate's James King in conversation with Chris Yates, Senior Vice President, Managing Director of Data and Architecture at Republic Bank to discover why monitoring your database environment helps your organization gain a competitive edge.

I’m excited to announce I’m giving a webinar f...

Today Paul sent out the quarterly numbers to the team letting us know the hours our courses have been viewed during March. Being a data guy, I decided that... The...

Pre-Register For My NEW Exclusive Performance Training! I used my over 30 years of SQL Server experience to personally curate my favorite performance tuning tips and tricks into this...

SQL Saturday Haiti is coming on Apr 30 in French. Join this online conference to learn a bit about data and databases for free.

Join Microsoft Data Platform MVP, and AWS Community Builder Grant Fritchey to discover his top tips on how to kick start DevOps in your organization. From starting small to making work visible and acting on feedback.

Introduction I will be speaking at SQL Saturday Ja...

In this blog post I will be sharing a PowerShell script that allows multiple Power BI reports to be downloaded at once.

Want to run a PowerShell script? Here are the steps to follow.

New on the blog: Tips for #PowerShell Gallery Success

I am a Cmdlet and Module Working Group member that helps triage Github issues for PowerShell. Like all of you, resources are limited, and Microsoft can only take on so much work. We often see requests for new or enhanced cmdlets. Often, we recognize the value, but it comes down to prioritizing and criticality. Even if Microsoft had the resources to extend PowerShell, meeting every valid request, PowerShell would grow to an unmanageable size. Thus, we need to turn to the PowerShell community.

This is a guest post written by SWD community member Lucia Stefanuto. She was a recent Member Spotlight nominee—you can read more about her and her work here—and in...

In the first part of my DevOps 101 series, we tackled the fundamentals of What, who why and how. In this follow-up, I’m going to focus on how to...

What’s the difference between Site Reliability Engineering and DevOps? In this article, Grant Fritchey defines each one and compares them.

Deleting a single document is pretty straightforward in Elasticsearch. We can simply issue a DELETE on the document id and the document will be deleted from the index.

Deleting a single document is pretty straightforward in Elasticsearch. We can simply issue a DELETE on the document id and the document will be deleted from […] The post How...

In this article learn how to build an SSIS package...

When Azure Data Factory (ADF) released the managed virtual network feature for Azure integration runtimes, it seemed like a no-brainer at first.

The last post in this series covered some simple Python code that leveraged twitter’s tweepy API in order to obtain tweets based on a query, sentiment score each tweet...

A couple / three days after delivering the April 2022 edition of Master the Fundamentals of Azure Data Factory (the next delivery is 20 May 2022 – check out...

While Uninterruptible Power Supplies are hardly a new thing in the PC space, the tried-and-true battery backups for desktop PCs have been undergoing a resurgence in popularity in recent...

RANK.EQ: Returns the rank of a number in a column ...

Understanding the difference between a row context and a filter context is the first and most important concept to learn to use DAX correctly. This video introduces the filter...

A lot of videos and articles teach you how to write [DAX]. In this video Daniil Maslyuk and I will show you you how avoid bad [DAX] patterns! This...

What?! Exploding Measures I hear you say. Exploding, expanding, use which ever word you like; what I mean is when you want to remove any refence to a measure...

POISSON.DIST: Returns the Poisson distribution. https://dax.guide/poisson-dist/

Microsoft has announced some important changes that should make it...

Flexible Server is a fully managed database servic...

Part 1: What is testable code, why is it important, and first glimpses on practices that can help

You’re investigating your SQL Server’s top wait stats, and you’re noticing a lot of LCK% waits. Normally, that indicates blocking and deadlocks, but you’re just not getting complaints from...

Dirty Work There are three classes of administrator when it comes to parallelism settings Never changed the defaults Changed the defaults to something wacky Changed the defaults to something safe The beginning of this...

TLDR: There ain’t one. I was privileged last week to be able to present a couple of sessions at the SQL Server and Azure SQL Conference (great event, I... The...

Do you use a central date table for your Power BI ...

I have previously written articles about how you can write a measure in DAX that helps with TOP N filtering. However, you may not need that calculation for many...

Learn the who, what, where, when and why of Power BI to help you get a better understanding of how this reporting tool fits into your data environment.

While Power BI is a SaaS service, backing up your dataset is still important. What if something was accidently deleted? How do you recover it? Patrick shows you how...

The merge operation (Table.NestedJoin) is the M language equivalent to creating relationships between tables in the model. The resulting ADX operation is join.

Power BI is a mobile-friendly reporting solution. Every report that you generate can be viewed on the Mobile too. However, it doesn’t mean that the report is designed for mobile,...

Data masking is a technique of hiding sensitive data from certain users who should not be able to view or access the actual values. This post will look at...

How to integrate Flyway database development with Source control, so that you can track what changes were made and who made them as well as which objects changed between...

I’ve said it before but I’ll say it again: I don’t publish book reviews here on my blog but I’m always happy to promote new Power BI books when...

Wow, talk about stable! The last 3 months have only introduced a series of small bug fixes. I held this release back as long as I could, waiting to...

Assess, receive Azure recommendations, and then migrate your SQL Server databases using the Azure SQL Migration extension in Azure Data Studio.

Set autoscale on your database or container with a new scale range of 100 RU/s – 1000 RU/s.

Make your general purpose provisioned and serverless Azure SQL databases and elastic pools more resilient with catastrophic datacenter outages, without any changes of the application logic by selecting zone...

Unique indexes can now be created in existing collections with data in the API for MongoDB.

Public preview enhancements and updates released for Azure SQL.

Easily move your previously migrated databases on Azure SQL Database Hyperscale back to the general purpose tier.

The new result set grid in log analytics boasts a brand-new experience, introduces numerous new features, and is much easier to use!

Build reliable, maintainable, and testable data processing pipelines with Delta Live Tables.

Massaging Data using Pandas by Adrian Tam on April 14, 2022 in Python for Machine Learning Tweet Tweet Share When we talk about managing data, it is quite inevitable to see data presented in tables. With column header, and sometimes with names for rows, it makes understanding data easier. In fact, it is quite often to see data of different types staying together. For example, we have quantity as numbers and name as strings in a table of ingredients for a recipe. In Python, we have the pandas library to help us handle tabular data.

In this article we look at various built-in functi...

This is a follow-up to a short post I wrote on R Access to Twitter’s v2 API. In this post I’ll walk through a few more examples of pulling data from twitter using a mix of Twitter’s v2 API as well as the {rtweet} package1.

Firms focus too narrowly on external attackers whe...

John Oliver has an excellent segment on data brokers and surveillance capitalism.

Open source code continues its steady takeover of codebases, and organizations have made slight gains in eliminating out-of-date and vulnerable components.

Court records unsealed Tuesday indicate that the United States recently obtained judicial authorization to seize three domains that long hosted the RaidForums website.

In cybersecurity, user error is the symptom, not the disease. A healthy culture acknowledges and addresses the underlying causes of lapses.

How to perform minimal blocking updates in a high ...

In my last post, I discussed some of the DML commands in SQL Server. This week I want to discuss another category of commands – Data Definition Language, or...

When I first created this JOIN Types poster or cheat sheet back in 2013 I didn’t realize there was such a need for it. I originally made this as...

I recently came across dbdiagram.io, a free, web-b...

Itzik Ben-Gan introduces another T-SQL challenge; this time, it involves finding islands without explicit sort operators. The post Islands T-SQL Challenge appeared first on SQLPerformance.com.

A power failure at a major London data center show...

The move comes as data volumes explode, driving de...

Chocolatey is a package manager for Windows, like the built-in package managers on Linux, and third-party ones on macOS like Homebrew and MacPorts. The idea is this: when you’re... The...

I had started a Stairway series on containers (which I need to get back to), and I got a question. How do we set the name of a container?... The...