Many companies have under provisioned lower environments since they are continuously running. What if you can turn on and off your major Azure Services to save money. If needed, these savings can be used to increase the size of your computing power to solve problems quickly during development and testing.
Here is a paragraph from Microsoft Learn about Logic Apps. “Azure Logic Apps is a cloud platform where you can create and run automated workflows with little to no code. By using the visual designer and selecting from prebuilt operations, you can quickly build a workflow that integrates and manages your apps, data, services, and systems.”
Business Problem
Our manager has asked us to save money by turning on our lower environment services at 10 am and off at 6 pm. We want to estimate the total savings by the end of the project.
Technical Solution
To make life easy, we are going to create two Logic Apps. One that is scheduled at 10 am to start the service and one at 6 pm to stop the service. Since this is a low code environment, most of the time will be finding the information to plug into each step.
List below is the high-level algorithm for each Logic App.
- Create a schedule trigger
- Create a step to stop/start the service
It has been noted that the following services are consuming the most amount of money in our Azure Subscription.
- Azure PostgreSQL Database – Flexible Server
- Microsoft Fabric Capacity
- Azure Virtual Machine with SQL Server 2019
- Azure Synapse Dedicated SQL Pool
- Azure Databricks Computing Cluster
Let’s start our journey into the world of low code applications.
Azure PostgreSQL Database
If you install PostgreSQL on a virtual machine, please see the section below on how to stop and start an Azure Virtual Machine. Today, we are focusing on the flexible server offered by Microsoft. This is considered a Platform as a Service (PaaS) offering.
The next few tasks will be common for all Logic Apps that we create. I will go over them in detail here. In future sections, these steps are skipped so that we can focus on what is truly different between services. The first step is to create a Logic App from the Marketplace. I am choosing the consumption model since it is very cost effective.
I found the following Logic App pricing as of August 2024. Since we are triggering the Logic App one time a day, the actions never exceed 4,000 and the application is essentially free. There is a minor cost for data retention and standard connectors.
The usual information must be supplied before deploying the service. Here is a list of items that you need to define: resource group, Logic App name and region of service.
Most services deployed via the Azure Portal ask the end user to check the details before starting the deployment. See the image below for a typical review screen.
If the deployment is successful, you can go to the resource to start configuring the Logic App.
Go to the overview page of the Logic App called la4azpgsql2stop. My naming convention is the following: la – logic application, 4 – break character, azpgsql – Azure PostgreSQL database, 2 – break character, and stop – action word. Use the edit button to start our low code journey.
The first step in testing is to check the current state (status) of our service. Right now, the Azure Database for PostgreSQL flexible server is in a stopped state.
Now that we have confirmed the status of our PostgreSQL database is stopped, we’ll go back and start to edit the Logic App. The recurrence task allows the developer to dictate the schedule for the job. The invoke resource operation allows the developer to talk to Azure Services via a connection.
The question is how complex it is to make the Logic App. I am trying to keep it simple stupid (KISS). Right now, the start operation will fire off at 10 am. More than one time period can be entered. However, branching logic would be required to determine the stop or start action. I leave that future design for you to code.
The invoke resource operation task is based upon a REST API call to Azure. We can find documentation for PostgreSQL here. The short resource id can be obtained from the Azure Portal by looking at the query string. The image below shows the complete details to start the PostgreSQL service at 10:00 hours.
The logic app keeps track of run history. The first run failed due to bad parameters. The second run took 2 m to start the database service.
We always want to verify the desired result of a given Logic App. In our case, we can see the database service is now available by looking at the image below.
Let us create the stop Logic App for the PostgreSQL database service. The image below shows the recurrence task with an 18:00 hours scheduled time.
I wanted to show the reader all the possibilities with Logic Apps. We can create both resource groups and resources at run time if we want to.
Please copy the information from the start task for invoking a resource operation. Instead of the start endpoint, use the stop endpoint.
For some reason, it took 3 minutes to stop the service instead of the 2 minutes it took to start the service.
Some services have an intermediate state that is displayed in the portal. We can see the service is in a stopping state. Once the action is completed, we will go into a stopped state.
To recap, the deployment and configuration of a Logic App to start and stop services on a schedule is very easy. In the next section, we will talk about saving money with Azure Fabric Capacity for Microsoft Fabric.
Azure Fabric Capacity
Microsoft Fabric is the new Software as a Service (SaaS) platform that everyone is talking about. If you want to use resources that are dedicated to your company, you will have to deploy some capacity in the Azure Subscription that you own. Of course, we can pause and resume the service to save money in the lower environments.
The image below shows the capacity in a paused state.
The hardest part of setting up these tasks is to find the REST API information. The image below shows the required information to resume the Fabric Capacity.
We always want to test after triggering the logic app. The image below shows the capacity in an active state.
The image below shows the required information to suspend the Fabric Capacity.
Just remember we are saving 2/3 of the cost in the lower environments. At the end of the article, we will tally up all the savings.
Synapse Dedicated SQL Pool
The dedicated SQL pool refers to the enterprise data warehousing features that are available in Azure Synapse Analytics. This architecture uses multiple parallel processing to solve data warehousing problems.
The image below shows the dedicated SQL pool in a paused state. Let us create a Logic App to start the service.
Below shows the Logic App that calls the resume endpoint to start the service.
It takes a while to start up the data warehouse. We can see the service is currently resuming.
The final state is achieved when the data warehouse is online.
Below shows the Logic App that calls the pause endpoint to stop the service.
I will let you explore whether the service has a pausing state. Since the Data Warehouse is a big-ticket item, a lot of money can be saved by limiting the availability of the service in the lower environments.
Azure Virtual Machine
Many companies want to lift and shift their assets to the Azure Cloud. Creating a virtual machine in Azure is part of Infrastructure as a Service (IaaS). Customers are charged for the hardware, operating system and software as one fixed monthly price. A higher computing configuration results in a higher cost. Today, we are going to look at Windows Server that is running SQL Server 2019.
The image below shows the virtual machine is a disabled state.
What I did not show previously is a connector that you might have to configure before creating a task. The image below shows the john@craftydba.com account being used to sign into Azure to execute the task.
The image below shows the start virtual machine task. There is no REST API information required for this action.
Starting up the virtual machine takes about 2 minutes.
If you are a developer, a good part of your day should be testing code with both negative and positive test cases. The desired outcome of this task is to place the virtual machine in a running state.
Use the deallocate virtual machine task to prevent your company from incurring charges. There is a power off option that you do not want to use since charges are still accumulating.
The run history shows the Logic App to stop a virtual machine takes less than a minute to execute.
The image below shows the virtual machine in a deallocating state.
The low code promise of Logic App has really paid off. Within less than 5 minutes, we can make and test an application.
Azure Databricks Clusters
Azure Databricks is a distributed processing system used for big data workloads. It utilizes in-memory caching, and optimized query execution for fast analytic queries against data. Because clusters are made up of many virtual machines, the cost of keeping up the cluster 24 x 7 is quite costly.
Up to this point, there was a task in Logic Apps that could be used to talk to the service. Microsoft does not own the Databricks software since it is cloud agnostic. Thus, the REST API endpoints are limited to actions in the portal. See full details here. Do not fret. We can use the endpoints in the Databricks service to manage the cluster. First, we need to create an personal access token to be used within the HTTPS header.
We need to grab both the Databricks Workspace and Databricks Cluster ids. They can be found in the URL string and/or settings pages. The image below uses the HTTP task to post a message to start the cluster.
At this time, we can see the cluster is in a terminated state.
After executing the logic app to start the cluster, we can see the green check mark indicating it is ready to use.
The image below uses the HTTP task to post a message to delete (terminate) the cluster.
The last task is to manually check if the Databricks cluster is in a terminated state.
I do not know if you realize the world is now your oyster. Most cloud services communicate with JSON over HTTPS endpoints. We just need to write code to log into a given service and retrieve a bearer token. Once we have the access token, we can make calls to the service provider.
Cost Savings
I used the Azure Portal to grab cost information when it was readily available. Otherwise, I used the Azure Pricing Calculator to figure out the cost of each of the five services.
The PostgreSQL service costs $427 dollars a month. I will be rounding up the cost to make the math easy.
The Fabric Capacity (F8) costs $1,051 dollars a month
Azure Synapse Dedicated Pool costs $1,102 dollars a month
The virtual machine running SQL Server 2019 costs $2,739 dollars a month
The Azure Databricks Cluster costs $1,094 dollars a month since the auto terminate has been turned off. Users are querying Azure Databricks directly using the Spark SQL endpoint.
Our company is spending a total of $6,413 dollars per month in the lower environments. By enforcing the services to be online only 8 hours a day, the company will be realizing cost savings of $4,272 per month. We can save even more money by turning the servers off on the weekend. Additionally, we can disable the start logic so that developers must manually turn on the services in each environment each day. Thus, if your team goes days without testing your company will enjoy greater savings. This does not mean I advocate for not testing code in lower environments!
During my design I choose to ignore errors. Thus, if the service is up and we try to start the service, the Logic App will fail. The same is true if the service is down and we try to stop it.
These errors are of little concern. If you want to get fancy, you can poll the service for the state and perform the correct action if needed.
Summary
Azure Logic Apps allow the developer to automate cloud processing using a low code approach. Today we looked at how to turn on and turn off five different services. Four of the services had built-in tasks that just needed the forms to be filled out with the correct information. Just remember, connections are at the heart of the tasks. I suggest creating a service account (Entra Id User) with a complex password. Use this account to set up all the connections in the Logic Apps. Remember to rotate the password on the account before it expires.
The Databricks Service is not owned by Microsoft. Therefore, we had to use old school HTTP calls to start and stop the interactive cluster. This same technique can be used for services that use a bearer token. Just add more logic to log into the service and capture the returned access token which will be used in additional HTTP calls.
If you are not turning off your services in the lower environments, you are missing out on an opportunity to save money. Just look at your bill and image paying only one third of the cost!