A while back I wrote a post on Adjusting Pod Eviction Timings in Kubernetes. To test the changes made in that post I had to shut down nodes in an Azure Kubernetes Service cluster.
This can be done easily in the Azure portal: –
However I did a presentation recently and didn’t want to have to keep jumping into the portal from VS Code…so I wanted to be able to shut down the nodes in code.
So here’s how to use the azure-cli to shut down a node in an Azure Kubernetes Service cluster.
DISCLAIMER – the following code should only be run against a test cluster!
Firstly, to test, let’s deployment a simple application to the cluster: –
kubectl create deployment test --image=nginx
Confirm: –
kubectl get all
To check the node that the pod is running on: –
kubectl get pods -o wide
kubectl get pods -o jsonpath="{.items[0].spec.nodeName}"
Assign the node that the pod is running on to a variable (we’ll use this in a minute): –
NODE=$(kubectl get pods -o jsonpath="{.items[0].spec.nodeName}" | sed 's/.$/U&/') && echo $NODE
N.B. – the sed command at the end of the statement above is to make sure that the last character of the node name is in upper case, which is needed to get the instance ID of the VM in a later statement.
OK now we can look at shutting down the node that the pod is running on.
Nodes in AKS run in a nodepool which is a virtual machine scale set that is in a different resource group that the kubernetes cluster itself. The naming convention of that resource group is: –
MC_resourcegroup_clustername_location
The cluster in the examples here is called kubernetes1 in the resource group kubernetes in EASTUS.
So set the resource group name: –
RESOURCEGROUP="MC_kubernetes_kubernetes1_eastus"
Now we can grab the VMSS name in two ways, firstly by running: –
VMSSNAME=$(az vmss list --resource-group $RESOURCEGROUP --query "[].name" -o tsv) && echo $VMSSNAME
N.B. – AKS clusters can have multiple nodepools in which case the query above will return multiple values and won’t work.
Or we use the $NODE variable we set earlier and strip out the last few characters: –
VMSSNAME=${NODE:0:27} && echo $VMSSNAME
Once we have the NODE name and the VMSS name, we need to get the instance ID of the VM in the scale set:-
INSTANCEID=$(az vmss list-instances --name $VMSSNAME --resource-group $RESOURCEGROUP --query "[?osProfile.computerName=='$NODE'].[instanceId]" -o tsv) && echo $INSTANCEID
And now we can shut down the node: –
az vmss deallocate --name $VMSSNAME --instance-ids $INSTANCEID --resource-group $RESOURCEGROUP
Confirm that the node is offline: –
kubectl get nodes
Great! The node is offline! Our test pod didn’t have any tolerations set so it’ll take 5 minutes for a new pod to be created on a healthy node. You can check out how to adjust this in my previous post.
Finally, to restart the node: –
az vmss start --name $VMSSNAME --instance-ids $INSTANCEID --resource-group $RESOURCEGROUP
And that’s how to shutdown a node in AKS to test pod eviction!
Thanks for reading!