Скачать или смотреть Resolving Inconsistent Node States in Azure AKS Clusters

Encountering issues with inconsistent node counts in Azure AKS? Learn how to diagnose and resolve state inconsistencies in your Kubernetes cluster efficiently.
---
This video is based on the question https://stackoverflow.com/q/65483044/ asked by the user 'arun' ( https://stackoverflow.com/u/1333610/ ) and on the answer https://stackoverflow.com/a/65499185/ provided by the user 'arun' ( https://stackoverflow.com/u/1333610/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Azure AKS: Inconsistent state and incorrect number of nodes in `kubectl get nodes`

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Troubleshooting Inconsistent Node States in Azure AKS Clusters

Azure Kubernetes Service (AKS) is a powerful tool for managing your containerized applications, but sometimes things can go awry. One common problem that users encounter is seeing an inconsistent state of nodes in the AKS cluster. This can manifest as a differing number of nodes reported by the scaling command and the kubectl get nodes command. In this post, we’ll explore what this inconsistency means, what might cause it, and how you can fix it.

Understanding the Problem

Imagine you're managing an AKS cluster that can scale up to 60 nodes. You successfully execute a scaling command to increase the number of nodes to 46 and everything appears to be working correctly. The command confirms the new number of nodes, reporting a "Succeeded" status:

[[See Video to Reveal this Text or Code Snippet]]

However, when checking the actual node count using:

[[See Video to Reveal this Text or Code Snippet]]

you notice that there are only 44 nodes visible, and among these, 7 nodes are in a "Ready,SchedulingDisabled" state. This discrepancy can lead to scaling issues, such as errors when you attempt to adjust the node count further.

Diagnosing the State Inconsistency

The root of this problem often lies in the state of the nodes themselves. In your case, two nodes were found to be in a corrupted state, which led to the nodes not being correctly reported in the cluster. Here are the steps to diagnose such issues effectively:

Inspect Node States: Begin by examining the state of all nodes in your cluster using:

[[See Video to Reveal this Text or Code Snippet]]

Look closely for any nodes reporting status other than "Ready".

Node Resource Group Check: Access the VM scale set in your AKS cluster's node resource group. You can find this in the Azure portal under your AKS resource. Often, discrepancies can be traced back to the underlying VM instances that comprise the node pool.

Review Errors: If you encounter specific errors when trying to scale down or interact with nodes, note these carefully. For example, an error indicating "nodes not found" suggests a mismatch between the expected node state and the actual VM state in the scale set.

Implementing a Fix

Once you've diagnosed a corrupted node state, you can take action to resolve it. Here's how to proceed:

Step 1: Deleting Corrupted Nodes

If you confirm certain nodes are in a corrupted or undesirable state, you may need to manually delete them from the VM scale set associated with your AKS cluster. Generally, you will:

Go to the Azure portal.

Navigate to Virtual Machine Scale Sets (VMSS) and select the scale set that corresponds to your AKS node pool.

Identify nodes that are not functioning properly and delete them.

Step 2: Rescaling Your Cluster

After removing the corrupted nodes, you can then scale your cluster as required. To do this, re-run your scaling command:

[[See Video to Reveal this Text or Code Snippet]]

This should now work smoothly, as the state of the remaining nodes should be consistent with your scaled configuration.

Step 3: Monitor Node Health

Keep an eye on the health of your nodes going forward. Regularly check their status using kubectl get nodes, and leverage Azure monitoring tools to alert you to any unexpected node behavior.

Conclusion

Inconsistent node states in Azure AKS can cause significant operational headaches. By understanding the problem, diagnosing the root cause, and methodically resolving it, you can maintain a healthy cluster environment. Always ensure you regularly monitor your nodes to preemptively identify such issues before they escalate.

With this deep dive, you'll be better equipped to