Скачать или смотреть Solving NGINX Upstream Timed Out (110: Operation Timed Out) in AKS Cluster

Learn how to effectively troubleshoot and resolve the `NGINX upstream timed out (110: Operation timed out)` error in Azure Kubernetes Services using proper timeout configurations.
---
This video is based on the question https://stackoverflow.com/q/73766431/ asked by the user 'basit khan' ( https://stackoverflow.com/u/12870797/ ) and on the answer https://stackoverflow.com/a/73853994/ provided by the user 'basit khan' ( https://stackoverflow.com/u/12870797/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: NGINX upstream timed out (110: Operation timed out)

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Introduction

If you have been using NGINX with AKS (Azure Kubernetes Services), you might have encountered the frustrating upstream timed out (110: Operation timed out) error in your logs. This error typically means that NGINX is not able to get a response from the upstream server within a specified timeout period. In scenarios like these, it's essential to understand why you're experiencing timeouts and how to effectively address them.

In this post, we'll explore the problem, highlight common pitfalls, and provide a comprehensive solution for ensuring your applications run smoothly in a Kubernetes environment.

Understanding the Timeout Error

The underlying problem is often linked to networking issues, configuration mismatches, or incorrect timeout settings. In this case, the user mentioned experiencing consistent timeout issues, particularly when their dotnet API runs within an AKS environment, despite functioning perfectly when deployed locally.

Key Observations

Timeout Duration: The timeout occurs precisely at 60 seconds, which is a common default in many configurations.

Different Environments: The application works locally but fails in the AKS setup.

Existing Timeout Configurations: Multiple timeout-related headers have already been set, including proxy-read-timeout and proxy-send-timeout, without resolving the issue.

Diagnosis of the Problem

After thorough investigation and adjustments to the NGINX settings, a crucial factor was identified – the configurations for handling GRPC requests which were overlooked.

Key Findings

As stated in the response, the configurations like proxy-read-timeout: "7200" and proxy-send-timeout: "7200" alone do not cater to the requirements for backend GRPC communication, which demands more explicit timeout settings.

Solution to the Timeout Issue

The resolution involves adding the right timeout configurations specifically for GRPC traffic in the NGINX setup. Follow these steps:

Step 1: Implement Server Snippet

You need to add server snippets to your NGINX configuration. This can be done by modifying your NGINX annotations or configuration files.

Example Server Snippet

[[See Video to Reveal this Text or Code Snippet]]

These directives explicitly define the timeout periods for GRPC read and send operations, as well as client body timeout, helping to alleviate issues with prolonged operations that may exceed the default settings.

Step 2: Review Existing Configurations

Double-check that you have the additional configuration related headers properly set up in the Ingress resource, including but not limited to:

nginx.ingress.kubernetes.io/proxy-read-timeout

nginx.ingress.kubernetes.io/proxy-send-timeout

nginx.ingress.kubernetes.io/client-body-timeout

Step 3: Test the Configuration

Once you've made these changes, restart your NGINX Ingress Controller and run tests to ensure the application behaves as expected without hitting the timeout issue.

Conclusion

Resolving the NGINX upstream timed out (110: Operation timed out) error can seem daunting, especially when configurations appear correct but don't yield results. By focusing on the need for specific timeout settings for GRPC communication, as demonstrated, you can effectively eliminate this common bottleneck within your Azure Kubernetes Services deployments.

Make sure to monitor your application after implementing changes, and perform relevant load tests to confirm the health of your service continues to meet performance expectations.

If you have any further questions or additional experiences to share regarding similar issues, feel free to drop a comment below!