Скачать или смотреть How to Effectively Calculate Percentage of Failed Requests in Prometheus Queries

Learn how to divide two metrics in Prometheus to accurately calculate the percentage of failed requests per client app. This guide provides a step-by-step approach for effective PromQL usage.
---
This video is based on the question https://stackoverflow.com/q/63019337/ asked by the user 'sdc' ( https://stackoverflow.com/u/1481755/ ) and on the answer https://stackoverflow.com/a/63021669/ provided by the user 'Marcelo Ávila de Oliveira' ( https://stackoverflow.com/u/4653675/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: How to divide two by prometheus queries to calculate a percentage

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
How to Effectively Calculate Percentage of Failed Requests in Prometheus Queries

In the world of monitoring and observability, being able to analyze the health and performance of your applications is critical. One common scenario developers face is calculating the percentage of failed requests for specific client applications using Prometheus queries. This guide will guide you through this process, helping you effectively use Prometheus Query Language (PromQL) to obtain the insights you need.

The Problem: Calculating Failed Request Percentage

You have two distinct metrics that are crucial for your analysis:

Failed Requests Count: A histogram metric that records the number of requests that failed over a specific time span (for example, 10 requests failed in the last minute).

Total Requests Count: A counter that tallies every incoming request made to your application.

Your goal is to divide these two metrics to compute the percentage of failed requests per client_app. However, if your initial query only provides a single output graph rather than separate graphs for each client_app, you may find it difficult to extract the detailed insights you're after.

The Solution: Using PromQL to Divide Metrics

To calculate the percentage of failed requests per client_app, you can modify your PromQL query appropriately. Below, we'll break down the recommended approach to achieve this.

Updated Query

You can use the following PromQL query:

[[See Video to Reveal this Text or Code Snippet]]

Explanation of the Query Components

Rate Function:

rate(ignored_events_sum[5m]): This function calculates the per-second rate of failed requests observed over the last 5 minutes. It helps smooth out the spikes in failures and provides a more stable view of failure rates.

rate(total_app_events[5m]): Similarly, this function gives the per-second rate of total requests over the same period.

Aggregation:

max by (client_app): This operation groups the results by the client_app label and returns the maximum rate of failed requests for each application.

sum by (client_app): This operation combines the total request rates for each client app.

The Division: The crucial part of our query, this operation divides the computed rate of failed requests by the rate of total requests, resulting in a percentage of failures.

What to Expect from the Query

Once implemented, this query will output a graph showing the percentage of failed requests broken down by each client_app. This allows you to swiftly identify which applications are experiencing issues, facilitating faster response and resolution actions.

Conclusion

Calculating the percentage of failed requests is vital for monitoring the health of your applications. By utilizing Prometheus and PromQL effectively, you can gain critical insights into performance issues and make informed decisions. The provided query example not only gives you the desired output but also empowers you with better observability into your client applications.

With this newfound knowledge, you are now ready to tackle metrics with confidence, improving the reliability and user experience of your services.