Скачать или смотреть How to Scrape Prometheus Metrics from Two Endpoints in the Same Kubernetes Pod

Discover the best ways to effectively `scrape metrics` from multiple endpoints within a single Kubernetes pod using Prometheus.
---
This video is based on the question https://stackoverflow.com/q/73227472/ asked by the user 'Drew Nicolette' ( https://stackoverflow.com/u/10616043/ ) and on the answer https://stackoverflow.com/a/73532742/ provided by the user 'SYN' ( https://stackoverflow.com/u/5607207/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Scrape Prometheus metrics from two endpoints in the same Kubernetes pod

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Introduction

Kubernetes is an incredibly flexible platform for deploying and managing containerized applications. However, when it comes to monitoring these applications with Prometheus, specific configurations can present challenges. A common scenario arises when you want to scrape metrics from two distinct endpoints—such as /metrics/prometheus and /metrics/executors/prometheus—that are both exposed by the same pod on the same port.

In this guide, we will explore how to leverage Prometheus's service discovery to effectively scrape these metrics from two separate endpoints within the same Kubernetes pod.

The Challenge

In a typical setup with Prometheus monitoring, one would define a job configuration that specifies which endpoints to scrape. If you have multiple metrics paths exposed by the same pod, you'll need to find a solution that allows you to differentiate between these two paths.

Example Configuration

Consider the following sample Prometheus configuration:

[[See Video to Reveal this Text or Code Snippet]]

This configuration sets up Prometheus to discover pods but does not account for scenarios where multiple metric paths exist. Hence, it becomes critical to redefine how metrics paths are scraped.

Solution: Using Service Annotations

The key to successfully scraping metrics from two endpoints lies in utilizing annotations on Services instead of Pods. By creating two separate services that each redirect to the same pod with different annotations, you can distinctly define which paths to scrape.

Step-by-Step Guide

Create Two Services:
Define two Kubernetes services - one for each metrics path.

Service for /metrics/prometheus

Service for /metrics/executors/prometheus

Define Annotations on Services:
Each service should have its own annotation specifying the metrics path:

[[See Video to Reveal this Text or Code Snippet]]

Update Prometheus Configuration:
Modify the Prometheus scraping configuration to utilize these service annotations rather than pod annotations. Change the relevant part of your prometheus.yml configuration as follows:

[[See Video to Reveal this Text or Code Snippet]]

Deploy Your Configuration:
Finally, apply your changes and ensure the services and Prometheus configuration are correctly deployed. Prometheus should now be able to scrape both metric paths seamlessly.

Conclusion

By utilizing service annotations instead of relying solely on pod annotations, you can successfully configure Prometheus to scrape multiple metric endpoints from the same Kubernetes pod. This method not only enhances the monitoring capabilities of your applications but also improves the operational efficiency of your Kubernetes deployments.

If you face any issues or need further clarification, feel free to reach out in the comments!