-
Notifications
You must be signed in to change notification settings - Fork 2.2k
fix(horizontalpodautoscaler): Added container label to ContainerResource metrics #2836
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…rce metrics Addressing kubernetes#2403 so that metrics are distinguishable via the container label, stopping prometheus duplicate issues
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: bxrne The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
This issue is currently awaiting triage. If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the The DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Welcome @bxrne! |
|
@CatherineF-dev @mrueg should i mark this PR as stale or backlog, i understand its priority is low? |
What this PR does / why we need it:
Fixes duplicate
kube_horizontalpodautoscaler_spec_target_metricandkube_horizontalpodautoscaler_status_target_metricmetrics when multiple containers use the same resource (e.g., CPU) in a HorizontalPodAutoscaler. I also added a unit test case to verify this fix.Previously, when an HPA specified multiple ContainerResource metrics with the same resource name but different containers, kube-state-metrics would generate metrics with identical label sets. This caused Prometheus 2.52.0+ to log duplicate metric errors and drop samples:
This PR adds a
containerlabel to ContainerResource metrics, making them distinguishable:kube_horizontalpodautoscaler_spec_target_metric{metric_name="cpu",metric_target_type="utilization"}kube_horizontalpodautoscaler_spec_target_metric{metric_name="cpu",metric_target_type="utilization",container="main"}For example, an HPA with three containers (main, sidecar1, sidecar2) all scaling on CPU now produces three distinct metrics instead of duplicates.
How does this change affect the cardinality of KSM: increases
Cardinality increases slightly for HPAs using ContainerResource metrics. Each container adds one additional metric cardinality dimension. This is the intended behavior to distinguish per-container resource targets.
Which issue(s) this PR fixes:
Fixes #2403