Add TLS support to Prometheus metrics server

[jkhelil]

This PR adds native TLS/HTTPS support to the Prometheus metrics server in knative.dev/pkg/observability/metrics/prometheus, enabling secure metrics collection without requiring additional sidecars or proxies.

Problem

Many production environments require encrypted metrics endpoints for compliance and security, but the current Prometheus server only supports plain HTTP. Existing solutions have significant drawbacks:

1. kube-rbac-proxy limitations:

   • Deprecated and unmaintained (archived by Brancz in 2023)
   • Requires sidecar container for each component (15+ containers in multi-component operators)
   • Increased memory overhead and operational complexity
   • Additional maintenance burden for certificate rotation

2. Post-Quantum Cryptography (PQC) Readiness:

   • Organizations need to configure TLS settings (MinVersion, MaxVersion, CipherSuites) to prepare for PQC migration
   • Current solution provides no programmatic control over TLS configuration

Solution

Native TLS support in the Prometheus server provides:

• Zero additional containers or sidecars
• Minimal memory footprint
• Built-in support for all Knative components
• Full programmatic control over TLS configuration (PQC-ready)
• Simple certificate file-based configuration for basic use cases
• Backward compatible (plain HTTP when TLS not configured)

Changes

1. observability/metrics/prometheus/server.go (+71, -8 lines)

   • Add crypto/tls import
   • Add METRICS_TLS_CERT and METRICS_TLS_KEY constants
   • Extend Server struct with certFile, keyFile fields
   • Extend options struct with tlsConfig, certFile, keyFile
   • Update ListenAndServe() to return error and implement TLS priority logic
   • Add WithTLSConfig() and WithTLSCertFiles() server options
   • Add ContextWithTLSConfig() and TLSConfigFromContext() context helpers

2. observability/metrics/prometheus/server_test.go (+118 lines)

   • Test WithTLSConfig() option
   • Test WithTLSCertFiles() option
   • Test environment variable configuration
   • Test environment variable priority over options
   • Test context-based TLS injection
   • Test TLS priority logic

3. observability/metrics/prometheus_enabled.go (+11, -2 lines)

   • Update buildPrometheus() to use ctx context.Context (not _)
   • Retrieve TLSConfig from context if present
   • Handle error return from ListenAndServe()

/kind enhancement

Fixes #

Release Note

The Prometheus metrics server in knative.dev/pkg now supports native HTTPS with flexible TLS configuration, including programmatic control over MinVersion, MaxVersion, and CipherSuites for Post-Quantum Cryptography (PQC) readiness. This eliminates the need for sidecar proxies like kube-rbac-proxy while supporting multiple configuration methods (programmatic, file-based, environment variables, and context injection) with full backward compatibility.

Docs

[knative-prow]

@jkhelil: The label(s) kind/<kind> cannot be applied, because the repository doesn't have them.

Details

In response to this:

This PR adds native TLS/HTTPS support to the Prometheus metrics server in knative.dev/pkg/observability/metrics/prometheus, enabling secure metrics collection without requiring additional sidecars or proxies.

Problem

Many production environments require encrypted metrics endpoints for compliance and security, but the current Prometheus server only supports plain HTTP. Existing solutions have significant drawbacks:

1. kube-rbac-proxy limitations:

• Deprecated and unmaintained (archived by Brancz in 2023)
• Requires sidecar container for each component (15+ containers in multi-component operators)
• Increased memory overhead and operational complexity
• Additional maintenance burden for certificate rotation

2. Post-Quantum Cryptography (PQC) Readiness:

• Organizations need to configure TLS settings (MinVersion, MaxVersion, CipherSuites) to prepare for PQC migration
• Current solution provides no programmatic control over TLS configuration

Solution

Native TLS support in the Prometheus server provides:

• Zero additional containers or sidecars
• Minimal memory footprint
• Built-in support for all Knative components
• Full programmatic control over TLS configuration (PQC-ready)
• Simple certificate file-based configuration for basic use cases
• Backward compatible (plain HTTP when TLS not configured)

Changes

1. observability/metrics/prometheus/server.go (+71, -8 lines)

• Add crypto/tls import
• Add METRICS_TLS_CERT and METRICS_TLS_KEY constants
• Extend Server struct with certFile, keyFile fields
• Extend options struct with tlsConfig, certFile, keyFile
• Update ListenAndServe() to return error and implement TLS priority logic
• Add WithTLSConfig() and WithTLSCertFiles() server options
• Add ContextWithTLSConfig() and TLSConfigFromContext() context helpers

2. observability/metrics/prometheus/server_test.go (+118 lines)

• Test WithTLSConfig() option
• Test WithTLSCertFiles() option
• Test environment variable configuration
• Test environment variable priority over options
• Test context-based TLS injection
• Test TLS priority logic

3. observability/metrics/prometheus_enabled.go (+11, -2 lines)

• Update buildPrometheus() to use ctx context.Context (not _)

• Retrieve TLSConfig from context if present

• Handle error return from ListenAndServe()

/kind

Fixes #

Release Note

Docs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[knative-prow]

Hi @jkhelil. Thanks for your PR.

I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[knative-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jkhelil
Once this PR has been reviewed and has the lgtm label, please assign leo6leo for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

❌ Patch coverage is 79.31034% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.66%. Comparing base (9fb97f5) to head (545d035).

Files with missing lines	Patch %	Lines
observability/metrics/prometheus/server.go	83.33%	4 Missing ⚠️
observability/metrics/prometheus_enabled.go	60.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3322      +/-   ##
==========================================
+ Coverage   74.61%   74.66%   +0.04%     
==========================================
  Files         188      188              
  Lines        8215     8237      +22     
==========================================
+ Hits         6130     6150      +20     
- Misses       1844     1846       +2     
  Partials      241      241              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.