RFC: App-to-App mTLS via GoRouter

[rkoster]

Summary

This RFC proposes enabling authenticated and authorized app-to-app communication via GoRouter using mutual TLS (mTLS).

View the full RFC

Applications connect to a shared internal domain (apps.mtls.internal), where GoRouter:

• Requires and validates client certificates (Instance Identity)
• Enforces per-route access control using allowed_sources
• Follows a default-deny model (aligned with C2C network policies)

Key Points

• No new infrastructure: Uses existing GoRouter with domain-specific mTLS configuration
• Default-deny security: Routes blocked unless explicitly allowed via allowed_sources
• Internal-only domain: apps.mtls.internal is a separate domain tree, avoiding conflicts with existing apps.internal routes
• Depends on RFC-0027: Route options framework required for allowed_sources support

Implementation Phases

• Phase 1a (mTLS Infrastructure): GoRouter validates client certificates for the mTLS domain
• Phase 1b (Authorization Enforcement): Per-route access control via allowed_sources (co-requisite with 1a)
• Phase 2 (Optional): Egress HTTP proxy on port 8888 for simplified client adoption

cc @cloudfoundry/toc @cloudfoundry/wg-app-runtime-interfaces

[silvestre]

I really like this proposal.

Just to be sure: It would be still possible to have an apps.mtls.internal route allowing access for any source app, so that the authorization check could be done in the app, right?

One use-case would be in the app-autoscaler service, where we expose an mTLS endpoint but check the authorization by determining if the app is bound to an autoscaler service instance, which is dynamic information we could not determine during route creation.

[rkoster]

I really like this proposal.

Just to be sure: It would be still possible to have an apps.mtls.internal route allowing access for any source app, so that the authorization check could be done in the app, right?

One use-case would be in the app-autoscaler service, where we expose an mTLS endpoint but check the authorization by determining if the app is bound to an autoscaler service instance, which is dynamic information we could not determine during route creation.

@silvestre I have update the RFC with:

applications:
- name: autoscaler-api
  routes:
  - route: autoscaler.apps.mtls.internal
    options:
      allowed_sources:
        any: true

[theghost5800]

This idea is really interesting but will be possible to have communication to app containers on different ports or different protocol than http?

[rkoster]

This idea is really interesting but will be possible to have communication to app containers on different ports or different protocol than http?

This RFC currently focuses on HTTP traffic via GoRouter, but non-HTTP protocol support is an interesting future direction.

Current constraints:

• GoRouter uses Go's httputil.ReverseProxy which handles HTTP semantics (headers, paths, etc.)
• Caller identity is forwarded via the XFCC HTTP header, which doesn't exist for raw TCP
• GoRouter does not currently support HTTP CONNECT method for tunneling

What would be needed for non-HTTP support:

1. HTTP CONNECT tunneling in GoRouter: GoRouter would need to detect CONNECT requests, validate mTLS + allowed_sources, then hijack the connection and relay raw TCP bytes to the backend. The pattern exists (similar to WebSocket upgrades), but would require new implementation.
2. Identity forwarding challenge: Inside a TCP tunnel there's no XFCC header. Options include:

• PROXY protocol v2 (GoRouter sends client cert info as TLV before the TCP stream)
• Backend also requires mTLS and validates the client cert directly
• Application-layer identity (less secure)

3. Envoy egress proxy: The Phase 2 egress proxy (Envoy) already supports HTTP CONNECT tunneling, so apps could potentially use CONNECT backend.apps.mtls.internal:5432 to tunnel arbitrary protocols. But GoRouter still needs to support CONNECT for this to work end-to-end.

For now, this is out of scope to keep the RFC focused and achievable. But feel free to create a follow up RFC for Non-HTTP use cases.

[maxmoehl]

We need to be careful with what we add to the per-route features. Each and every option we set in there will be transmitted via the NATS bus every 20s (unless adjusted by the operator) from each app instance to each gorouter instance. Even a slight increase in message size can have quite noticeable effects on the overall bandwidth consumption.

One of the thoughts we had, is to only allow very simple rules. Specifically allow from instances of the same app, from all apps within a space, from all apps within an org. This could be a simple enum option allowed_scope: app / space / org (name is just for illustration purposes) which would keep the size within a predictable scope.

If we go for a more flexible option we must introduce strict limits on the number of bytes each user is allowed to add to each route.

[rkoster]

Thanks for raising this concern, NATS bandwidth is an important consideration.

However, I'd argue this is a generic problem that should be addressed in RFC-0027 (Generic Per-Route Features), not by limiting the functionality of individual features built on top of it. RFC-0027 already acknowledges this at line 26-27:

"Other components MAY limit the size of the map or size of keys / values for technical reasons."

The proposed allowed_scope: app | space | org enum would fundamentally change the security model:

• It only supports "relative" policies (same app/space/org as the target app)
• It doesn't support cross-boundary access (app in org A calling app in org B)
• It doesn't support specific allowlisting ("only apps X, Y, Z can call me")

These cross-boundary and specific-app use cases are core to what makes this feature valuable. For example, a shared service in one org needs to accept calls from apps in multiple other orgs - the enum approach can't express this.

Regarding limits: I agree that some limit on route options size makes sense, but it should be a single global limit on the total route options size, configured by the operator. This keeps the concern in RFC-0027 where it belongs, rather than requiring each individual route option to implement its own size restrictions.

Operators already tune bandwidth-related settings like route_emitting_interval based on their deployment characteristics. A configurable max size for route options would follow the same pattern - operators can balance flexibility vs bandwidth based on their specific needs.

Proposed path forward:

1. Add a note to RFC-0027 specifying an operator-configurable global size limit for route options
2. Keep the flexible allowed_sources design in this RFC

@maxmoehl shall I take a stab at creating a PR for the global route options size limit, or do you have fundamental concerns with this approach?

[maxmoehl]

Sounds good to me! The adjustment to RFC-0027 should target CC so that the user gets immediate feedback instead of some internal routing component which silently fails.

[rkoster]

@maxmoehl PR has been created: #1447

[cweibel]

First, I really like the idea behind this RFC.

I have a unique constraint where I need a fine grained access control at the org level on whether app-to-app mtls communications are allowed. For instance, at the platform layer, I need to enforce app-to-app mtls between organizations is not allowed, but within a space it would be, meaning you would need to be a Space Developer in both spaces.

[rkoster]

Implementation Update

Draft PRs implementing Phase 1 (1a + 1b):

• routing-release: Add mTLS app-to-app routing support (RFC draft) routing-release#535
• cloud_controller_ng: Add mTLS app-to-app routing support (RFC draft) cloud_controller_ng#4910
• capi-release: Update cloud_controller_ng for mTLS app-to-app routing (RFC draft) capi-release#625

Just a note about the PRs, I have not yet reviewed them myself, just wanted to get something functional.

Tested end-to-end on BOSH-Lite.

---

Finding: Route Options Format

RFC-0027 doesn't allow nested objects/arrays in route options. We adapted to a flat format:

// Instead of nested mtls_allowed_sources: {apps: [...]}
{"mtls_allowed_apps": "guid1,guid2", "mtls_allowed_spaces": "space-guid", "mtls_allow_any": true}

Should the RFC be updated to reflect this, or should RFC-0027 be extended?

---

Open Issue: Application Security Groups

Apps need to reach GoRouter on port 443, but default ASGs block internal IPs. Currently requires manual security group creation with router IPs.

Proposal: Auto-manage ASG via BOSH link when feature flag is enabled. This is not blocking (manual workaround exists) but improves operator experience.

[rkoster]

Also recorded a demo here: https://asciinema.org/a/zLXrO9ERP3lXqGuM, but this was before refactoring to flat options (still uses the nested structure, which is why cf curl is being used).