cronet: when trailers are missing, gRPC reuses the headers

[mxalbert1996]

What version of gRPC-Java are you using?

1.78.0

What is your environment?

Server: Cloudflare with HTTP/3 enabled
Client: Android 16 / gRPC with CronetChannelBuilder

What did you expect to see?

gRPC works properly.

What did you see instead?

gRPC fails with the following exception.

io.grpc.StatusException: UNKNOWN: missing GRPC status in response
	at io.grpc.Status.asException(Status.java:547)
	at io.grpc.kotlin.ClientCalls$rpcImpl$1$1$1.onClose(ClientCalls.kt:264)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:566)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:734)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:715)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

This issue doesn't always happen, and when I logged the protocols of the requests (RequestFinishedInfo.getResponseInfo().getNegotiatedProtocol()) by CronetEngine.addRequestFinishedListener(), I found that all the successful request were using h2 protocol, and all the failed requests were using quic/1+spdy/3.

Steps to reproduce the bug

Use CronetChannelBuilder to request a backend with QUIC enabled.

[mxalbert1996]

As per https://groups.google.com/g/grpc-io/c/ogqs3WWhed4/m/9y44vUk2FAAJ, QUIC is supported via Cronet in gRPC Java, but I had to disable QUIC on the CronetEngine for gRPC to work.

[ejona86]

It looks like the client received HTTP headers with 200 and correct content type, but when receiving trailers grpc-status was missing. We haven't tested with Cloudflare's HTTP/3, but it is surprising that trailers were received, just with missing grpc-status.

If you control the server, I'd suggest doing a test where the server adds metadata.put(Metadata.Key.of("trailers", Metadata.ASCII_STRING_MARSHALLER), "grpc-status") to the response headers. That's not something that I'd expect to change with HTTP/2 vs HTTP/3, but the proxy requiring the header is the easiest explanation.

This seems like something that is going to need reproduction or a frame log, and I'm not familiar with the easiest ways to do that.

[mxalbert1996]

I assume you mean "trailer" but anyway neither of them worked.
I also tried the NetLog logging of Cronet, and found that QUIC_CHROMIUM_CLIENT_STREAM_READ_RESPONSE_HEADERS was logged with the initial headers but there wasn't QUIC_CHROMIUM_CLIENT_STREAM_READ_RESPONSE_TRAILERS.
This might be Cloudflare's problem so I also asked there.

[ejona86]

Yeah, trailer. Drat.

I can't make sense of not seeing READ_RESPONSE_TRAILERS. When the error was generated it was for headers with endOfStream == true. That could be either initial headers or trailers from Cronet's perspective. But this error only happens if headersReceived was already true, which means we previously received initial headers.

If you're willing to modify gRPC for a test, dumping trailers either to a log or concatenating it to the status code may give some clue what this thing we received is:

grpc-java/core/src/main/java/io/grpc/internal/Http2ClientStreamTransportState.java

Line 200 in 024fdd0

return Status.UNKNOWN.withDescription("missing GRPC status in response");

Just for completeness, what version of Cronet are you using?

[mxalbert1996]

I did some debugging and found that Http2ClientStreamTransportState.transportTrailersReceived() is called with exactly the same metadata as initial headers. Here's the call stack.

at io.grpc.internal.Http2ClientStreamTransportState.transportTrailersReceived(Http2ClientStreamTransportState.java:173)
at io.grpc.cronet.CronetClientStream$TransportState.transportHeadersReceived(CronetClientStream.java:319)
at io.grpc.cronet.CronetClientStream$TransportState.access$2600(CronetClientStream.java:247)
at io.grpc.cronet.CronetClientStream$BidirectionalStreamCallback.reportHeaders(CronetClientStream.java:561)
at io.grpc.cronet.CronetClientStream$BidirectionalStreamCallback.onSucceeded(CronetClientStream.java:509)
at m1.mz.onSucceeded(:com.google.android.gms.dynamite_cronetdynamite@260235035@26.02.35 (260400-0):3)
...

In CronetClientStream.BidirectionalStreamCallback.onSucceeded(), if trailers are not received, it will call reportHeaders() with headers received and endOfStream = true, which will eventually call transportTrailersReceived() becuase endOfStream is true.

grpc-java/cronet/src/main/java/io/grpc/cronet/CronetClientStream.java

Lines 505 to 513 in 61d5492

	if (!haveTrailersBeenReported()) {
	if (trailerList != null) {
	reportHeaders(trailerList, true);
	} else if (info != null) {
	reportHeaders(info.getAllHeadersAsList(), true);
	} else {
	throw new AssertionError("No response header or trailer");
	}
	}

While this logic seems a little bit suspicious, I believe the real problem is that trailers are not received. I confirmed that BidirectionalStream.Callback.onResponseTrailersReceived() is not called at all.

Just for completeness, what version of Cronet are you using?

I'm using org.chromium.net:cronet-api:143.7445.0, com.google.android.gms:play-services-cronet:18.1.1 and the actual cronet version on device is 144.0.7509.3.

[ejona86]

@mxalbert1996, reportHeaders(info.getAllHeadersAsList(), true); seems like the missing piece. Yeah, that looks like it would duplicate the headers. I'm going to mark this as a bug and change the title for that to be fixed, as it seems that should be an error instead ("no trailers received").

So now the client handling makes sense. And the real underlying problem seems to be Cloudflare may be dropping the trailers when using QUIC.