Repository navigation
ALTS: GRPCLB LoadBalancer had an error #7643
Description
Activity
1.34.0_error.txt is a debug log of gRPC.
There is an earlier error in the log. Is that related or is that also seen with 1.33.1
Nov 19, 2020 2:25:10 AM io.grpc.grpclb.GrpclbNameResolver resolveBalancerAddresses FINE: Balancer resolution failure javax.naming.OperationNotSupportedException: DNS service refused [response code 5]; remaining name '_grpclb._tcp.metadata.google.internal.' at com.sun.jndi.dns.DnsClient.checkResponseCode(DnsClient.java:663) at com.sun.jndi.dns.DnsClient.isMatchResponse(DnsClient.java:578) at com.sun.jndi.dns.DnsClient.doUdpQuery(DnsClient.java:426) at com.sun.jndi.dns.DnsClient.query(DnsClient.java:211) at com.sun.jndi.dns.Resolver.query(Resolver.java:81) at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:434) at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:235) at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:141) at com.sun.jndi.toolkit.url.GenericURLDirContext.getAttributes(GenericURLDirContext.java:103) at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:142) at io.grpc.internal.JndiResourceResolverFactory$JndiRecordFetcher.getAllRecords(JndiResourceResolverFactory.java:216) at io.grpc.internal.JndiResourceResolverFactory$JndiResourceResolver.resolveSrv(JndiResourceResolverFactory.java:133) at io.grpc.grpclb.GrpclbNameResolver.resolveBalancerAddresses(GrpclbNameResolver.java:79) at io.grpc.grpclb.GrpclbNameResolver.doResolve(GrpclbNameResolver.java:62) at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)@dapengzhang0 any idea what changes in grpclb could have caused this?
@apolcyn
It may relate to Identity bound token enforced by gRPC LB.
The problem is only on a particular version of grpc?I think the
'_grpclb._tcp.metadata.google.internal.'query failure is expected, and just happens because we have SRV queries enabled (so the inner ALTS channel will also make the SRV query). However,metadata.google.internal.isn't a load balanced name, so that query should fail.+1 that this seems likely related to ID bound token. Can we check what the target name (https://github.com/grpc/grpc/blob/e6e6be4b0b12562e38083181aee27438d8258b5a/src/proto/grpc/gcp/handshaker.proto#L104) is that is passed to the ALTS handshake service?
I haven't seen this issue with gRPC Java prior to 1.34. I just checked that gRPC 1.33.1 doesn't have this issue. So commits after v1.33.1 most likely caused this.
I don't think Id bound token rollout in grpclb touches grpc code. It is all backend enabling. Let me check with Jianing who is responsible for Id bound token.
Although ID bound token rollout itself doesn't touch grpc code, if I'm correct it does newly require the client to pass ALTS target names and RPC authority headers on the LB channel properly, which might have changed at the client
Confirmed with Jianing, rollout only change ESF config and fallback code is still there.
I haven't seen this issue with gRPC Java prior to 1.34. I just checked that gRPC 1.33.1 doesn't have this issue. So commits after v1.33.1 most likely caused this.
Based on this, it looks like some changes in gRPC code caused the failure.
This should have been a release blocker for v1.34 since it seems to be a serious regression.
gRPCLB doesn't provide any call credentials to the server. If the server now requires it, that would definitely break gRPC. But why would older versions succeed? That's the important question.
Looking at the logs, it appears that the gRPCLB request includes a JWT. That's surprising. I wonder if the swap to ChannelCredentials somehow added it.
This is a regression caused by
ChannelCredentials. ManagedChannelImpl injects the call credentials to the existingCallCredentialsApplyingTransportFactorywhich is used by both normal and OOB connections.2 remaining items
How was this not detected by interop tests?
Created b/175066772 to follow up on this internally.
I spoke with @veblush, and the service was not DP-only, so fallback to CFE should have occurred.
I think if RPCs continued the fallback may have taken place. The initial RPCs seem to fail because the picker is immediately failed when lb RPC fails and there are no backends. Changing the logic to try fallback backends first should be possible, but it also seems it will be a hard to define/implement all the edge cases.
@apolcyn, does it seem appropriate that some RPCs failed, assuming later ones would have succeeded? It seems like we may need to change this error behavior.
This is interesting and I cannot reproduce this anymore on the same machine. Luckily I could run the exact same 1.34.0-SNAPSHOT binary which was used for the error case above and it somehow ran successfully over DirectPath.
1.34.0.log: Success log with the one which failed previously
1.34.1.log: Success log with the one with Eric's patch.This might be caused by a combination of the server and the client issue. Since I cannot reproduce this issue anymore with the same client, something which made the client fail to the GCS backend via DirectPath appears to be addressed.
- added 2 commits that reference this issue
on Dec 9, 2020 - added a commit that references this issue
on Dec 10, 2020 - added 2 commits that reference this issue
on Dec 10, 2020 - added a commit that references this issue
on Jan 15, 2021 - locked as resolved and limited conversation to collaborators
on Jun 3, 2021
When running GCS benchmark over DirectPath with gRPC 1.34.0-SNAPSHOT, it failed with the error below. It has been working with gRPC 1.33.1.
What version of gRPC-Java are you using?
gRPC 1.34.0-SNAPSHOT
What is your environment?
Linux Debian/10
What did you expect to see?
Connecting to the backend using ALTS.
What did you see instead?
GRPCLB Error
Steps to reproduce the bug