Java QpsWorker produces many exceptions

[jtattermusch]

We've recently enabled printing output from QPS workers in the benchmark suite, but the logs are 93MB big, which makes them completely useless.
It seems that it's mostly because java worker is super chatty (tons of exceptions get printed to the console)

https://grpc-testing.appspot.com/view/Performance/job/gRPC_performance_experiment/3192/consoleFull

[ejona86]

I agree that log file is very big, but I don't see the exceptions you are talking about. Could you paste a bit of it here or say which scenarie it is in?

The most output for a test I saw was:

Oct 26, 2016 12:03:08 AM io.grpc.internal.ManagedChannelImpl <init>
INFO: [ManagedChannelImpl@677df3d4] Created with target directaddress:///grpc-performance-server-8core/10.240.0.90:38564
Oct 26, 2016 12:03:53 AM io.grpc.benchmarks.driver.LoadClient$GenericAsyncPingPongWorker$1 onClose
INFO: Error in Generic Async Ping-Pong call
Oct 26, 2016 12:03:53 AM io.grpc.internal.ManagedChannelImpl maybeTerminateChannel
INFO: [ManagedChannelImpl@677df3d4] Terminated
Oct 26, 2016 12:03:54 AM io.grpc.benchmarks.driver.LoadWorker$WorkerServiceImpl quitWorker
INFO: Received quitWorker request.
Oct 26, 2016 12:03:54 AM io.grpc.benchmarks.driver.LoadWorker main
INFO: DriverServer has terminated.

[carl-mastrangelo]

@ejona86 Its a huge number of "half closed" already called exceptions.

[jtattermusch]

Thanks!

Now the log went from 90MB to 60MB and there are still some unwanted log messages:
https://grpc-testing.appspot.com/view/Performance/job/gRPC_performance_experiment/3203/consoleFull

Oct 27, 2016 7:19:33 AM io.grpc.netty.NettyServerHandler onStreamError
WARNING: Stream Error
io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 143
    at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:127)
    at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:556)
    at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:176)
    at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
    at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:411)
    at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:245)
    at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:155)
    at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
    at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
    at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:335)
    at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:395)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1069)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:902)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:574)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:488)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    at java.lang.Thread.run(Thread.java:745)

Oct 27, 2016 7:20:25 AM io.grpc.netty.NettyServerHandler onConnectionError
WARNING: Connection Error
java.nio.channels.ClosedChannelException
    at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)

[jtattermusch]

Reopened this issue.

[ejona86]

@jtattermusch, I'm finding the performance tests far too hard to use. All I want to do is run it locally to test. I've spent hours trying to find the right script and am still pretty discontent. tools/run_tests/run_performance_tests.py -l java worked, but appears to run a ton of different things, and I want to just run one and modifying the test to shortening it. I tried running benchmarks/build/install/grpc-benchmarks/bin/benchmark_worker manually, but then couldn't figure out how to point a driver at it. I tried running qps_json_driver directly and I see corruption/uninitialized variable:

RUNNING SCENARIO: java_protobuf_async_streaming_qps_unconstrained_secure
I1027 17:38:09.551090312    2199 driver.cc:251]              Starting server on ���v�� (worker #0)
I1027 17:38:09.551406215    2199 ev_epoll_linux.c:86]        epoll engine will be using signal: 40
D1027 17:38:09.551454482    2199 ev_posix.c:106]             Using polling engine: epoll
D1027 17:38:10.080072811    2200 dns_resolver.c:186]         dns resolution failed: {"created":"@1477615090.080004673","description":"OS Error","errno":-2,"file":"src/core/lib/iomgr/resolve_address_posix.c","file_line":115,"os_error":"Name or service not known","syscall":"getaddrinfo","target_address":"\u00d8\u0019\u00e1v\u00ca\u007f"}

I remember at one point you had to use environment variables to drive some of this, but I can't find a script to get me going further in that direction.