syscall:read(..) failed: Connection reset by peer

[deepkimchi]

Actual behavior

    @Bean
    public WebClient webClient(ReutersSetting reutersSetting, ExchangeStrategies exchangeStrategies) {
            return WebClient.builder()
                        .baseUrl(ReutersEndPoints.HOST)
                        .defaultHeader(HEADER_APP_ID, reutersSetting.getApplicationId())
                        .exchangeStrategies(exchangeStrategies)
                        .build();
    }

    @Bean
    public ExchangeStrategies exchangeStrategies() {
        ObjectMapper mapper = objectMapper();
        return ExchangeStrategies
                .builder()
                .codecs(clientDefaultCodecsConfigurer -> {
                    clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonEncoder(new Jackson2JsonEncoder(mapper, MediaType.APPLICATION_JSON));
                    clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonDecoder(new Jackson2JsonDecoder(mapper, MediaType.APPLICATION_JSON));
                }).build();
    }

    public ObjectMapper objectMapper() {
        return  Jackson2ObjectMapperBuilder
                .json()
                .failOnUnknownProperties(false)
                .featuresToEnable(SerializationFeature.WRAP_ROOT_VALUE)
                .featuresToEnable(DeserializationFeature.UNWRAP_ROOT_VALUE)
                .build();
    }

WebClient initiated via above configuration, throws Exception occasionally such as below.

2019-01-07 11:33:22.188 ERROR [-,,,] 92270 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x6f488001, L:/xx.xx.xx.xx:53500 - R:api.trkd.thomsonreuters.com/xx.xx.xx.xx:443] Pooled connection observed an error

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

WebClient is called regularly under Scheduled task.

Steps to reproduce

Happen randomly

Reactor Netty version

reactor-netty:0.8.3.RELEASE

JVM version (e.g. java -version)

openjdk version "11" 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

OS version (e.g. uname -a)

Linux xxxx 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[darekurlik]

I've set ChannelOption.SO_KEEPALIVE to false to make it go away as follows:

@Bean
public WebClient webClient(final ClientHttpConnector clientHttpConnector) {
    return WebClient.builder()
        .clientConnector(clientHttpConnector)
        .build();
}

@Bean
public ClientHttpConnector clientHttpConnector(@Value("${webclient.enable-keep-alive}") final boolean keepAlive,
                                               @Value("${webclient.read-timeout-in-seconds}") final int readTimeout,
                                               @Value("${webclient.write-timeout-in-seconds}")
                                               final int writeTimeout) {
    return new ReactorClientHttpConnector(HttpClient.from(TcpClient.create()
        .option(ChannelOption.SO_KEEPALIVE, keepAlive)
        .doOnConnected(connection -> connection
            .addHandlerLast(new ReadTimeoutHandler(readTimeout))
            .addHandlerLast(new WriteTimeoutHandler(writeTimeout)))));
}

Please be aware that you are disabling keep alive connections and it may have and impact on latency.
Also check if there is nothing between your client and server that you are trying to hit. I found that any proxy or load balancer in between can complicate your life.

[qingmu-io]

I use haproxy as a pre-entry and continue to throw the same exception when the backend is reactor-netty.

[darekurlik]

We still have a lot of problems due to invalid haproxy configuration... This post helped us a lot to clear most of the issues. Unfortunately we still need to deal with problems on the health cheak. I hope it helps.

[violetagg]

@seungjincho @247687009 Can you provide tcpdump so that we can analyse why the connection is reset?

[qingmu-io]

@darekurlik Thanks!
I used the following configuration in haproxy, which is a perfect solution.

defaults
mode http
option forwardfor except 10.42.0.0/24
option prefer-last-server
balance leastconn
timeout server 600s

[deepkimchi]

@violetagg Sorry, I've been away from a while from this issue.
As it keep happening, let me prepare the dump data you asked in a near future.
Thanks

[deepkimchi]

@violetagg I'm attaching the dump and you can find [RST, ACK] in No.86
dump.zip

[violetagg]

@seungjincho Do you have some error log messages on the server and on the client for the corresponding ports?

[deepkimchi]

As I'm client side, this is only one i got..

2019-01-18 18:34:31.770 ERROR [-,,,] 4772 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x6378b7b6, L:/10.127.152.122:49600 - R:api.trkd.thomsonreuters.com/159.220.24.218:80] Pooled connection observed an error

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

[violetagg]

@seungjincho For this specific port 49600 there is nothing in the tcpdump. The only thing that I can say now is that the server closed the connection.

[antongrinenko]

I have the same issue in my spring boot project:

ERROR [reactor-http-epoll-1] [reactor.core.publisher.Operators] Operator called default onErrorDropped io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

If I can help you with some additional debug info fill free to ask.
Waiting for fix

[igorkatz]

I have the same issue:

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source) ~[netty-transport-native-unix-common-4.1.33.Final.jar!/:4.1.33.Final]

I use spring boot 2.1.3.RELEASE that brings netty 4.1.33.Final
The same version is used on the server and client side
The applications are dockerized
The problem occurs after an idle time.
If I disable the connection pool the problem disappears, but this has performance penalties

[violetagg]

@antongrinenko What's the Reactor Netty (Spring Boot) version?

@igorkatz Do you see this as an ERROR log or as a DEBUG log? Do you see this on the server or on the client?

[igorkatz]

It was ERROR on the client side.
I understood the problem. I use Swarm to orchestrate my dockerized applications. Swarm has a load balancing (LB) to implement its routing mesh. LB closes inactive connections. Enabling of keep alive for channels was not enough because it started after connection closing.
Implementation of keep alive on application level for idle connections solved the problem.