io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer
- v1.0.0-latest-prod
When integrating Broadleaf microservices with external APIs (and sometimes internal services), you may occasionally encounter the following connection reset exception in your logs:
io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peerThis error typically occurs because Reactor Netty (the underlying engine for Spring’s WebClient) retains open connections in a pool to optimize performance. However, intermediate network infrastructure (such as an intermediate NAT Gateway, Load Balancer, Nginx, or an external provider’s firewall) often silently drops connections that have been idle for a specific duration.
If the client’s connection pool is unaware that the connection has been dropped and attempts to reuse it for a subsequent request, Reactor Netty leases the stale connection and transmits data over it. The load balancer or firewall then immediately drops the packet and responds with a TCP RST (Reset) packet, resulting in the "Connection reset by peer" exception.
|
Important
|
Avoid Global Timeout Configuration:
It is highly recommended not to set connection pool properties globally via properties like |
The fundamental solution is to define a client-side connection pool with a maximum idle time that is strictly shorter than the idle timeout of any intermediate load balancers or peer endpoints. Additionally, enabling background eviction ensures that expired connections are proactively cleared out of the pool rather than waiting for a request to trigger the cleanup.
Key settings to utilize within a custom ConnectionProvider:
maxIdleTime: The maximum time a connection is allowed to sit idle in the pool before being eligible for eviction. This should be set conservatively below the threshold of the intermediate network device. For example, if a vendor or load balancer has a 60-second idle limit, configure this to 45 seconds or less. If a vendor advises setting the idle time below 10 seconds, adjust this to a conservative number like 5–9 seconds.
evictInBackground: Periodically cleans up expired connections in the pool. It is recommended to configure this to proactively remove connections.
Below is an example of creating a hardened ConnectionProvider:
buildHardenedConnectionProvider code exampleprivate static ConnectionProvider buildHardenedConnectionProvider(String poolName, int maxConnections, boolean applyMaxLifeTime) {
// Define a pool that evicts connections before external or intermediate NAT Gateways close them
ConnectionProvider.Builder builder = ConnectionProvider.builder(poolName)
.maxConnections(maxConnections)
// Core fix: Evict idle connections before they are silently closed by intermediate peers
.maxIdleTime(Duration.ofSeconds(45))
// Proactive removal instead of at request time
.evictInBackground(Duration.ofSeconds(30));
if (applyMaxLifeTime) {
/* Configuring a max lifetime does not protect against intermediate NAT Gateway idle timeouts,
since it only affects idle connections that sit doing nothing for 350 seconds.
This setting forcibly closes a connection after a set time (e.g., 10 minutes) even
if it is actively being used. Since active connections are constantly sending packets,
the intermediate NAT Gateway reset timer is never reached anyway. And by setting maxIdleTime
we're evicting well before that threshold.
Nonetheless, we set this anyway for a periodic total refresh. */
builder.maxLifeTime(Duration.ofMinutes(10));
}
return builder.build();
}If you need to apply these settings to a default WebClient bean already provided by the Broadleaf framework:
Keep the Bean Name Identical: Your custom @Bean definition must use the exact same name as the framework bean to ensure it overrides the default configuration (e.g., @Bean(name = "externalWebClient")).
Auto-Configuration Ordering: Ensure that the custom @Configuration class where your override is defined is loaded ahead of the framework’s configuration class. You can achieve this by using Spring Boot’s ordering annotations, such as @AutoConfigureBefore(FrameworkWebClientAutoConfiguration.class).
As an extra layer of defense, you can also configure TCP Keep-Alive on the underlying HttpClient. This forces the operating system to send periodic TCP probes for idle connections, keeping the NAT Gateway or firewalls from dropping the mapping.
While the maxIdleTime / evictInBackground setting remains the primary fix, TCP Keep-Alive serves as a robust safety net. Note that production systems (e.g., running on Linux in cloud production environments) can leverage native Linux TCP optimizations via Netty’s Epoll transport for precise control, while local development environments can gracefully fall back.
Below is an example of applying TCP probing:
applyTcpKeepAlive code example// Enable TCP Probing to keep intermediate NAT Gateway mappings alive
public static HttpClient applyTcpKeepAlive(HttpClient client) {
// Base Keep-Alive (Works on all platforms)
client = client.option(ChannelOption.SO_KEEPALIVE, true);
if (Epoll.isAvailable()) {
// Linux-specific optimizations (Cloud Production)
return client
.option(EpollChannelOption.TCP_KEEPIDLE, 120) // Probe intermediate NAT after 2 mins idle
.option(EpollChannelOption.TCP_KEEPINTVL, 30)
.option(EpollChannelOption.TCP_KEEPCNT, 3);
} else {
// Local Dev (MacOS/Windows) Fallback
// Note: Modern Java (11+) supports setting these via NioChannelOption if needed,
// but for local dev, the defaults are usually fine as there is no intermediate NAT Gateway.
return client;
}
}Here is a complete example configuration defining a custom WebClient bean with the hardened connection provider and TCP keep-alive applied:
HardenedWebClientConfiguration example codeimport com.fasterxml.jackson.databind.ObjectMapper;
import io.netty.channel.ChannelOption;
import io.netty.channel.epoll.Epoll;
import io.netty.channel.epoll.EpollChannelOption;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.reactive.function.client.ExchangeStrategies;
import org.springframework.web.reactive.function.client.WebClient;
import org.springframework.web.util.DefaultUriBuilderFactory;
import reactor.netty.http.client.HttpClient;
import reactor.netty.resources.ConnectionProvider;
import java.time.Duration;
@Configuration
public class HardenedWebClientConfiguration {
@Bean(name = "externalWebClient")
public WebClient externalWebClient(ObjectMapper objectMapper, ExchangeStrategies strategies) {
return buildBaseWebClient("external-pool", objectMapper, strategies).build();
}
private static WebClient.Builder buildBaseWebClient(String poolName, ObjectMapper objectMapper, ExchangeStrategies strategies) {
ConnectionProvider provider = buildHardenedConnectionProvider(poolName, 50, true);
HttpClient httpClient = applyTcpKeepAlive(HttpClient.create(provider));
DefaultUriBuilderFactory uriBuilderFactory = new DefaultUriBuilderFactory();
uriBuilderFactory.setEncodingMode(DefaultUriBuilderFactory.EncodingMode.NONE);
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.uriBuilderFactory(uriBuilderFactory)
.exchangeStrategies(strategies);
}
private static ConnectionProvider buildHardenedConnectionProvider(String poolName, int maxConnections, boolean applyMaxLifeTime) {
ConnectionProvider.Builder builder = ConnectionProvider.builder(poolName)
.maxConnections(maxConnections)
.maxIdleTime(Duration.ofSeconds(45))
.evictInBackground(Duration.ofSeconds(30));
if (applyMaxLifeTime) {
builder.maxLifeTime(Duration.ofMinutes(10));
}
return builder.build();
}
public static HttpClient applyTcpKeepAlive(HttpClient client) {
client = client.option(ChannelOption.SO_KEEPALIVE, true);
if (Epoll.isAvailable()) {
return client
.option(EpollChannelOption.TCP_KEEPIDLE, 120)
.option(EpollChannelOption.TCP_KEEPINTVL, 30)
.option(EpollChannelOption.TCP_KEEPCNT, 3);
} else {
return client;
}
}
}