httpclient NoHttpResponseException问题优化

1、问题

Error :
-|2021-05-13 11:06:05.937|ERROR|TID:da920806e3664802828a17e0f41add76.3517.16208751052759419|ConsumeMessageThread_18|com.xxx.operate.util.CommonUtils.uploadImg:202-调用图片服务上传图片失败！
org.apache.http.NoHttpResponseException: upload.xxxlocal.com:80 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute$original$Zpvnprgr(InternalHttpClient.java:185)
at org.apache.http.impl.client.InternalHttpClient.doExecute$original$Zpvnprgr$accessor$Mi7PpGaV(InternalHttpClient.java)
at org.apache.http.impl.client.InternalHttpClient$auxiliary$WA7Qdtlz.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at com.xxx.operate.util.CommonUtils.uploadImg(CommonUtils.java:169)
at com.xxx.operate.popup.service.PopupService.picture(PopupService.java:119)
at com.xxx.operate.popup.mq.AdSkuPictureMakeConsumer.createOrGetPicUrl(AdSkuPictureMakeConsumer.java:84)
at com.xxx.operate.popup.mq.AdSkuPictureMakeConsumer.doService(AdSkuPictureMakeConsumer.java:112)
at com.xxx.rocketmq.PushRocketMQConsumer$1$1.consumeMessage$original$ESnURphN(PushRocketMQConsumer.java:66)
at com.xxx.rocketmq.PushRocketMQConsumer$1$1.consumeMessage$original$ESnURphN$accessor$COmSrU2z(PushRocketMQConsumer.java)
at com.xxx.rocketmq.PushRocketMQConsumer$1$1$auxiliary$QJxNxUck.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
at com.xxx.rocketmq.PushRocketMQConsumer$1$1.consumeMessage(PushRocketMQConsumer.java)
at com.alibaba.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:146)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

2、分析

http长连接(keep-alive)导致的问题

由于服务端默认开启http keepalive，nginx超时配置为60s，springboot默认也为60s，而服务端response header 默认只有Connection: keep-alive无timeout

而apache httpclient 4.x版本，默认的keep-alive是读取response heade中timeout字段,没有的话默认为-1（-1代表无限）。"If the Keep-Alive header is not present in the response, HttpClient assumes the connection can be kept alive indefinitely"，详细代码见HttpClientBuilder.create().build()，里面keepalive默认实现类DefaultConnectionKeepAliveStrategy代码如下

public long getKeepAliveDuration(final HttpResponse response, final HttpContext context) {
        Args.notNull(response, "HTTP response");
        final HeaderElementIterator it = new BasicHeaderElementIterator(
                response.headerIterator(HTTP.CONN_KEEP_ALIVE));
        while (it.hasNext()) {
            final HeaderElement he = it.nextElement();
            final String param = he.getName();
            final String value = he.getValue();
            if (value != null && param.equalsIgnoreCase("timeout")) {
                try {
                    return Long.parseLong(value) * 1000;
                } catch(final NumberFormatException ignore) {
                }
            }
        }
        return -1;
    }

apache httpclient 5.x版本，已经优化

@Override
public TimeValue getKeepAliveDuration(final HttpResponse response, final HttpContext context) {
    Args.notNull(response, "HTTP response");
    final Iterator<HeaderElement> it = MessageSupport.iterate(response, HeaderElements.KEEP_ALIVE);
    while (it.hasNext()) {
        final HeaderElement he = it.next();
        final String param = he.getName();
        final String value = he.getValue();
        if (value != null && param.equalsIgnoreCase("timeout")) {
            try {
                return TimeValue.ofSeconds(Long.parseLong(value));
            } catch(final NumberFormatException ignore) {
            }
        }
    }
    final HttpClientContext clientContext = HttpClientContext.adapt(context);
    final RequestConfig requestConfig = clientContext.getRequestConfig();
    return requestConfig.getConnectionKeepAlive();
}

3、建议

自定义ConnectionKeepAliveStrategy，开启重试策略

 

参考

https://cloud.tencent.com/developer/article/1339986

https://juejin.cn/post/6844904069442568205

https://segmentfault.com/a/1190000020370495

https://www.baeldung.com/httpclient-connection-management