Lettuce监控LatencyUtils引发的JVM GC

最近线上其中一个服务经常内存告警,老年代不足,让运维dump了线上的服务gc日志,发现long[]占用内存总量48%左右,经过JPofiler和MAT分析,定位故障原因大致在lettuce的org.LatencyUtils.LatencyStats中:

经过jdk自带的VisualVM分析dump文件,lettuce在调用get方法时生成大量对象

在get方法中断点到lettuce的org.LatencyUtils.LatencyStats#LatencyStats(long, long, int, int, long, org.LatencyUtils.PauseDetector)方法中,count中3456个对象;
由参数highestTrackableLatency指定:


继续断点到io.lettuce.core.protocol.CommandHandler#recordLatency


发现是this.clientResources.commandLatencyCollector().isEnabled()这个配置项默认开启了监控服务,
在lettuce官网中https://lettuce.io/core/5.3.7.RELEASE/reference/index.html

翻译内容:

The client can collect latency metrics during while dispatching commands. 
Command latency metrics is collected on connection or server level. 
Command latency collection is enabled by default and can be disabled 
by setting commandLatencyCollectorOptions(…) 
to DefaultCommandLatencyCollectorOptions.disabled().

客户端可以在调度命令期间收集延迟度量。命令延迟度量是在连接或服务器级别收集的。默认情况下,命令延迟收集是启用的,
可以通过将commandLatencyCollectorOptions(…)设置为DefaultCommandLatencyCCollectorOptions.disabled()来禁用

于是网上各种搜索关闭LatencyUtils监考的配置项:最后在:这个 Redis 连接池的新监控方式针不戳~我再加一点佐料 搜到

import io.lettuce.core.event.DefaultEventPublisherOptions;
import io.lettuce.core.metrics.DefaultCommandLatencyCollectorOptions;
import io.lettuce.core.resource.DefaultClientResources;
import java.time.Duration;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * @Author:huzhiyang
 * @Date:2023/2/24 15:49
 * @Desc:
 */
@Configuration(proxyBeanMethods = false)
public class LettuceConfiguration {

    /**
     * 每5s 采集一次命令统计
     *
     * @return
     */
    @Bean
    public DefaultClientResources getDefaultClientResources() {
        DefaultClientResources build = DefaultClientResources.builder()
                //关闭监控
                .commandLatencyCollectorOptions(DefaultCommandLatencyCollectorOptions.disabled())
                .commandLatencyPublisherOptions(
                        //每 5s 采集一次命令统计
                        DefaultEventPublisherOptions.builder().eventEmitInterval(Duration.ofSeconds(5)).build())
                .build();
        return build;
    }

}

启动发现不好使,还是会进入org.LatencyUtils.LatencyStats#LatencyStats方法中,
原因是我们项目在config中配置了LettuceConnectionFactory,于是在LettuceConnectionFactory中进行配置:

private LettuceConnectionFactory getLettuceConnectionFactory(RedisConfiguration poolConfig) {
        //连接池配置
        GenericObjectPoolConfig genericObjectPoolConfig =
                new GenericObjectPoolConfig();
        genericObjectPoolConfig.setMaxIdle(redisProperties.getMaxIdle());
        genericObjectPoolConfig.setMinIdle(redisProperties.getMinIdle());
        genericObjectPoolConfig.setMaxTotal(redisProperties.getMaxActive());
        genericObjectPoolConfig.setMaxWaitMillis(redisProperties.getMaxWait());

        //redis客户端配置
        LettucePoolingClientConfiguration.LettucePoolingClientConfigurationBuilder
                builder = LettucePoolingClientConfiguration.builder().
                commandTimeout(redisProperties.getTimeout());

        ClusterTopologyRefreshOptions clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                .enablePeriodicRefresh(Duration.ofSeconds(60))// 开启周期刷新(默认60秒)
                .build();
        ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
                .topologyRefreshOptions(clusterTopologyRefreshOptions)//拓扑刷新
                .disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
                .autoReconnect(true)
                .socketOptions(SocketOptions.builder().keepAlive(true).build())
                .build();

        builder.shutdownTimeout(redisProperties.getTimeout());
        builder.poolConfig(genericObjectPoolConfig);
        builder.readFrom(ReadFrom.SLAVE_PREFERRED);
        builder.clientOptions(clusterClientOptions);
        //关闭监控配置
        builder.clientResources(DefaultClientResources.builder()
                .commandLatencyCollectorOptions(DefaultCommandLatencyCollectorOptions.disabled()).build());
        LettuceClientConfiguration lettuceClientConfiguration = builder.build();

        //根据配置和客户端配置创建连接
        LettuceConnectionFactory lettuceConnectionFactory = new
                LettuceConnectionFactory(poolConfig, lettuceClientConfiguration);
        lettuceConnectionFactory.setDatabase(redisProperties.getDatabase());
        lettuceConnectionFactory.afterPropertiesSet();
        return lettuceConnectionFactory;
    }

再次启动,发现服务好使了,通过VisualVM实时健康对比开启和关闭之后的dump文件;

关闭监控前:

关闭监控后:

posted @ 2023-02-24 17:38  748573200000  阅读(1060)  评论(0编辑  收藏  举报