误用.Net Redis客户端CSRedisCore，自己挖坑自己填

前导　　

　　上次Redis MQ分布式改造完成之后， 编排的容器稳定运行了一个多月，昨天突然收到ETL端同事通知，没有采集到解析日志了。

 赶紧进服务器看了一下，用于数据接收的receiver容器挂掉了， 尝试docker container start [containerid],  几分钟后该容器再次崩溃。 

 

Redis连接超限

  docker log [containerid]  查看容器日志； 重点：CSRedis.RedisException: ERR max number of clients reached

Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
      Executed action EqidManager.Controllers.EqidController.BatchPutEqidAndProfileIds (EqidReceiver) in 7.1767ms
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLPR3AP8ODKH", Request id "0HLPR3AP8ODKH:00000001": An unhandled exception was thrown by the application.
CSRedis.RedisException: ERR max number of clients reached
   at CSRedis.CSRedisClient.GetAndExecute[T](RedisClientPool pool, Func`2 handler, Int32 jump, Int32 errtimes)
   at CSRedis.CSRedisClient.ExecuteScalar[T](String key, Func`3 hander)
   at CSRedis.CSRedisClient.LPush[T](String key, T[] value)
   at RedisHelper.LPush[T](String key, T[] value)
   at EqidManager.Controllers.EqidController.BatchPutEqidAndProfileIds(List`1 eqidPairs) in /home/gitlab-runner/builds/haD2h5xC/0/webdissector/datasource/eqid-manager/src/EqidReceiver/Controllers/EqidController.cs:line 31
   at lambda_method(Closure , Object )
   at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 8.9549ms 500 
【dockerhost:6379/0】仍然不可用，下一次恢复检查时间：09/17/2019 03:11:25，错误：(ERR max number of clients reached)
【dockerhost:6379/0】仍然不可用，下一次恢复检查时间：09/17/2019 03:11:25，错误：(ERR max number of clients reached)
【dockerhost:6379/0】仍然不可用，下一次恢复检查时间：09/17/2019 03:11:25，错误：(ERR max number of clients reached)
【dockerhost:6379/0】仍然不可用，下一次恢复检查时间：09/17/2019 03:11:25，错误：(ERR max number of clients reached)【dockerhost:6379/0】仍然不可用，下一次恢复检查时间：09/17/2019 03:11:25，错误：(ERR max number of clients reached)

【dockerhost:6379/0】仍然不可用，下一次恢复检查时间：09/17/2019 03:11:25，错误：(ERR max number of clients reached)
【dockerhost:6379/0】仍然不可用，下一次恢复检查时间：09/17/2019 03:11:25，错误：(ERR max number of clients reached)
docker logs [containerid]

 日志上显示连接Redis服务器的客户端数量超限，头脑快速思考，目前编排的某容器使用CSRedisCore 对于16个Redis DB实例化了16个客户端，但Redis服务器也不至于这么不经折腾吧。

赶紧进redis.io官网搜集相关资料。

After the client is initialized, Redis checks if we are already at the limit of the number of clients that it is possible to handle simultaneously (this is configured using the maxclients configuration directive, see the next section of this document for further information).

In case it can't accept the current client because the maximum number of clients was already accepted, Redis tries to send an error to the client in order to make it aware of this condition, and closes the connection immediately. The error message will be able to reach the client even if the connection is closed immediately by Redis because the new socket output buffer is usually big enough to contain the error, so the kernel will handle the transmission of the error.

 大致意思是：Redis服务器maxclients配置了客户端最大并发连接数， 如果当前连接的客户端超限，Redis会回发一个错误消息给客户端，并迅速关闭客户端连接。

立刻登录Redis服务器查看默认配置，确认当前Redis服务器maxclients=10000（这是一个动态值，由maxclients和最大进程文件句柄决定），

# Set the max number of connected clients at the same time. By default
# this limit is set to 10000 clients, however if the Redis server is not
# able to configure the process file limit to allow for the specified limit
# the max number of allowed clients is set to the current file limit
# minus 32 (as Redis reserves a few file descriptors for internal uses).
#
# Once the limit is reached Redis will close all the new connections sending
# an error 'max number of clients reached'.
# maxclients 10000

 左图表明：通过Redis-Cli 登录进服务器立即就被踢下线。

基本可认定redis客户端使用方式有问题。

 

CSRedisCore使用方式

　继续查看相关资料，可在redis服务器上利用redis-cli命令：info clients、client list仔细分析客户端连接。

info clients 命令显示现场确实有10000的连接数；

id=2205 addr=172.16.1.3:36954 fd=1276 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=2215 addr=172.16.1.3:45923 fd=1277 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=2216 addr=172.16.1.3:44233 fd=1278 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=2217 addr=172.16.1.3:41144 fd=1279 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=2218 addr=172.16.1.3:44528 fd=1280 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=2219 addr=172.16.1.3:41626 fd=1281 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=2220 addr=172.16.1.3:39045 fd=1282 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=2221 addr=172.16.1.3:42862 fd=1283 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=2222 addr=172.16.1.3:41356 fd=1284 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=2223 addr=172.16.1.3:36076 fd=1285 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1798 addr=172.16.1.3:44865 fd=1070 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1799 addr=172.16.1.3:40315 fd=1072 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1800 addr=172.16.1.3:44051 fd=1073 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1801 addr=172.16.1.3:45183 fd=1074 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1802 addr=172.16.1.3:42352 fd=1075 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1803 addr=172.16.1.3:44401 fd=1076 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1804 addr=172.16.1.3:41325 fd=1068 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1805 addr=172.16.1.3:42309 fd=1069 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1815 addr=172.16.1.3:33341 fd=1077 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1816 addr=172.16.1.3:42546 fd=1078 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1817 addr=172.16.1.3:43985 fd=1079 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lpush
id=1818 addr=172.16.1.3:44852 fd=1080 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1819 addr=172.16.1.3:40936 fd=1081 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1820 addr=172.16.1.3:36922 fd=1082 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1821 addr=172.16.1.3:40507 fd=1083 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1822 addr=172.16.1.3:37327 fd=1084 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1823 addr=172.16.1.3:44966 fd=1085 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1825 addr=172.16.1.3:38138 fd=1086 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1836 addr=172.16.1.3:45613 fd=1087 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1837 addr=172.16.1.3:39475 fd=1089 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1838 addr=172.16.1.3:43459 fd=1090 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1839 addr=172.16.1.3:37892 fd=1088 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1840 addr=172.16.1.3:40415 fd=1092 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1841 addr=172.16.1.3:37844 fd=1093 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1842 addr=172.16.1.3:34432 fd=1091 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1843 addr=172.16.1.3:38402 fd=1094 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1844 addr=172.16.1.3:41417 fd=1095 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1845 addr=172.16.1.3:44452 fd=1096 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1856 addr=172.16.1.3:37699 fd=1097 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1857 addr=172.16.1.3:43107 fd=1098 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1858 addr=172.16.1.3:46324 fd=1099 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1859 addr=172.16.1.3:33636 fd=1100 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1860 addr=172.16.1.3:42645 fd=1101 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1861 addr=172.16.1.3:46533 fd=1102 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1862 addr=172.16.1.3:45811 fd=1103 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1863 addr=172.16.1.3:43083 fd=1104 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1864 addr=172.16.1.3:34539 fd=1105 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1865 addr=172.16.1.3:43872 fd=1106 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1874 addr=172.16.1.3:32960 fd=1107 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1875 addr=172.16.1.3:42920 fd=1109 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1876 addr=172.16.1.3:37355 fd=1110 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1877 addr=172.16.1.3:41505 fd=1111 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1878 addr=172.16.1.3:34633 fd=1112 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1879 addr=172.16.1.3:44362 fd=1114 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1882 addr=172.16.1.3:41947 fd=1108 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1883 addr=172.16.1.3:46534 fd=1113 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1884 addr=172.16.1.3:36814 fd=1115 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1885 addr=172.16.1.3:42278 fd=1116 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1893 addr=172.16.1.3:43971 fd=1120 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1894 addr=172.16.1.3:42935 fd=1122 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1896 addr=172.16.1.3:42742 fd=1119 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1898 addr=172.16.1.3:34410 fd=1117 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1899 addr=172.16.1.3:34112 fd=1118 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1900 addr=172.16.1.3:39654 fd=1121 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1901 addr=172.16.1.3:41308 fd=1123 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1902 addr=172.16.1.3:44353 fd=1125 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1904 addr=172.16.1.3:36208 fd=1126 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1905 addr=172.16.1.3:32785 fd=1124 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1915 addr=172.16.1.3:32928 fd=1127 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=1916 addr=172.16.1.3:43645 fd=1128 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
client list命令显示连接如下：

官方对client list命令输出字段的解释：

  • addr: The client address, that is, the client IP and the remote port number it used to connect with the Redis server.

  • fd: The client socket file descriptor number.

  • name: The client name as set by CLIENT SETNAME.

  • age: The number of seconds the connection existed for.

  • idle: The number of seconds the connection is idle.

  • flags: The kind of client (N means normal client, check the full list of flags).

  • omem: The amount of memory used by the client for the output buffer.

  • cmd: The last executed command.

根据以上解释，表明 Redis服务器收到很多ip=172.16.1.3（故障容器在网桥内的Ip 地址）的客户端连接，这些连接最后发出的是ping命令（这是一个测试命令）

故障容器使用的Redis客户端是CSRedisCore，该客户端只是单纯将 Msg 写入Redis list 数据结构，CSRedisCore上相关github issue给了我一些启发。

发现自己将CSRedisClient实例化代码写在 .netcore api Controller构造函数，这样每次请求构造Controller时都实例化一次Redis客户端，最终Redis客户端连接数达到最大允许连接值。

依赖注入三种模式：　单例（系统内单一实例，一次性注入）；瞬态（每次请求产生实例并注入）；自定义范围。

有关dotnet apicontroller 以瞬态模式 注入，请查阅链接。

还有一个疑问？ 为什么Redis服务器没有释放空闲的 客户端连接，如果空闲连接被释放了，即使我写了low代码也不至于如此吧？

查询官方：

By default recent versions of Redis don't close the connection with the client if the client is idle for many seconds: the connection will remain open forever.

However if you don't like this behavior, you can configure a timeout, so that if the client is idle for more than the specified number of seconds, the client connection will be closed.

You can configure this limit via redis.conf or simply using CONFIG SET timeout <value>.

大致意思是最近的Redis服务端版本默认不会释放空闲的客户端连接：

# Close the connection after a client is idle for N seconds (0 to disable)
timeout 0

可通过修改Redis配置释放 空闲客户端连接。

我们最佳实践当然不是修改Redis idle timeout 配置，问题核心还是因为我实例化了多客户端，赶紧将CSRedisCore实例化代码移到startup.cs并注册为单例。

大胆求证

info clients命令显示稳定在53个Redis连接。

client list命令显示：172.16.1.3（故障容器）建立了50个客户端连接，编排的另一个容器webapp建立了2个连接，redis-cli命令登录到服务器建立了1个连接。

127.0.0.1:6379> client list
id=20409 addr=172.16.1.3:44834 fd=18 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20410 addr=172.16.1.3:39881 fd=20 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20411 addr=172.16.1.3:42756 fd=17 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20419 addr=172.16.1.3:46224 fd=21 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20423 addr=172.16.1.3:34748 fd=28 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20424 addr=172.16.1.3:37483 fd=22 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20425 addr=172.16.1.3:44064 fd=29 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20426 addr=172.16.1.3:43993 fd=25 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20427 addr=172.16.1.3:34092 fd=24 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20428 addr=172.16.1.3:35347 fd=27 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20429 addr=172.16.1.3:46126 fd=30 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20430 addr=172.16.1.3:42627 fd=23 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20431 addr=172.16.1.3:35098 fd=26 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20442 addr=172.16.1.3:34471 fd=31 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20443 addr=172.16.1.3:35092 fd=32 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20444 addr=172.16.1.3:46168 fd=33 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20445 addr=172.16.1.3:42879 fd=34 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20446 addr=172.16.1.3:46627 fd=35 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20447 addr=172.16.1.3:44731 fd=36 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20448 addr=172.16.1.3:36705 fd=37 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20449 addr=172.16.1.3:38668 fd=38 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20450 addr=172.16.1.3:45484 fd=39 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20451 addr=172.16.1.3:40802 fd=40 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20459 addr=172.16.1.3:36973 fd=45 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20460 addr=172.16.1.3:37814 fd=41 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20462 addr=172.16.1.3:44642 fd=44 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20463 addr=172.16.1.3:35272 fd=42 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20465 addr=172.16.1.3:42843 fd=47 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20466 addr=172.16.1.3:46785 fd=48 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20468 addr=172.16.1.3:38481 fd=49 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20504 addr=127.0.0.1:40902 fd=60 name= age=1478 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 qbuf-free=32742 obl=0 oll=0 omem=0 events=r cmd=client
id=20469 addr=172.16.1.3:45822 fd=50 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20470 addr=172.16.1.3:37211 fd=43 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20471 addr=172.16.1.3:39386 fd=46 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20481 addr=172.16.1.3:37346 fd=51 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20482 addr=172.16.1.3:42387 fd=52 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20483 addr=172.16.1.3:41523 fd=53 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20484 addr=172.16.1.3:37088 fd=54 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20485 addr=172.16.1.3:41371 fd=55 name=receiver age=73384 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lpush
id=20486 addr=172.16.1.3:34362 fd=56 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20487 addr=172.16.1.3:45409 fd=57 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20488 addr=172.16.1.3:36119 fd=58 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20489 addr=172.16.1.3:46631 fd=59 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20389 addr=172.16.1.3:42971 fd=8 name=receiver age=73387 idle=73387 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20392 addr=172.16.1.4:42699 fd=11 name=f176f125a4c5 age=73386 idle=4 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20393 addr=172.16.1.4:40179 fd=12 name=f176f125a4c5 age=73386 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=rpop
id=20400 addr=172.16.1.3:36255 fd=10 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20401 addr=172.16.1.3:36118 fd=9 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20402 addr=172.16.1.3:42346 fd=13 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20403 addr=172.16.1.3:40437 fd=14 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20404 addr=172.16.1.3:37910 fd=15 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20405 addr=172.16.1.3:35374 fd=16 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=20408 addr=172.16.1.3:34197 fd=19 name=receiver age=73384 idle=73384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pin
client list命令显示连接

那么问题来了，修改之后，receiver容器为什么还稳定建立了50个redis连接？

进一步与CSRedisCore原作者沟通，确定CSRedisCore有预热机制，默认在连接池中预热了５０个连接。

bingo，故障和困惑全部排查清楚。

 

总结

经此一役，在使用CSRedisCore客户端时，　要深入理解

①  本例我自己用在controller实例化csrediscore客户端，导致每次请求均建立了redis连接 ，

② 虽然很多redis连接也许并没有发出实质性的redis命令， 但是由于redis服务器默认不会释放连接， 日积月累redis服务器到达客户端最大连接数之后，打崩了服务器。

 --- 

③ 一般情况下，对于redis这样的高频服务， 客户端和服务端都会建立连接池。

    -  redis 默认的客户端最大并发连接数是10000，连接池中连接是触发机制，从0开始上涨，并且默认不会释放连接。

   -  客户端CSRedisCore会默认建立连接池，但是会预热50个连接， 开发者心里要有数。

 

posted @ 2019-09-18 14:13  博客猿马甲哥  阅读(9631)  评论(26)  编辑  收藏  举报