.Net Core结合AspNetCoreRateLimit实现限流

前言

  相信使用过WebApiThrottle的童鞋对AspNetCoreRateLimit应该不陌生,AspNetCoreRateLimit是一个ASP.NET Core速率限制的解决方案,旨在控制客户端根据IP地址或客户端ID向Web API或MVC应用发出的请求的速率。AspNetCoreRateLimit包含一个IpRateLimitMiddlewareClientRateLimitMiddleware,每个中间件可以根据不同的场景配置限制允许IP或客户端,自定义这些限制策略,也可以将限制策略应用在每​​个API URL或具体的HTTP Method上。

实践

   起初是因为新做的项目中,有天查询日志发现,对外的几个公共接口经常被“恶意”调用,考虑到接口安全性问题,增加限流策略。

  AspNetCoreRateLimit GayHub:https://github.com/stefanprodan/AspNetCoreRateLimit

根据IP进行限流

  通过nuget安装AspNetCoreRateLimit,当前版本是3.0.5,因为实际项目中用的都是分布式缓存,在这里不用内存存储,而是结合Redis进行使用,内存存储直接参考官方的Wiki就可以了。

Install-Package AspNetCoreRateLimit 

Install-Package Microsoft.Extensions.Caching.Redis

  在Startup.ConfigureServices中将服务和其他依赖注入

public void ConfigureServices(IServiceCollection services)
        {
            #region MVC
            services.AddMvc(
              options =>
              {
                  options.UseCentralRoutePrefix(new RouteAttribute("api/"));
              }
              ).SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
            #endregion

            services.AddDistributedRedisCache(options =>
            {
                options.Configuration = "127.0.0.1:6379,password=123456,connectTimeout=5000,syncTimeout=10000"; 
                options.InstanceName = "WebRatelimit";
            }); 
            //加载配置
            services.AddOptions();
            //从appsettings.json获取相应配置
            services.Configure<IpRateLimitOptions>(Configuration.GetSection("IpRateLimiting"));
            
            //注入计数器和规则存储
            services.AddSingleton<IIpPolicyStore, DistributedCacheIpPolicyStore>();
            services.AddSingleton<IRateLimitCounterStore, DistributedCacheRateLimitCounterStore>();
            
            services.AddSingleton<IHttpContextAccessor, HttpContextAccessor>();
            //配置(计数器密钥生成器)
            services.AddSingleton<IRateLimitConfiguration, RateLimitConfiguration>();
        }

  在Startup.Configure启用

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            else
            {
                app.UseHsts();
            }
            //启用限流,需在UseMvc前面
            app.UseIpRateLimiting();
            app.UseMvc();
        }

  为了不影响appsettings.json的美观吧,可以新建一个RateLimitConfig.json,并Program中启动加载中增加

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
                .UseStartup<Startup>().ConfigureAppConfiguration((host,config)=> 
                {
                    config.AddJsonFile($"RateLimitConfig.json", optional: true, reloadOnChange: true);
                });

  RateLimitConfig.json 配置如下:

{
  "IpRateLimiting": {
    //false则全局将应用限制,并且仅应用具有作为端点的规则* 。 true则限制将应用于每个端点,如{HTTP_Verb}{PATH}
    "EnableEndpointRateLimiting": true,
    //false则拒绝的API调用不会添加到调用次数计数器上
    "StackBlockedRequests": false,
   //注意这个配置,表示获取用户端的真实IP,我们的线上经过负载后是 X-Forwarded-For,而测试服务器没有,所以是X-Real-IP
"RealIpHeader": "X-Real-IP", "ClientIdHeader": "X-ClientId", "HttpStatusCode": 200, "QuotaExceededResponse": { "Content": "{{\"code\":429,\"msg\":\"访问过于频繁,请稍后重试\",\"data\":null}}", "ContentType": "application/json", "StatusCode": 200 }, "IpWhitelist": [ ], "EndpointWhitelist": [], "ClientWhitelist": [], "GeneralRules": [ { "Endpoint": "*:/api/values/test", "Period": "5s", "Limit": 3 } ] } }

  重要配置说明:

       QuotaExceededResponse 是自定义返回的内容,所以必须设置HttpStatusCodeStatusCode为200。

  GeneralRules是具体的策略,根据不同需求配置不同端点即可, Period的单位可以是s, m, h, dLimint是单位时间内的允许访问的次数;

  IpWhitelist是IP白名单,本地调试或者UAT环境,可以加入相应的IP,略过策略的限制;

       EndpointWhitelist是端点白名单,如果全局配置了访问策略,设置端点白名单相当于IP白名单一样,略过策略的限制;

       其他配置项请参考Wiki:https://github.com/stefanprodan/AspNetCoreRateLimit/wiki/IpRateLimitMiddleware#setup

Fiddler开始测试

测试接口:http://127.0.0.1:5000/api/values/Test

        [HttpGet]
        public object test()
        {
            return "ok";
        }

调用结果:

 

 调用次数和剩余调用次数在Head可以看到,(吃我一个链接:https://www.cnblogs.com/EminemJK/p/12720691.html

 

 如果调用超过策略后,调用失败,返回我们自定义的内容

 

 在Redis客户端可以看到策略的一些情况,

 其他

  通常在项目中,Authorization授权是少不了了,加入限流后,在被限流的接口调用后,限流拦截器使得跨域策略失效,故重写拦截器中间件,继承IpRateLimitMiddleware 即可:

    public class IPLimitMiddleware : IpRateLimitMiddleware
    {
        public IPLimitMiddleware(RequestDelegate next, IOptions<IpRateLimitOptions> options, IRateLimitCounterStore counterStore, IIpPolicyStore policyStore, IRateLimitConfiguration config, ILogger<IpRateLimitMiddleware> logger)
            : base(next, options, counterStore, policyStore, config, logger)
        {
        }

        public override Task ReturnQuotaExceededResponse(HttpContext httpContext, RateLimitRule rule, string retryAfter)
        {
            httpContext.Response.Headers.Append("Access-Control-Allow-Origin", "*");
            return base.ReturnQuotaExceededResponse(httpContext, rule, retryAfter);
        }
    }

  然后修改Startup.Configure

        //启用限流,需在UseMvc前面
        //app.UseIpRateLimiting();
        app.UseMiddleware<IPLimitMiddleware>();
        app.UseMvc();        

  特别需要注意的坑是,在其他文章的教程中,他们会写成:

        app.UseMiddleware<IPLimitMiddleware>().UseIpRateLimiting();//错误的演示 https://www.cnblogs.com/EminemJK/p/12720691.html

  这些写你测试的时候会发现,

X-Rate-Limit-Remaining 递减量会变成2,而正常的递减量应该是1,举栗子,配置如下:

        "Endpoint": "*:/api/values/test",
        "Period": "3s",
        "Limit": 1

表示3秒内可以访问的次数是一次,当发生调用的时候会直接返回被限制的提示,而不能正常访问接口,原因就是因为调用了两次中间件。

最后

  AspNetCoreRateLimit还可以根据客户端ID进行配置策略,具体可以看一下官方的Wiki吧。

 

#2020-06-09 遇到的BUG,并解决

2020-05-21 02:51:42,900 [10] ERROR System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.Extensions.Caching.Redis.RedisCache.RefreshAsync(String key, Nullable`1 absExpr, Nullable`1 sldExpr, CancellationToken token)
   at Microsoft.Extensions.Caching.Redis.RedisCache.GetAndRefreshAsync(String key, Boolean getData, CancellationToken token)
   at Microsoft.Extensions.Caching.Redis.RedisCache.GetAsync(String key, CancellationToken token)
   at Microsoft.Extensions.Caching.Distributed.DistributedCacheExtensions.GetStringAsync(IDistributedCache cache, String key, CancellationToken token)
   at AspNetCoreRateLimit.DistributedCacheRateLimitStore`1.GetAsync(String id, CancellationToken cancellationToken)
   at AspNetCoreRateLimit.IpRateLimitProcessor.GetMatchingRulesAsync(ClientRequestIdentity identity, CancellationToken cancellationToken)
   at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
   at DigitalCertificateSystem.Handlers.ExceptionHandlerMiddleWare.Invoke(HttpContext context)

也不清楚是什么时候会出现这个问题,在QPS达到一定的时候,访问网站非常非常慢,同一服务器的后台管理却没有问题,最终发现是这个组件的问题。于是,改造:

一、实现 IRateLimitStore<T>,引入自己的组件

/// <summary>
    /// 重写使用我们的自己的redis
    /// </summary>
    /// <typeparam name="T"></typeparam>
    public class RateLimitStore<T> : IRateLimitStore<T>
    {
        public Task SetAsync(string id, T entry, TimeSpan? expirationTime = null, CancellationToken cancellationToken = default)
        {
            return RedisUtils.StringSetAsync(RedisKEY.UserBehaviorCache, id, JsonConvert.SerializeObject(entry), expirationTime);
        }

        public async Task<bool> ExistsAsync(string id, CancellationToken cancellationToken = default)
        {
            var stored = await RedisUtils.StringGetAsync(RedisKEY.UserBehaviorCache, id);

            return !string.IsNullOrEmpty(stored);
        }

        public async Task<T> GetAsync(string id, CancellationToken cancellationToken = default)
        {
            var stored = await RedisUtils.StringGetAsync(RedisKEY.UserBehaviorCache, id);

            if (!string.IsNullOrEmpty(stored))
            {
                return JsonConvert.DeserializeObject<T>(stored);
            }

            return default;
        }

        public Task RemoveAsync(string id, CancellationToken cancellationToken = default)
        {
            return RedisUtils.KeyDeleteAsync(RedisKEY.UserBehaviorCache, id);
        }
    }
View Code

二、实现 IRateLimitCounterStore

    public class RedisCounterStore: RateLimitStore<RateLimitCounter?>, IRateLimitCounterStore
    {
        
    }
View Code

三、实现 IIpPolicyStore

    public class RedisIpPolicyStore : RateLimitStore<IpRateLimitPolicies>, IIpPolicyStore
    {
        private readonly IpRateLimitOptions _options;
        private readonly IpRateLimitPolicies _policies;
        public RedisIpPolicyStore(
            IOptions<IpRateLimitOptions> options = null,
            IOptions<IpRateLimitPolicies> policies = null)
        {
            _options = options?.Value;
            _policies = policies?.Value;
        }

         
        public async Task SeedAsync()
        {
            // on startup, save the IP rules defined in appsettings
            if (_options != null && _policies != null)
            {
                await SetAsync($"{_options.IpPolicyPrefix}", _policies).ConfigureAwait(false);
            }
        }
    }
View Code

四、修改 Startup

                services.AddOptions();
                services.Configure<IpRateLimitOptions>(Configuration.GetSection("IpRateLimiting"));
                services.AddSingleton<IIpPolicyStore, RedisIpPolicyStore>();
                services.AddSingleton<IRateLimitCounterStore, RedisCounterStore>();
                services.AddSingleton<IRateLimitConfiguration, RateLimitConfiguration>();
View Code

搞定


 2020-07-09

Redis使用的是 StackExchange.Redis,帮助类封装在  https://github.com/EminemJK/Banana

直接nuget搜索 Banana.Utility

PM>Install-Package Banana.Utility

本文已独家授权给xLong设计(ID:xlongsheji)公众号发布

posted @ 2020-04-17 16:37  山治先生  阅读(1527)  评论(18编辑  收藏