数据筛选和API优化

筛选数据

需求：如果数据库中存在OrderNum相同，且IsDefault不同的记录，那么IsDefault值为0的记录将替换值为1的记录（IsDefault值为1的记录不展示）。

由于查出来的数据不多，100条以内，所以我是直接全部查询到List内存中，然后在内存中进行数据过滤的操作，思来想去都觉得我如下的实现方式很low，但是我一时又没想到好的办法，不知道大家有没有好的办法？

            var newList = list.ToList();
            //筛选出哪些排序号有重复
            var orderNumList = newList.GroupBy(g => g.OrderNum).Select(g => new { orderNum = g.Key, count = g.Count() }).Where(g => g.count > 1).Select(s => s.orderNum).ToList();
            var cfList = newList.Where(w => orderNumList.Contains(w.OrderNum)); //获取有重复排序号的记录

            var cfDefaultList = cfList.Where(w => w.IsDefault);  //默认模块记录
            var cfNoDefaultList = cfList.Where(w => w.IsDefault == false); //非默认模块记录

            var intersectedList = from d in cfDefaultList join f in cfNoDefaultList on d.OrderNum equals f.OrderNum where d.IsDefault!= f.IsDefault select d;

            var newIntersectedList = intersectedList.Distinct().ToArray(); //排序号相同，既存在默认记录也存在非默认记录的数据 

                if (newIntersectedList != null && newIntersectedList.Length > 0)
                {
                    for (int i = 0; i < newIntersectedList.Length; i++)
                    {
                        if (newList.Contains(newIntersectedList[i]))
                        {
                            newList.Remove(newIntersectedList[i]);
                        }
                    }
                }
           newList = newList.OrderBy(x => x.OrderNum).ToList();

以上的newList就代码截图中的数据。

优化API接口

有一个API接口经常卡顿，而且很不稳定，快的时候2~3秒，慢的时候10秒去了。

接口需求：根据社区ID获取优惠券记录。

分析：

负责给API接口提供数据的系统中，缺少许多索引，存在许多慢查询视图。
原来的LINQ实现方式是在内存中分页，响应速度太慢。
并发请求的情况下，资源占用。

优化思路：

1、使用缓存

同一个社区的人在同一时间所看到的优惠券记录应该是一样的，而且我们应该允许脏读，我们在12306上面买火车票的时候，经常也会看到显示有票，但是下单又没有了，可能是使用了缓存，那么我们这里其实同样的可以采用缓存来缓解并发问题。

在WebAPI上面加缓存，那么又分为客户端缓存和服务器缓存。而我们知道，在ASP.NET WebForm和ASP.NET MVC中都是有页面输出缓存的，而在WebAPI中默认没有，从NuGet上面下载WebApi.OutputCache.V2，然后再API接口上添加

[CacheOutput(ClientTimeSpan = 5)]//, ServerTimeSpan = 5

我这里没法直接使用服务器输出缓存，那是因为无法捕获缓存变量参数。因为我们API接口的请求参数是string appParam，字符串类型的，它是一个json对象进过base64位编码，然后再进过url编码生成的字符串。

我们只能解析后，获取社区ID，然后根据社区ID来设置缓存，把社区ID+ pageIndex就作为缓存的Key，考虑到需要缓存的数据量很小，这里我直接使用.NET自带的缓存，引入命名空间：System.Web.Caching;

      private static System.Web.Caching.Cache ObjCache = HttpRuntime.Cache;
        /// <summary>
        /// 设置当前指定Key的Cache值,并限定过期时间 
        /// </summary>
        /// <param name="Key">缓存Key</param>
        /// <param name="Obj">缓存的值</param>
        /// <param name="TimeOuts">超时时间（秒）</param>
        public static void SetCacheSeconds(string Key, object Obj, double TimeOuts)
        {
            ObjCache.Insert(Key, Obj, null, System.DateTime.Now.AddSeconds(TimeOuts), TimeSpan.Zero);
        }
        /// <summary>
        /// 获取当前指定Key的Cache值
        /// </summary>
        /// <param name="Key">缓存Key</param>
        /// <returns>缓存的值</returns>
        public static object GetCache(string Key)
        {
            return ObjCache[Key];
        }

缓存操作类Cache完整代码如下：

/*==================================
 * Author:
 * CreateTime：2014-7-15 17:26:29
 * Description：Cache操作类
 ===================================*/
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Web.Caching;
using System.Runtime.CompilerServices;
using System.Web;
using System.Security.Policy;

namespace SSY.Util
{
    /// <summary>
    /// 缓存处理相关类
    /// </summary>
    public class Cache
    {
        private static System.Web.Caching.Cache ObjCache = HttpRuntime.Cache;
        private static short TimeOut = 720;

        #region 清除指定键值的缓存
        /// <summary>
        /// 清除指定键值的缓存
        /// </summary>
        /// <param name="Key">要清除的缓存的key值</param>
        public static void Clear(string Key)
        {
            if (ObjCache[Key] != null)
            {
                ObjCache.Remove(Key);
            }
        }
        #endregion

        #region 返回系统中缓存的个数
        /// <summary>
        /// 返回系统中缓存的个数
        /// </summary>
        /// <returns>缓存个数</returns>
        public static int Count()
        {
            return ObjCache.Count;
        }
        #endregion

        #region 获取当前指定Key的Cache值
        /// <summary>
        /// 获取当前指定Key的Cache值
        /// </summary>
        /// <param name="Key">缓存Key</param>
        /// <returns>缓存的值</returns>
        public static object GetCache(string Key)
        {
            return ObjCache[Key];
        }
        #endregion

        #region 设置当前指定Key的Cache值
        /// <summary>
        /// 设置当前指定Key的Cache值 
        /// </summary>
        /// <param name="Key">缓存Key</param>
        /// <param name="Obj">缓存的值</param>
        public static void SetCache(string Key, object Obj)
        {
            ObjCache.Insert(Key, Obj);
        }
        #endregion

        #region 设置当前指定Key的Cache值,并限定过期时间
        /// <summary>
        /// 设置当前指定Key的Cache值,并限定过期时间 
        /// </summary>
        /// <param name="Key">缓存Key</param>
        /// <param name="Obj">缓存的值</param>
        /// <param name="TimeOuts">超时时间（分钟）</param>
        public static void SetCache(string Key, object Obj, int TimeOuts)
        {
            ObjCache.Insert(Key, Obj, null, System.DateTime.Now.AddMinutes((double)TimeOuts), TimeSpan.Zero);
        }
        /// <summary>
        /// 设置当前指定Key的Cache值,并限定过期时间 
        /// </summary>
        /// <param name="Key">缓存Key</param>
        /// <param name="Obj">缓存的值</param>
        /// <param name="TimeOuts">超时时间（秒）</param>
        public static void SetCacheSeconds(string Key, object Obj, double TimeOuts)
        {
            ObjCache.Insert(Key, Obj, null, System.DateTime.Now.AddSeconds(TimeOuts), TimeSpan.Zero);
        }
        #endregion

        #region 设置当前指定Key的Cache值,依赖文件过期
        /// <summary>
        /// 设置当前指定Key的Cache值,依赖文件过期 
        /// </summary>
        /// <param name="Key">缓存Key</param>
        /// <param name="Obj">缓存的值</param>
        /// <param name="Files">相对地址，例如"~/files.xml"</param>
        public static void SetCache(string Key, object Obj, string Files)
        {
            CacheDependency cacheDep = new CacheDependency(System.Web.HttpContext.Current.Server.MapPath(Files),System.DateTime.Now);
            SetCache(Key, Obj, TimeOut, cacheDep, CacheItemPriority.High);
        }
        #endregion

        #region 设置当前指定Key的Cache值
        /// <summary>
        /// 设置当前指定Key的Cache值
        /// </summary>
        /// <param name="Key">缓存Key</param>
        /// <param name="Obj">缓存的值</param>
        /// <param name="Priority">撤销缓存的优先值，此参数的值取自枚举变量“CacheItemPriority”，优先级低的数据项将先被删除。此参数主要用在缓存退出对象时.</param>
        public static void SetCache(string Key, object Obj, CacheItemPriority Priority)
        {
            SetCache(Key, Obj, TimeOut, null, Priority);
        }
        #endregion

        #region 设置当前指定Key的Cache值
        /// <summary>
        /// 设置当前指定Key的Cache值
        /// </summary>
        /// <param name="Key">缓存Key</param>
        /// <param name="Obj">缓存的值</param>
        /// <param name="TimeOuts">一个TimeSpan，表示缓存参数将在多长时间以后被删除</param>
        /// <param name="CacheDep">缓存的依赖项,需要一个CacheDependency，可初始化一个</param>
        /// <param name="Priority">撤销缓存的优先值，此参数的值取自枚举变量“CacheItemPriority”，优先级低的数据项将先被删除。此参数主要用在缓存退出对象时</param>
        public static void SetCache(string Key, object Obj, int TimeOuts, CacheDependency CacheDep, CacheItemPriority Priority)
        {
            ObjCache.Insert(Key, Obj, CacheDep, System.DateTime.MaxValue, TimeSpan.FromHours((double)TimeOuts), Priority, null);
        }
        #endregion
    }
}

View Code

修改API接口代码：

           #region added by zouqj 2017-3-7
           var result = Util.Cache.GetCache(CommunityID+ pageIndex) as Result<List<GetAvailableCouponsModel>>;
           if (result==null) //不存在则写入缓存
           {
               //组装参数
               Dictionary<string, string> inParams = new Dictionary<string, string>();
               inParams.Add("UserId", userId);
               inParams.Add("PageIndex", pageIndex);
               inParams.Add("PageSize", pageSize);
               inParams.Add("CommunityID", CommunityID+ pageIndex);
               ...

               RequestParam RequestParam = GetRequestParam(methodName, inParams, AuthenticationId);
               var jsonContent = JsonConvert.SerializeObject(RequestParam);
               result = DoPost<List<GetAvailableCouponsModel>>(jsonContent, PostUrl);

               Util.Cache.SetCacheSeconds(CommunityID, result, 10); //写入缓存
           }
           return result;

2、改为存储过程实现

因为这个接口的业务逻辑比较复杂，之前的Linq代码写了好长一大串，获取的记录数很多，而且还是在内存中进行分页实现，所以我将原来的LINQ实现代码修改为分页存储过程实现。

存储过程代码如下：

------------------------------------------创建领券中心存储过程 created by zouqj-2017-3-1-----------------------------------
IF EXISTS(Select Name From Sysobjects Where Name='usp_GetAvailableCoupons' And Xtype='P')
DROP PROCEDURE usp_GetAvailableCoupons
GO
CREATE PROC usp_GetAvailableCouponsl
(
@PageIndex int, --页码
@PageSize int, --每页容纳的记录数
@Sort NVARCHAR(50), --排序字段及规则,不用加order by
@hostName nvarchar(100),--服务器地址
@CommunityID UNIQUEIDENTIFIER, --社区
@IsGetCount BIT --是否得到记录总数，1为得到记录总数，0为不得到记录总数，返回记录集
)
AS
-------------------------------定义变量-----------------------
declare @strSql NVARCHAR(max);
DECLARE  @dt datetime2(7) --查询时间
SET @dt=GETDATE();
set nocount on;
----------------------------------------SQL开始--------------------------------------------无分类
IF @IsGetCount=1
BEGIN

SET @strSql=N'SELECT COUNT(*) FROM (SELECT DISTINCT t.CampaignID from
(SELECT 
c.ID AS CampaignID
 FROM MK_Campaign c WITH ( NOLOCK ) INNER JOIN MK_CouponConfig f WITH ( NOLOCK ) ON c.ID=f.CampaignID
           INNER JOIN MK_Coupon p WITH ( NOLOCK ) ON f.CouponID=p.ID
           WHERE f.CouponGetType=2 AND f.ReceiveStartTime <=@dt AND f.ReceiveEndTime>= @dt
           AND c.State=4 --执行中
           AND p.WholeNetwork=1 --全网优惠
UNION ALL
SELECT
    a.[ID] AS CampaignID
    FROM  [dbo].[MK_Campaign] AS a WITH ( NOLOCK )
        INNER JOIN [dbo].[MK_CouponConfig] AS b  WITH ( NOLOCK ) ON a.[ID] = b.[CampaignID]
        AND (b.[ReceiveStartTime] <= @dt) AND (b.[ReceiveEndTime] >= @dt) AND (2 = b.[CouponGetType]) AND (4 = a.[State]) 
        INNER JOIN [dbo].[MK_Coupon] AS c WITH ( NOLOCK ) ON b.[CouponID] = c.[ID]
        INNER JOIN [dbo].[MK_CouponRestriction] AS d  WITH ( NOLOCK ) ON c.[ID] = d.[CompainID]
        LEFT OUTER JOIN [dbo].[MK_CouponRestrictCategory] AS e WITH ( NOLOCK ) ON d.[ID] = e.[CouponRestrictionID]
        LEFT OUTER JOIN [dbo].[MK_CouponRestrictionOrg] AS f WITH ( NOLOCK ) ON d.[ID] = f.[CouponRestrictionID]
        INNER JOIN [dbo].[ViewOrganizationCommunityForInterface] AS v ON f.[OrgID] = v.[ID]    
    where v.CommunityID=@CommunityID  and 1 = d.[Type]
UNION ALL
SELECT 
   a.ID AS CampaignID
   FROM MK_Campaign a  WITH ( NOLOCK ) 
             INNER JOIN MK_CouponConfig b WITH ( NOLOCK ) ON a.ID=b.CampaignID AND (b.[ReceiveStartTime] <= @dt) AND (b.[ReceiveEndTime] >=@dt) 
             INNER JOIN [dbo].[MK_Coupon] AS c WITH ( NOLOCK ) ON b.[CouponID] = c.[ID] and (2 = b.[CouponGetType]) AND (4 = a.[State]) 
             INNER JOIN [dbo].[MK_CouponRestriction] AS d WITH ( NOLOCK ) ON c.[ID] = d.[CompainID]  AND (2 = d.[Type]) 
             LEFT OUTER JOIN MK_CouponRestrictionProduct AS e WITH ( NOLOCK ) ON d.[ID] = e.[CouponRestrictionID] 
             LEFT OUTER JOIN SL_Product AS f WITH ( NOLOCK ) ON (e.ProductID = f.ID)
             INNER JOIN ViewOrganizationCommunityForInterface v ON v.ID=f.PublisherID
   where v.CommunityID=@CommunityID
   ) t
 ) AS tt'
----------------------------------------------------------------------------------
END
ELSE
BEGIN
SET @strSql=N'SELECT DISTINCT t.* from(
SELECT 
          c.ID AS CampaignID,
          c.Name AS CampaignName,
          f.ValidityStartTime AS CampaignStartTime,
          f.ValidityEndTime AS CampaignEndTime,
          p.Name AS CouponsName,
          p.Price AS CouponsAmount,
          (CASE WHEN p.IsLimited =1 THEN 1 ELSE 2 END) AS IsLimited,
          p.FullAmount AS MinAmount,
          f.ValidityEndTime AS CouponsEndTime,
          f.ValidityStartTime AS CouponsStartTime,
          (CASE WHEN f.IsRepeateGet =1 THEN 1 ELSE 2 END) AS IsCanRepeatedReceive,
          f.ReceiveAddress AS ReceiveAddress,
          f.ReceiveEndTime AS ReceiveEndTime,
          f.ReceiveStartTime AS ReceiveStartTime,
          f.ReceiveMode AS ReceiveMethod,
          f.ProvideNum AS ReceiveNum,
          p.CreateTime AS CreateTime,
          p.Price AS Price,
          f.RemainCouponNum AS RemainCouponNum,
          f.ID AS CouponConfigId,
          p.WholeNetwork AS CouponsType,
          (CASE WHEN f.IconUrl IS NULL THEN N'''' WHEN f.IconUrl=N'''' THEN N'''' ELSE @hostName+f.IconUrl END) AS IconUrl 
FROM MK_Campaign c WITH ( NOLOCK ) 
                 INNER JOIN MK_CouponConfig f WITH ( NOLOCK ) ON c.ID=f.CampaignID
                 INNER JOIN MK_Coupon p WITH ( NOLOCK ) ON f.CouponID=p.ID
          WHERE f.CouponGetType=2 AND f.ReceiveStartTime <= @dt AND f.ReceiveEndTime>= @dt AND c.State=4 AND p.WholeNetwork=1 --全网优惠
--
UNION ALL
          SELECT a.[ID] AS CampaignID,
          a.[Name] AS CampaignName, 
          b.[ValidityStartTime] AS CampaignStartTime,
          b.[ValidityEndTime] AS CampaignEndTime, 
          c.[Name] AS CouponsName, 
          c.[Price] AS CouponsAmount, 
          CASE WHEN (c.[IsLimited] = 1) THEN 1 ELSE 2 END AS IsLimited, 
          c.[FullAmount] AS MinAmount, 
          b.[ValidityEndTime] AS CouponsEndTime, 
          b.[ValidityStartTime] AS CouponsStartTime, 
          CASE WHEN (b.[IsRepeateGet] = 1) THEN 1 ELSE 2 END AS IsCanRepeatedReceive, 
          b.[ReceiveAddress] AS [ReceiveAddress], 
          b.[ReceiveEndTime] AS [ReceiveEndTime], 
          b.[ReceiveStartTime] AS [ReceiveStartTime], 
          b.[ReceiveMode] AS ReceiveMethod, 
          b.[ProvideNum] AS ReceiveNum, 
          c.[CreateTime] AS CreateTime, 
          c.[Price] AS Price, 
          b.[RemainCouponNum] AS RemainCouponNum, 
          b.[ID] AS CouponConfigId, 
          e.[Type] AS CouponsType, 
          CASE WHEN (b.[IconUrl] = N'''' OR b.[IconUrl] IS NULL) THEN N'''' ELSE @hostName+b.[IconUrl] END AS IconUrl
                    FROM  [dbo].[MK_Campaign] AS a WITH ( NOLOCK )
                        INNER JOIN [dbo].[MK_CouponConfig] AS b  WITH ( NOLOCK ) ON a.[ID] = b.[CampaignID] AND (b.[ReceiveStartTime] <= @dt) AND (b.[ReceiveEndTime] >= @dt) AND (2 = b.[CouponGetType]) AND (4 = a.[State]) 
                        INNER JOIN [dbo].[MK_Coupon] AS c WITH ( NOLOCK ) ON b.[CouponID] = c.[ID]
                        INNER JOIN [dbo].[MK_CouponRestriction] AS d  WITH ( NOLOCK ) ON c.[ID] = d.[CompainID]
                        LEFT OUTER JOIN [dbo].[MK_CouponRestrictCategory] AS e WITH ( NOLOCK ) ON d.[ID] = e.[CouponRestrictionID]
                        LEFT OUTER JOIN [dbo].[MK_CouponRestrictionOrg] AS f WITH ( NOLOCK ) ON d.[ID] = f.[CouponRestrictionID]
                        INNER JOIN [dbo].[ViewOrganizationCommunityForInterface] AS v ON f.[OrgID] = v.[ID]    
                    where v.CommunityID=@CommunityID  and 1 = d.[Type]
UNION ALL
SELECT 
       a.ID AS CampaignID,
       a.Name AS CampaignName,
       b.ValidityStartTime AS CampaignStartTime,
       b.ValidityEndTime AS CampaignEndTime,
       c.Name AS CouponsName,
       c.Price AS CouponsAmount,
       (CASE WHEN c.IsLimited=1 then 1 else 2 END) AS IsLimited,
       c.FullAmount AS MinAmount,
       b.ValidityEndTime AS CouponsEndTime,
       b.ValidityStartTime AS CouponsStartTime,
       (case when b.IsRepeateGet=1 then 1 else 2 END) AS IsCanRepeatedReceive,
       b.ReceiveAddress AS ReceiveAddress,
       b.ReceiveEndTime AS ReceiveEndTime,
       b.ReceiveStartTime AS ReceiveStartTime,
       b.ReceiveMode AS ReceiveMethod,
       b.ProvideNum AS ReceiveNum,
       c.CreateTime AS CreateTime,
       c.Price AS Price,
       b.RemainCouponNum AS RemainCouponNum,
       b.ID AS CouponConfigId,
       d.[TYPE] AS CouponsType,
       (CASE WHEN b.IconUrl IS NULL THEN N'''' WHEN b.IconUrl=N'''' THEN N'''' ELSE @hostName+b.IconUrl END) AS IconUrl
   FROM MK_Campaign a  WITH ( NOLOCK ) 
                    INNER JOIN MK_CouponConfig b WITH ( NOLOCK ) ON a.ID=b.CampaignID AND (b.[ReceiveStartTime] <= @dt) AND (b.[ReceiveEndTime] >= @dt) 
                    INNER JOIN [dbo].[MK_Coupon] AS c WITH ( NOLOCK ) ON b.[CouponID] = c.[ID] and (2 = b.[CouponGetType]) AND (4 = a.[State]) 
                    INNER JOIN [dbo].[MK_CouponRestriction] AS d WITH ( NOLOCK ) ON c.[ID] = d.[CompainID]  AND (2 = d.[Type]) 
                    LEFT OUTER JOIN MK_CouponRestrictionProduct AS e WITH ( NOLOCK ) ON d.[ID] = e.[CouponRestrictionID] 
                    LEFT OUTER JOIN SL_Product AS f WITH ( NOLOCK ) ON (e.ProductID = f.ID)
                    INNER JOIN ViewOrganizationCommunityForInterface v ON v.ID=f.PublisherID
                    where v.CommunityID=@CommunityID
   ) t
ORDER BY t.Price--@Sort
offset (@PageIndex-1)*@PageSize ROWS  FETCH NEXT @PageIndex*@PageSize ROWS ONLY'
end
--执行SQL
exec sp_executesql @strSql,N'@PageIndex int,@PageSize int,@Sort nvarchar(50),@hostName nvarchar(100),@CommunityID UNIQUEIDENTIFIER,@IsGetCount bit,@dt datetime2(7)',@PageIndex =@PageIndex,@PageSize =@PageSize,@Sort=@Sort,@hostName=@hostName,@CommunityID=@CommunityID,@IsGetCount=@IsGetCount,@dt=@dt
set nocount off;

View Code

这里需要注意的是，存储执行是，先关闭计数，set nocount on;，然后再打开set nocount off;，这样可以提升性能。还有就是使用WITH ( NOLOCK )允许脏读，提升查询效率。

这里遇到一个很诡异的问题，我使用exec sp_executesql @strSql,N'....'的方式来执行是没有问题的，而如果我使用拼接sql的方式，会报错，因为sql字符串被截断了，只截取到了4000个字符长度，即便我把字符串变量长度设置为nvarchar(max)也没用。

分页方式采用Sqlserver2012以上版本才支持的高效方式：offset ... FETCH NEXT ...ROWS ONLY

原来Linq的执行时间测试：

我修改为存储过程之后的执行时间测试：

性能差距非常明显。

posted @ 2017-03-08 10:17 邹琼俊阅读(2775) 评论(4) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

博客地址：	http://www.cnblogs.com/jiekzou/
博客版权：	本文以学习、研究和分享为主，欢迎转载，但必须在文章页面明显位置给出原文连接。如果文中有不妥或者错误的地方还望高手的你指出，以免误人子弟。如果觉得本文对你有所帮助不如【推荐】一下！如果你有更好的建议，不如留言一起讨论，共同进步！再次感谢您耐心的读完本篇文章。
其它：	.net-QQ群4：612347965 java-QQ群：805741535 H5-QQ群：773766020 我的拙作《Vue3.x+TypeScript实践指南》《ASP.NET MVC企业级实战》《H5+移动应用实战开发》《Vue.js 2.x实践指南》《JavaScript实用教程》《Node+MongoDB+React 项目实战开发》已经出版，希望大家多多支持！

数据筛选和API优化

筛选数据

优化API接口

1、使用缓存

2、改为存储过程实现

公告