sdt
旋转门压缩算法研究（实时数据库）
实时数据库旋转门压缩算法研究
用于物联网时序数据线性压缩
1、
var strDatas = @"340 341 342 343 344 345 346 347 348 349 350 350.5 349.5 349 348 347.5 346.5 345.5 345 344 343 342 341
1 2 3 9 11 88 4 5 155
339.5 340 343 343 343 343 343.5 341 341.5 341.7 347 349 349 347 341.5 340.5 340 341.5 341 340.5 340 342 342.5
2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1
0.0998 0.1987 0.2955 0.3894 0.4794 0.5646 0.6442 0.7174 0.7833 0.8415 0.8912 0.932 0.9636";
var datas = strDatas.Split(new string[] { "\r\n", " " }, StringSplitOptions.None);
var utility = new SDTUtility();
var points = new List<Point>();
//var dateTime = DateTime.Now;
int yValue = 0;
foreach (var data in datas)
{
yValue++;
//dateTime = dateTime.AddMilliseconds(2000);
if (string.IsNullOrWhiteSpace(data))
continue;
points.Add(new Point { Y = Convert.ToDouble(data.Trim()),X= yValue});
}
var list = utility.SDTCompress(points, 0.65F);
foreach(var l in list)
Console.WriteLine(l.X+","+l.Y+" ");
Console.WriteLine("-------------------------------------------------------------------------------------");
var ori= utility.SDTUncompress(points, list);
foreach (var l in ori)
Console.WriteLine(l.X + "," + l.Y + " ");
public class SDTUtility
{
/// <summary>
/// http://www.blogjava.net/oathleo/archive/2011/09/14/358603.html
/// </summary>
/// <param name="originData"></param>
/// <param name="AccuracyE"></param>
/// <returns></returns>
public List<Point> SDTCompress(List<Point> originData, double AccuracyE)//后两个参数为其异步编程使用
{
List<Point> listSDT = new List<Point>();
//上门和下门，初始时是关着的
double upGate = -double.MaxValue;//定义上门
double downGate = double.MaxValue;//定义下门
double nowUp, nowDown;//当前数据的上下斜率
if (originData.Count <= 0)
return null;
PointContent status = new PointContent();
status.LastReadData = originData[0];//当前数据的前一个数据
status.LastStoredData = status.LastReadData;//最近保存的点
listSDT.Add(status.LastReadData);
int i = 0;
foreach (var p in originData)
{
status.CurrentReadData = p;//当前读取到的数据
nowUp = (p.Y - status.LastStoredData.Y - AccuracyE) / (p.X - status.LastStoredData.X);
//判断最大的上斜率,最小的下斜率
if (nowUp > upGate)
upGate = nowUp;
nowDown = (p.Y - status.LastStoredData.Y + AccuracyE) / (p.X - status.LastStoredData.X);
if (nowDown < downGate)
downGate = nowDown;
//不在平行四边内，即内角和大于等于180度
if (upGate >= downGate)
{
listSDT.Add(status.LastReadData);//保存前一个点
status.LastStoredData = status.LastReadData;//修改最近保存的点
upGate = (p.Y - status.LastStoredData.Y - AccuracyE) / (p.X - status.LastStoredData.X);
downGate = (p.Y - status.LastStoredData.Y + AccuracyE) / (p.X - status.LastStoredData.X);
}
status.LastReadData = p;
i++;
}
if (listSDT.Count == 1)
{
listSDT.Add(originData[originData.Count - 1]);
}
return listSDT;
}
/// <summary>
/// 线性插值解压
/// </summary>
/// <param name="originData">原始数据</param>
/// <param name="SDTData">压缩数据</param>
/// <param name="SDTData">目标值横坐标</param>
/// <returns></returns>
public List<Point> GetSingleSDTUncompress(IList<Point> SDTData,int n)
{
List<Point> UncompreeDate = new List<Point>();
Point newPoint = new Point();
if (SDTData.Count <= 1)
return null;
var before=SDTData.Where(data=>data.X<=n).LastOrDefault();
var after = SDTData.Where(data => data.X >= n).FirstOrDefault();
if (before.X == after.X)
{
UncompreeDate.Add(before);
return UncompreeDate;
}
double k = (after.Y - (before.Y)) / (after.X - before.X);
newPoint.X =n;
newPoint.Y = k * (n - before.X)
+ before.Y;
UncompreeDate.Add(newPoint);
return UncompreeDate;
}
/// <summary>
/// 线性插值解压
/// </summary>
/// <param name="originData">原始数据</param>
/// <param name="SDTData">压缩数据</param>
/// <param name="progress">进度</param>
/// <param name="cancel">取消</param>
/// <returns></returns>
public List<Point> SDTUncompress(List<Point> originData, List<Point> SDTData)
{
List<Point> UncompreeDate = new List<Point>();
Point newPoint = new Point();
int num = 0;
if (SDTData.Count <= 1)
return null;
for (int i = 0; i < SDTData.Count - 1; i++)
{
double k = (SDTData[i + 1].Y - (SDTData[i].Y)) / (SDTData[i + 1].X - (SDTData[i].X));
int startIndex = FindIndex(originData, SDTData[i].X);
int endIndex = FindIndex(originData, SDTData[i + 1].X);
for (int j = startIndex; j < endIndex; j++)
{
newPoint.X = originData[j].X;
newPoint.Y = k * (originData[j].X - SDTData[i].X)
+ SDTData[i].Y;
UncompreeDate.Add(newPoint);
num++;
}
}
if (UncompreeDate.Count < originData.Count)
{
int startIndex1 = FindIndex(originData, SDTData.LastOrDefault().X);
for (int j = startIndex1; j < originData.Count; j++)
{
newPoint.X = originData[j].X;
double k = (SDTData[SDTData.Count - 1].Y - (SDTData[SDTData.Count - 2].Y)) / (SDTData[SDTData.Count - 1].X - (SDTData[SDTData.Count - 2].X));
newPoint.Y = k * (originData[j].X - SDTData[SDTData.Count - 1].X)
+ SDTData[SDTData.Count - 1].Y;
UncompreeDate.Add(newPoint);
num++;
}
}
return UncompreeDate;
}
private int FindIndex(List<Point> list, double value)
{
int index = list.FindIndex(s => s.X == value);
return index;
}
public double ToTimestamp(DateTime value)
{
TimeSpan span = (value - new DateTime(1970, 1, 1, 0, 0, 0, 0).ToLocalTime());
return (double)span.TotalSeconds;
}
public DateTime ConvertTimestamp(double timestamp)
{
DateTime converted = new DateTime(1970, 1, 1, 0, 0, 0, 0);
DateTime newDateTime = converted.AddSeconds(timestamp);
return newDateTime.ToLocalTime();
}
}
public struct PointContent
{
public Point CurrentReadData { get; set; }//当前读取数据，当前读取到的数据
public Point LastReadData { get; set; }//上一个读取数据，当前数据的前一个数据
public Point LastStoredData { get; set; }//上一个保存数据,最近保存的点
}
public struct Point
{
public Point(double pointx, double pointy)
{
this.X = pointx;
this.Y = pointy;
}
public double X { get; set; }
public double Y { get; set; }
}
假设第一个归档数据点为A，以A到下一个数据点B之间的连线为中轴，经过AB作两边与X轴垂直的且宽度为2倍的压缩偏差精度的平行四边形，随着数据的不断更新，继续使用相同的方法，画出新的平行四边形并继续扩展。如果当前点与上一个归档值之间存在点Z落在平行四边形的外面，则将当前点压缩，上一个数据点归档保存，其他点不存档。
以下算法已经过验证
/// <summary>
/// http://www.blogjava.net/oathleo/archive/2011/09/14/358603.html
/// </summary>
/// <param name="originData"></param>
/// <param name="AccuracyE"></param>
/// <returns></returns>
public List<Point> TestSDTCompress1(List<Point> originData,PointContent contentStatus,double AccuracyE)//后两个参数为其异步编程使用
{
List<Point> sdtDataCompressed = new List<Point>();
//上门和下门，初始时是关着的
double upGate = -double.MaxValue;//定义上门
double downGate = double.MaxValue;//定义下门
double nowUp, nowDown;//当前数据的上下斜率
if (originData.Count <= 0)
return null;
//软件初始化时
if (contentStatus.LastReadData == null)
contentStatus.LastReadData = originData[0];//当前数据的前一个数据
if (contentStatus.LastStoredData == null)
contentStatus.LastStoredData = originData[0];//最近保存的点
sdtDataCompressed.Add(contentStatus.LastReadData);
int i = 0;
foreach (var currentPoint in originData)
{
contentStatus.CurrentReadData = currentPoint;//当前读取到的数据
nowUp = (currentPoint.Value - contentStatus.LastStoredData.Value - AccuracyE) / (currentPoint.Time - contentStatus.LastStoredData.Time).TotalSeconds;
//判断最大的上斜率,最小的下斜率
if (nowUp > upGate)
upGate = nowUp;
nowDown = (currentPoint.Value - contentStatus.LastStoredData.Value + AccuracyE) / (currentPoint.Time - contentStatus.LastStoredData.Time).TotalSeconds;
if (nowDown < downGate)
downGate = nowDown;
//不在平行四边内，即内角和大于等于180度
if (upGate >= downGate)
{
contentStatus.LastStoredData = contentStatus.LastReadData;//修改最近保存的点
sdtDataCompressed.Add(contentStatus.LastReadData);//保存前一个点
upGate = (currentPoint.Value - contentStatus.LastStoredData.Value - AccuracyE) / (currentPoint.Time - contentStatus.LastStoredData.Time).TotalSeconds;
downGate = (currentPoint.Value - contentStatus.LastStoredData.Value + AccuracyE) / (currentPoint.Time - contentStatus.LastStoredData.Time).TotalSeconds;
}
contentStatus.LastReadData = currentPoint;
i++;
}
//if(sdtDataCompressed.Count==1)
if(originData.Count>1)
{
contentStatus.LastStoredData =originData[originData.Count - 1];
sdtDataCompressed.Add(originData[originData.Count - 1]);
}
return sdtDataCompressed;
}
unit test:
var sdtCompressed = utility.TestSDTCompress1(points, contentStatus, 0.5F);
//if (contentStatus.LastReadData!=null)
// redis.HashSet(defaultKey + key,"lastreaddata",JsonConvert.SerializeObject(contentStatus.LastReadData));
//if(contentStatus.LastStoredData!=null)
// redis.HashSet(defaultKey + key, "laststoreddata", JsonConvert.SerializeObject(contentStatus.LastStoredData));
foreach (var sdt in sdtCompressed)
{
var document = new BsonDocument
{
{"pointid",sdt.PointId},
{"time",sdt.Time},
{"value",sdt.Value.ToString()}
};
mongoCollection.InsertOne(document);
}
浙公网安备 33010602011771号