Zstd-数据压缩组件

Zstandard 简称Zstd，是一款快速实时的开源数据压缩程序，由Facebook开发，源码是用C语言编写的。相比业内其他压缩算法（如Gzip、Snappy、Zlib）它的特点是：当需要时，它可以将压缩速度交换为更高的压缩比率（压缩速度与压缩比率的权衡可以通过小增量来配置），反之亦

Zstd-jni

Zstd拥有丰富的API，几乎支持所有流行的编程语言，Zstd-jni 是Java中提供的API然。 它具有小数据压缩的特殊模式，称为字典压缩，可以从任何提供的样本集中构建字典。

<dependency>

    <groupId>com.github.luben</groupId>

    <artifactId>zstd-jni</artifactId>

    <version>VERSION</version>

</dependency>

 
/*
 
序列化
 
*/
 
public static <T> byte[] serialize(T result) throws IOException {
 
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(512);
 
try (OutputStream outputStream = new ZstdOutputStream(byteArrayOutputStream)) {
 
// protostuff serialize
 
ProtostuffSerializer.serialize(result, outputStream);
 
return byteArrayOutputStream.toByteArray();
 
}
 
}
 
 
 
/*
 
反序列化
 
*/
 
public static <T> T deserialize(byte[] bytes, Class<T> clazz) throws IOException {
 
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
 
try (InputStream inputStream = new ZstdInputStream(byteArrayInputStream)) {
 
// protostuff deserialize
 
return ProtostuffSerializer.deserialize(clazz, inputStream);
 
}
 
}
 
ZstdDictTrainer zstdDictTrainer = new ZstdDictTrainer(1024 * 1024, 32 * 1024);
 
// fileInput is a sample file
 
zstdDictTrainer.addSample(fileInput);
 
byte[] dic = zstdDictTrainer.trainSamples(true);
 
 
 
/*
 
Zstd's training model
 
*/
 
public static <T> byte[] serialize(T result) throws IOException {
 
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(512);
 
try (ZstdOutputStream outputStream = new ZstdOutputStream(byteArrayOutputStream)) {
 
if (dic != null) {
 
outputStream.setDict(dic);
 
}
 
// protostuff serialize
 
ProtostuffSerializer.serialize(result, outputStream);
 
return byteArrayOutputStream.toByteArray();
 
}
 
}
 
 
 
/*
 
Zstd's training model
 
*/
 
public static <T> T deserialize(byte[] bytes, Class<T> clazz) throws IOException {
 
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
 
try (ZstdInputStream inputStream = new ZstdInputStream(byteArrayInputStream)) {
 
if (dic != null) {
 
inputStream.setDict(dic);
 
}
 
// protostuff deserialize
 
return ProtostuffSerializer.deserialize(clazz, inputStream);
 
}
 
}

 

 

服务器：VM/4CPU i5-6500/CentOS 6

exp1: 4KB

	version	compress	decompress	Ratio
Gzip	jdk1.8	0.162ms	0.059ms	2.31
Snappy	1.1.7.2	0.005ms	0.002ms	1.83
Zstd	1.3.7-3	0.033ms	0.012ms	2.27
Zstd_Dic	1.3.7-3	0.020ms	0.029ms	3.81

exp2: 16KB

	version	compress	decompress	Ratio
Gzip	jdk1.8	0.279ms	0.126ms	4.29
Snappy	1.1.7.2	0.022ms	0.007ms	3.18
Zstd	1.3.7-3	0.049ms	0.002ms	4.39
Zstd_Dic	1.3.7-3	0.062ms	0.017ms	5.70

exp3: 43KB

	version	compress	decompress	Ratio
Gzip	jdk1.8	0.767ms	0.339ms	6.49
Snappy	1.1.7.2	0.111ms	0.044ms	4.36
Zstd	1.3.7-3	0.257ms	0.018ms	6.67
Zstd_Dic	1.3.7-3	0.199ms	0.061ms	8.12

exp4: 134KB

	version	compress	decompress	Ratio
Gzip	jdk1.8	1.786ms	1.026ms	13.34
Snappy	1.1.7.2	0.894ms	0.595ms	6.53
Zstd	1.3.7-3	0.411ms	0.198ms	14.74
Zstd_Dic	1.3.7-3	0.220ms	0.089ms	16.48

exp5:  654KB

	version	compress	decompress	Ratio
Gzip	jdk1.8	4.587ms	1.865ms	33.64
Snappy	1.1.7.2	2.069ms	1.430ms	8.86
Zstd	1.3.7-3	2.864ms	0.116ms	45.57
Zstd_Dic	1.3.7-3	0.426ms	0.218ms	47.38

 

 

引用链接：

https://github.com/facebook/zstd

https://github.com/luben/zstd-jni