微服务中处理雪崩的方案
一、什么是雪崩效应
二、雪落而不雪崩的解决方案
隔离
熔断
降级
限流
超时
三、容错
容错的三个核心思想
- 保证自己不被上游服务压垮
- 保证自己不被下游服务拖垮
- 保证自己不受外界环境影响
![]()
四、解决方案（Sentinel）
Sentinel规则
1. 流控规则
流量控制，其原理是监控流量的QPS（每秒查询率）或者并发线程数等指标
1.1流控模式
-
直接（默认）
-
关联（当关联的资源达到限流条件时，开启限流（适用于资源让步））
-
链路（当从某个接口的过来的资源达到限流时，开启限流，有点类似于针对来源配置项，区别在针对来源配置项针对上级微服务，链路流控针对于上级接口，也就是粒度最细）
如果要使链路规则生效，则版本需要2.1.1之后，并且增加配置spring.cloud.sentinel.web-context-unify=false
1.2 流控效果
- 快速失败
- warm up（预热）
- 排队等待
2. 降级规则
降级规则指的就是在满足什么条件的时候对服务进行降级
- 慢调用比例
RT：平均响应时间
时间窗口：降级持续时间
当平均响应时间 大于 输入的值的时候，进入预降级状态，如果1秒内持续进入的请求响应时间依然大于输入的值，则进入降级状态，降级时间为时间窗口输入的值 - 异常比例
当资源的每秒异常总数占总量的比例超过阈值后，就会进入降级状态 - 异常数
当1分钟之内的异常数大于异常数，则进入降级状态，注意 时间窗口需要大于60秒
3. 热点规则
热点参数流控规则是一种更细粒度的规则，它允许将规则具体细化到具体的参数上
4. 授权规则
自定义授权规则,继承 RequestOriginParse,自定义授权。
import com.alibaba.csp.sentinel.adapter.spring.webmvc.callback.RequestOriginParser;
import org.apache.commons.lang3.StringUtils;
import org.springframework.context.annotation.Configuration;
import javax.servlet.http.HttpServletRequest;
/**
* 自定义授权规则
*
* @author mrhy
* @date 2020/10/3 11:25 下午
* Copyright (C), 2018-2020
*/
@Configuration
public class MyRequestOriginParser implements RequestOriginParser {
@Override
public String parseOrigin(HttpServletRequest request) {
// 判断来源或者指定带有的规则
String origin = request.getHeader("origin");
if (StringUtils.isEmpty(origin)){
throw new RuntimeException("origin is not empty");
}
return origin;
}
}
5. 系统规则
（不推荐）
自定义异常界面
继承UrlBlockHandler（Spring cloud alibaba 2.2.0 realease 之前）
继承 BlockExceptionHandler（Spring cloud alibaba 2.2.0 realease 之后）
import com.alibaba.csp.sentinel.adapter.spring.webmvc.callback.BlockExceptionHandler;
import com.alibaba.csp.sentinel.slots.block.BlockException;
import com.alibaba.csp.sentinel.slots.block.authority.AuthorityException;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeException;
import com.alibaba.csp.sentinel.slots.block.flow.FlowException;
import com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowException;
import com.alibaba.csp.sentinel.slots.system.SystemBlockException;
import com.alibaba.fastjson.JSON;
import com.mrhy.wisdomcommon.common.ObjectResponse;
import com.mrhy.wisdomcommon.common.OperationFlag;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.MediaType;
import javax.print.attribute.standard.Media;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
/**
* 自定义sentinel界面
*
* @author mrhy
* @date 2020/10/3 11:34 下午
* Copyright (C), 2018-2020
*/
@Configuration
public class ExceptionHandlerPage implements BlockExceptionHandler {
@Override
public void handle(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse, BlockException e) throws Exception {
ObjectResponse response = null;
if (e instanceof FlowException) {
response = new ObjectResponse(OperationFlag.INTERFACE_LIMITING.getReturnCode(), OperationFlag.INTERFACE_LIMITING.getDescription());
} else if (e instanceof DegradeException) {
response = new ObjectResponse(OperationFlag.SERVICE_DEGRADATION.getReturnCode(), OperationFlag.SERVICE_DEGRADATION.getDescription());
} else if (e instanceof ParamFlowException) {
response = new ObjectResponse(OperationFlag.HOT_PARAMETER_LIMITING.getReturnCode(), OperationFlag.HOT_PARAMETER_LIMITING.getDescription());
} else if (e instanceof SystemBlockException) {
response = new ObjectResponse(OperationFlag.TRIGGER_SYSTEM_PROTECT_ROLE.getReturnCode(), OperationFlag.TRIGGER_SYSTEM_PROTECT_ROLE.getDescription());
} else if (e instanceof AuthorityException) {
response = new ObjectResponse(OperationFlag.AUTHORIZATION_RULES.getReturnCode(), OperationFlag.AUTHORIZATION_RULES.getDescription());
}
httpServletResponse.setCharacterEncoding("utf-8");
httpServletResponse.setContentType(MediaType.APPLICATION_JSON_VALUE);
httpServletResponse.getWriter().write(JSON.toJSONString(response));
}
}
状态码
import lombok.AllArgsConstructor;
import lombok.Getter;
/**
* 操作符
*
* @author mrhy
*/
@AllArgsConstructor
@Getter
public enum OperationFlag {
/**
* 成功
*/
SUCCESS(0, "操作成功"),
/**
* 失败
*/
FAIL(-1, "操作失败"),
/**
* 参数违法
*/
ILLEGAL_ARGUMENT(-2, "参数违法"),
/**
* 未登录
*/
NOT_LOGIN(401, "未登录"),
/**
* 服务降级
*/
SERVICE_DEGRADATION(100, "服务降级"),
/**
* 接口限流
*/
INTERFACE_LIMITING(101, "接口限流"),
/**
* 热电参数限流
*/
HOT_PARAMETER_LIMITING(102, "热点参数限流"),
/**
* 触发系统保护规则
*/
TRIGGER_SYSTEM_PROTECT_ROLE(103, "触发系统保护规则"),
/**
* 授权规则不通过
*/
AUTHORIZATION_RULES(104, "授权规则不通过"),
;
private final Integer returnCode;
private final String description;
}
/**
* 响应实体类
* @author mrhy
*/
public class ObjectResponse {
private Integer returnCode;
private String description;
private Object result;
public ObjectResponse() {
this.returnCode = OperationFlag.SUCCESS.getReturnCode();
this.description = OperationFlag.SUCCESS.getDescription();
}
public ObjectResponse(Integer returnCode) {
this(returnCode, (String) null);
}
public ObjectResponse(Object result) {
this.returnCode = OperationFlag.SUCCESS.getReturnCode();
this.description = OperationFlag.SUCCESS.getDescription();
this.result = result;
}
public ObjectResponse(Integer returnCode, String description) {
this(returnCode, description, (Object) null);
}
public ObjectResponse(Integer returnCode, String description, Object result) {
this.returnCode = returnCode;
this.description = description;
this.result = result;
}
public Integer getReturnCode() {
return returnCode;
}
public void setReturnCode(Integer returnCode) {
this.returnCode = returnCode;
}
public String getDescription() {
return description;
}
public void setDescription(String description) {
this.description = description;
}
public Object getResult() {
return result;
}
public void setResult(Object result) {
this.result = result;
}
}
将配置同步到nacos
引入依赖
<dependency>
<groupId>com.alibaba.csp</groupId>
<artifactId>sentinel-datasource-nacos</artifactId>
</dependency>
配置application
## ds指的数据源，可以随便写
## server-addr 指的是nacos地址
spring.cloud.sentinel.datasource.ds.nacos.server-addr=localhost:8848
## 对应配置中的data-id
spring.cloud.sentinel.datasource.ds.nacos.data-id=sentinel-wisdom-web
## 对应配置中的group-id
spring.cloud.sentinel.datasource.ds.nacos.group-id=DEFAULT_GROUP
## 对应文件类型
spring.cloud.sentinel.datasource.ds.nacos.data-type=json
## 降级规则
spring.cloud.sentinel.datasource.ds.nacos.rule-type=flow
配置nacos（配置内容）
[
{
"resource": "test", // 资源名称
"limitApp": "default", //来源应用
"grade": 1,//阈值类型 0 线程数 1QPS
"count": 5,// 单机阈值
"strategy": 0,// 流控模式 0 直接 1 关联 2 链路
"controlBehavior": 0,// 流控效果 0 快速失败 1 warmup 2 排队等待
"clusterMode": false // 是否是集群模式
}
]
将配置持久化到本地
编写配置类
import com.alibaba.csp.sentinel.command.handler.ModifyParamFlowRulesCommandHandler;
import com.alibaba.csp.sentinel.datasource.*;
import com.alibaba.csp.sentinel.init.InitFunc;
import com.alibaba.csp.sentinel.slots.block.authority.AuthorityRule;
import com.alibaba.csp.sentinel.slots.block.authority.AuthorityRuleManager;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRule;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRuleManager;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRule;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager;
import com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowRule;
import com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowRuleManager;
import com.alibaba.csp.sentinel.slots.system.SystemRule;
import com.alibaba.csp.sentinel.slots.system.SystemRuleManager;
import com.alibaba.csp.sentinel.transport.util.WritableDataSourceRegistry;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.TypeReference;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import java.io.File;
import java.io.IOException;
import java.util.List;
/**
* sentinel 配置持久化到本地
*
* @author mrhy
* @date 2020/10/4 10:17 上午
* Copyright (C), 2018-2020
*/
@Configuration
public class FilePersistence implements InitFunc {
private final String APPLICATION_NAME = "wisdom-web";
@Override
public void init() throws Exception {
String ruleDir = System.getProperty("user.home") + "/sentinel-rules/" + APPLICATION_NAME;
String flowRulePath = ruleDir + "/flow-rule.json";
String degradeRulePath = ruleDir + "/degrade-rule.json";
String systemRulePath = ruleDir + "/system-rule.json";
String authorityRulePath = ruleDir + "/authority-rule.json";
String paramFlowRulePath = ruleDir + "/param-flow-rule.json";
this.mkdirIfNotExits(ruleDir);
this.createFileIfNotExits(flowRulePath);
this.createFileIfNotExits(degradeRulePath);
this.createFileIfNotExits(systemRulePath);
this.createFileIfNotExits(authorityRulePath);
this.createFileIfNotExits(paramFlowRulePath);
// 注册一个可读数据源，用来定时读取本地的json文件，更新到规则缓存中
// 流控规则
ReadableDataSource<String, List<FlowRule>> flowRuleRDS =
new FileRefreshableDataSource<>(flowRulePath, flowRuleListParser);
// 将可读数据源注册至FlowRuleManager
// 这样当规则文件发生变化时，就会更新规则到内存
FlowRuleManager.register2Property(flowRuleRDS.getProperty());
WritableDataSource<List<FlowRule>> flowRuleWDS = new FileWritableDataSource<>(
flowRulePath,
this::encodeJson
);
// 将可写数据源注册至transport模块的WritableDataSourceRegistry中
// 这样收到控制台推送的规则时，Sentinel会先更新到内存，然后将规则写入到文件中
WritableDataSourceRegistry.registerFlowDataSource(flowRuleWDS);
// 降级规则
ReadableDataSource<String, List<DegradeRule>> degradeRuleRDS = new FileRefreshableDataSource<>(
degradeRulePath,
degradeRuleListParser
);
DegradeRuleManager.register2Property(degradeRuleRDS.getProperty());
WritableDataSource<List<DegradeRule>> degradeRuleWDS = new FileWritableDataSource<>(
degradeRulePath,
this::encodeJson
);
WritableDataSourceRegistry.registerDegradeDataSource(degradeRuleWDS);
// 系统规则
ReadableDataSource<String, List<SystemRule>> systemRuleRDS = new FileRefreshableDataSource<>(
systemRulePath,
systemRuleListParser
);
SystemRuleManager.register2Property(systemRuleRDS.getProperty());
WritableDataSource<List<SystemRule>> systemRuleWDS = new FileWritableDataSource<>(
systemRulePath,
this::encodeJson
);
WritableDataSourceRegistry.registerSystemDataSource(systemRuleWDS);
// 授权规则
ReadableDataSource<String, List<AuthorityRule>> authorityRuleRDS = new FileRefreshableDataSource<>(
authorityRulePath,
authorityRuleListParser
);
AuthorityRuleManager.register2Property(authorityRuleRDS.getProperty());
WritableDataSource<List<AuthorityRule>> authorityRuleWDS = new FileWritableDataSource<>(
authorityRulePath,
this::encodeJson
);
WritableDataSourceRegistry.registerAuthorityDataSource(authorityRuleWDS);
// 热点参数规则
ReadableDataSource<String, List<ParamFlowRule>> paramFlowRuleRDS = new FileRefreshableDataSource<>(
paramFlowRulePath,
paramFlowRuleListParser
);
ParamFlowRuleManager.register2Property(paramFlowRuleRDS.getProperty());
WritableDataSource<List<ParamFlowRule>> paramFlowRuleWDS = new FileWritableDataSource<>(
paramFlowRulePath,
this::encodeJson
);
ModifyParamFlowRulesCommandHandler.setWritableDataSource(paramFlowRuleWDS);
}
private Converter<String, List<FlowRule>> flowRuleListParser = source -> JSON.parseObject(
source,
new TypeReference<List<FlowRule>>() {
}
);
private Converter<String, List<DegradeRule>> degradeRuleListParser = source -> JSON.parseObject(
source,
new TypeReference<List<DegradeRule>>() {
}
);
private Converter<String, List<SystemRule>> systemRuleListParser = source -> JSON.parseObject(
source,
new TypeReference<List<SystemRule>>() {
}
);
private Converter<String, List<AuthorityRule>> authorityRuleListParser = source -> JSON.parseObject(
source,
new TypeReference<List<AuthorityRule>>() {
}
);
private Converter<String, List<ParamFlowRule>> paramFlowRuleListParser = source -> JSON.parseObject(
source,
new TypeReference<List<ParamFlowRule>>() {
}
);
private void mkdirIfNotExits(String filePath) throws IOException {
File file = new File(filePath);
if (!file.exists()) {
file.mkdirs();
}
}
private void createFileIfNotExits(String filePath) throws IOException {
File file = new File(filePath);
if (!file.exists()) {
file.createNewFile();
}
}
private <T> String encodeJson(T t) {
return JSON.toJSONString(t);
}
}
在resource 下 新建这个目录
META-INF.services
在这个目录下新建文件
com.alibaba.csp.sentinel.init.InitFunc
添加内容
上面写的java文件的路径 copy reference
浙公网安备 33010602011771号