在SpringBoot当中集合ElasticSearch进行搜索
前言
在学习easyLive项目当中接触到了elasticSearch中间件来进行搜索.本随笔记录一下项目中用到的操作. 这里就省略掉一些es的基础介绍和为什么要使用es了.
1. 准备阶段
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
在这里使用的是SpringData集合的elasticSearch,初始化方法和直接引入highLevelRestClient有所区别(新版本已经废弃掉这个方法了,现在还没学).
创建Config类,这里我们可以参考SpringData文档 中的操作来对es初始化,创建一个Config类
EsConfiguration
@Configuration
public class EsConfiguration extends AbstractElasticsearchConfiguration implements DisposableBean {
private RestHighLevelClient client;
@Resource
private AppConfig appConfig;
@Override
public void destroy() throws Exception {
if (client != null) {
client.close();
}
}
@Override
public RestHighLevelClient elasticsearchClient() {
final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(appConfig.getEsHostPort())
.build();
client = RestClients.create(clientConfiguration).rest();
return client;
}
}
AppConfig类
//省去了getter和setter方法来节省空间,之后的随笔也是这样
@Configuration
public class AppConfig {
@Value("${project.folder:}")
private String projectFolder;
@Value("${admin.account:}")
private String adminAccount;
@Value("${admin.password:}")
private String adminPassword;
@Value("${showFFmpegLog:true}")
private boolean showFFmpegLog;
//这里还没琢磨过有没有AutoConfig类能直接在application.yml里直接配置,之后再看吧,其实这里写yml配置文件里更好,这样子docker部署的时候可以通过外置挂载卷来修改es地址
@Value("${es.host.port:127.0.0.1:9200}")
private String esHostPort;
@Value("${ex.index.video.name:easylive_video}")
private String esIndexVideoName;
}
在每次启动的时候初始化es而不是在es里好了再找,这样子在换运行环境的时候可以方便很多
所以我们创建一个esComponent
@Component("esSearchComponent")
public class EsSearchComponent {
@Resource
private AppConfig appConfig;
@Resource
private RestHighLevelClient restHighLevelClient;
private Boolean isExsitIndex() throws IOException {
GetIndexRequest getIndexRequest = new GetIndexRequest(appConfig.getEsIndexVideoName());
return restHighLevelClient.indices().exists(getIndexRequest, RequestOptions.DEFAULT);
}
public void createIndex() {
try {
if (isExsitIndex()) {
return;
}
CreateIndexRequest request = new CreateIndexRequest(appConfig.getEsIndexVideoName());
request.settings("{\"analysis\": {\n" +
" \"analyzer\": {\n" +
" \"comma\": {\n" +
" \"type\": \"pattern\",\n" +
" \"pattern\": \",\"\n" +
" }\n" +
" }\n" +
" }}", XContentType.JSON);
request.mapping("{\"properties\": {\n" +
" \"videoName\":{\n" +
" \"type\": \"text\",\n" +
" \"analyzer\": \"ik_max_word\"\n" +
" },\n" +
" \"tags\":{\n" +
" \"type\": \"text\",\n" +
" \"analyzer\": \"comma\"\n" +
" },\n" +
" \"playCount\":{\n" +
" \"type\":\"integer\",\n" +
" \"index\":false\n" +
" },\n" +
" \"danmuCount\":{\n" +
" \"type\":\"integer\",\n" +
" \"index\":false\n" +
" },\n" +
" \"collectCount\":{\n" +
" \"type\":\"integer\",\n" +
" \"index\":false\n" +
" },\n" +
" \"createTime\":{\n" +
" \"type\":\"date\",\n" +
" \"format\": \"yyyy-MM-dd HH:mm:ss\",\n" +
" \"index\": false\n" +
" },\n" +
" \"videoId\":{\n" +
" \"type\":\"text\",\n" +
" \"index\": false\n" +
" }\n" +
" }" +
"}", XContentType.JSON);
CreateIndexResponse createIndexResponse = restHighLevelClient.indices().create(request, RequestOptions.DEFAULT);
Boolean acknowledged = createIndexResponse.isAcknowledged();
if (!acknowledged) {
throw new BusinessException("初始化ES失败");
}
} catch (IOException e) {
log.error("初始化ES失败", e);
throw new BusinessException("初始化ES失败");
}
}
}
因为Es是Docs存储的,所以这里的配置得用JS格式来发送请求到esClient,这里其实就是创建视频的搜索信息,约束和分词器.
在创建号Component之后启动SpringBoot程序,用一个实现了ApplicationRunner接口的类实现run回调函数
public void run(ApplicationArguments args) throws Exception {
esSearchComponent.createIndex();
}
一切正常没有报错的话就进去下一步
2.使用阶段
在此之前,我们需要创建一个和es里docs存储mapping一致的DTO类,因为当有多余属性的话,es就会自动添加一些属性进去,虽然目前没什么问题,但这是预期之外的为了防止问题就先解决掉.
public class VideoInfoEsDto {
/**
* 视频ID
*/
private String videoId;
/**
* 视频封面
*/
private String videoCover;
/**
* 视频名称
*/
private String videoName;
/**
* 用户ID
*/
private String userId;
/**
* 创建时间
*/
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8")
@JSONField(format = "yyyy-MM-dd HH:mm:ss")
private Date createTime;
/**
* 标签
*/
private String tags;
/**
* 播放数量
*/
private Integer playCount;
/**
* 弹幕数量
*/
private Integer danmuCount;
/**
* 收藏数量
*/
private Integer collectCount;
这里需要注意一点的就是createTime上的注解JsonFormat是Jackson的格式化,我们用的是fastJson所以得使用fastJson的格式化注解@JsonField 不然没有序列化进去就不是我们预期的效果了
1.保存对象Dto到es当中
在我们的后台审核视频业务当中,我们在通过视频审核之后就要将VideoInfo保存到es当中,在auditVideo方法的最后才使用saveDocs方法
这是考虑到事务的问题,因为es不支持Transactional的rollback,也就是说当es里写入成功之后如果后面抛出异常的话那es里的信息其实是不会回滚的
public void saveDoc(VideoInfo videoInfo) {
VideoInfoEsDto videoInfoEsDto = CopyTools.copy(videoInfo, VideoInfoEsDto.class);
try {
if (docExist(videoInfo.getVideoId())) {
updateDoc(videoInfoEsDto);
} else {
insertDoc(videoInfoEsDto);
}
} catch (IOException e) {
log.error("保存到Es失败", e);
throw new BusinessException("保存到Es失败");
}
}
private void updateDoc(VideoInfoEsDto videoInfoEsDto) throws IOException {
try {
videoInfoEsDto.setCreateTime(null);
HashMap<String, Object> dataMap = new HashMap<>();
Field[] fields = videoInfoEsDto.getClass().getDeclaredFields();
for (Field field : fields) {
String methodName = "get" + StringTools.upperCaseFirstLetter(field.getName());
Method method = videoInfoEsDto.getClass().getMethod(methodName);
Object invoke = method.invoke(videoInfoEsDto);
if (Objects.nonNull(invoke) && invoke instanceof String && !StringTools.isEmpty(invoke.toString()) || Objects.nonNull(invoke) && !(invoke instanceof String)) {
dataMap.put(field.getName(), invoke);
}
if (dataMap.isEmpty()) {
return;
}
UpdateRequest updateRequest = new UpdateRequest(appConfig.getEsIndexVideoName(), videoInfoEsDto.getVideoId());
updateRequest.doc(dataMap);
restHighLevelClient.update(updateRequest, RequestOptions.DEFAULT);
}
} catch (Exception e) {
log.error("保存视频失败", e);
throw new BusinessException("保存视频失败");
}
}
private void insertDoc(VideoInfoEsDto videoInfoEsDto) throws IOException {
videoInfoEsDto.setCollectCount(0);
videoInfoEsDto.setDanmuCount(0);
videoInfoEsDto.setPlayCount(0);
IndexRequest indexRequest = new IndexRequest(appConfig.getEsIndexVideoName());
indexRequest.id(videoInfoEsDto.getVideoId()).source(JsonUtils.convertObj2Json(videoInfoEsDto), XContentType.JSON);
restHighLevelClient.index(indexRequest, RequestOptions.DEFAULT);
}
private Boolean docExist(String id) throws IOException {
GetRequest getIndexRequest = new GetRequest(appConfig.getEsIndexVideoName(), id);
GetResponse getResponse = restHighLevelClient.get(getIndexRequest, RequestOptions.DEFAULT);
return getResponse.isExists();
}
2. 更新数量到ES
public void updateDocCount(String videoId, String fieldName, Integer count) {
try {
UpdateRequest updateRequest = new UpdateRequest(appConfig.getEsIndexVideoName(), videoId);
//这里类似于SQL里的行锁,来防止高并发的情况下导致数据出现错误
Script script = new Script(ScriptType.INLINE, "painless", "ctx._source." + fieldName + " += params.count", Collections.singletonMap("count", count));
updateRequest.script(script);
restHighLevelClient.update(updateRequest, RequestOptions.DEFAULT);
} catch (Exception e) {
log.error("保存视频失败", e);
throw new BusinessException("保存视频失败");
}
}
3. 从es里删除视频
public void delDoc(String videoId) {
try {
DeleteRequest deleteRequest = new DeleteRequest(appConfig.getEsIndexVideoName(), videoId);
restHighLevelClient.delete(deleteRequest, RequestOptions.DEFAULT);
} catch (Exception e) {
log.error("删除视频失败", e);
throw new BusinessException("删除视频失败");
}
}
4. 搜索
public PaginationResultVO<VideoInfo> search(Boolean highLight, String keyWord, Integer orderType, Integer pageNo, Integer pageSize) {
try {
SearchOrderTypeEnum searchOrderTypeEnum = SearchOrderTypeEnum.getByType(orderType);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.multiMatchQuery(keyWord, "videoName", "tags"));
// 高亮
if (highLight) {
HighlightBuilder highlightBuilder = new HighlightBuilder();
highlightBuilder.field("videoName");
// <span class = 'highlight'> { 关键字 } </span>
highlightBuilder.preTags("<span class = 'highlight'>");
highlightBuilder.postTags("</span>");
searchSourceBuilder.highlighter(highlightBuilder);
}
// 排序
searchSourceBuilder.sort("_score", SortOrder.ASC);
if (Objects.nonNull(orderType)) {
searchSourceBuilder.sort(searchOrderTypeEnum.getField(), SortOrder.DESC);
}
pageNo = pageNo == null ? Constants.ONE : pageNo;
pageSize = pageSize == null ? PageSize.SIZE20.getSize() : pageSize;
searchSourceBuilder.size(pageSize);
searchSourceBuilder.from((pageNo - 1) * pageSize);
SearchRequest searchRequest = new SearchRequest(appConfig.getEsIndexVideoName());
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
SearchHits searchHits = searchResponse.getHits();
Integer totalCount = (int) searchHits.getTotalHits().value;
List<VideoInfo> videoInfoList = new ArrayList<>();
List<String> userIdList = new ArrayList<>();
for (SearchHit searchHit : searchHits.getHits()) {
VideoInfo videoInfo = JsonUtils.convertJson2Obj(searchHit.getSourceAsString(), VideoInfo.class);
if (Objects.nonNull(searchHit.getHighlightFields().get("videoName"))) {
videoInfo.setVideoName(searchHit.getHighlightFields().get("videoName").fragments()[0].string());
}
videoInfoList.add(videoInfo);
userIdList.add(videoInfo.getUserId());
}
UserInfoQuery userInfoQuery = new UserInfoQuery();
userInfoQuery.setUserIdList(userIdList);
List<UserInfo> userInfoList = userInfoMapper.selectList(userInfoQuery);
Map<String, UserInfo> userInfoMap = userInfoList.stream()
.collect(Collectors.toMap(item -> item.getUserId(), Function.identity(), (data1, data2) -> data2));
videoInfoList.forEach(item -> {
item.setNickName(userInfoMap.get(item.getUserId()).getNickName());
});
SimplePage simplePage = new SimplePage(pageNo, totalCount, pageSize);
PaginationResultVO<VideoInfo> resultVO = new PaginationResultVO<>(totalCount, simplePage.getPageSize(), simplePage.getPageNo(), simplePage.getPageTotal(), videoInfoList);
return resultVO;
} catch (IOException e) {
log.error("视频搜索失败", e);
throw new BusinessException("视频搜索失败");
}
}
3. 总结
在这段学习的时间里查看文档了解了一些elasticSearch的API方法,以及最重要的一句话:"注释不是让你写每行代码逐字逐句的去解释,而是让你一眼就明白这代码的业务功能".这个问题我再之前的filePost文件上传的时候犯了一次,因为我逐字逐句的解释每行代码的功能从而导致后续找Bug的时候阅读到那段代码及其吃力,因为我现在已经有了阅读代码的能力不需要注释这么多荣誉的注解,有这个时间不如提高一下代码的可阅读性.
浙公网安备 33010602011771号