ElasticSearch搜索学习笔记

1. 倒排索引原理

说到ElasticSearch不得不说他的倒排索引，这是他全文检索方面为何搜索快的原因。

倒排索引需要把文档数据逐个编号（从0递增），存储到文档表中。并且给每一个编号创建索引，这样根据编号检索文档的速度会非常快。

所谓的倒排索引就是一段话分成若干个部分分开储存。比如你有一亿条数据，里面肯定有大量重复的字，将所有的数据分词，得到单个字或者词最多也就上百万条。对这百万条数据建立索引，然后将词条（Term）对应的所有数据的文档编号（就是一个id）存到里面，再通过文档编号直接去取数据。这样就将搜索简化了。

文档（Document）：用来检索的海量数据，其中的每一条数据就是一个文档。例如一个网页、一个商品信息
词条（Term）：对文档数据或用户搜索数据，利用某种算法分词，得到的具备含义的词语就是词条。

倒排索引的数据存储方式与数据库类似，但检索方式不同。

全文检索主要是通过倒排索引的方式实现，倒排索引主要是分成两大步骤完成：

数据按照规则处理完成后存储到索引库
用户输入关键词按照规则处理完成后在索引库中检索

2.ElasticSearch的强大之处就在于它具备了完善且强大的查询功能。

搜索相关功能主要包括：

基本查询
- 分词查询
- 词条查询
- 范围查询
- 布尔查询
  - Filter功能
source筛选

排序

分页
高亮
聚合

依赖：

<dependencies>
  <!--elastic客户端-->
  <!--elastic客户端-->
  <dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-high-level-client</artifactId>
    <version>7.4.2</version>
  </dependency>
  <!-- Junit单元测试 -->
  <dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
  </dependency>
  <!--lombok  @Data -->
  <dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.8</version>
  </dependency>
  <!--JSON工具 -->
  <dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>fastjson</artifactId>
    <version>1.2.49</version>
  </dependency>
  <!--common工具-->
  <dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-lang3</artifactId>
    <version>3.8.1</version>
  </dependency>
</dependencies>

下面为实体类user

@Data
public class User {
    private Long id;
    private Integer age;
    private String name;
    private String gender;
    private String note;
}

kibana建立映射：

PUT /user
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "long"
      },
      "name":{
        "type": "keyword"
      },
      "age":{
        "type": "integer"
      },
      "gender":{
        "type": "keyword"
      },
      "note":{
        "type": "text",
        "analyzer": "ik_max_word"
      }
    }
  }
}

kibana建立数据：

# 批量导入的脚本
POST _bulk
{"index":{"_index":"user","_type":"_doc","_id":"1"}}
{"age":18,"gender":"1","id":1,"name":"Rose","note":"鲍勃同学在酒馆学表演"}
{"index":{"_index":"user","_type":"_doc","_id":"2"}}
{"age":38,"gender":"1","id":2,"name":"Jack","note":"安度因同学在学圣光"}
{"index":{"_index":"user","_type":"_doc","_id":"3"}}
{"age":38,"gender":"1","id":2,"name":"Jack","note":"Jack同学在酒馆下棋"}
{"index":{"_index":"user","_type":"_doc","_id":"4"}}
{"age":23,"gender":"0","id":3,"name":"小红","note":"小红同学在酒馆下棋"}
{"index":{"_index":"user","_type":"_doc","_id":"5"}}
{"age":20,"gender":"1","id":4,"name":"小明","note":"小明同学在酒馆下棋"}
{"index":{"_index":"user","_type":"_doc","_id":"6"}}
{"age":33,"gender":"1","id":5,"name":"达摩","note":"吉安娜在酒馆玩愚人套牌"}
{"index":{"_index":"user","_type":"_doc","_id":"7"}}
{"age":24,"gender":"1","id":6,"name":"鲁班","note":"鲁班同学躺在峡谷里"}
{"index":{"_index":"user","_type":"_doc","_id":"8"}}
{"age":26,"gender":"0","id":7,"name":"孙尚香","note":"孙尚香同学想来一发"}
{"index":{"_index":"user","_type":"_doc","_id":"9"}}
{"age":27,"gender":"1","id":8,"name":"李白","note":"李白同学在山顶喝酒"}
{"index":{"_index":"user","_type":"_doc","_id":"10"}}
{"age":28,"gender":"0","id":9,"name":"甄姬","note":"甄姬同学弹奏曲子"}
{"index":{"_index":"user","_type":"_doc","_id":"11"}}
{"age":27,"gender":"0","id":10,"name":"虞姬","note":"虞姬同学在谈情说爱"}

数据建成了，接下来就可以去玩下查询啦。

1.查询所有match_all

kibana测试脚本：

GET /user/_search
{
  "query": {
    "match_all": {}
  }
}

测试代码：

//查询所有match_all
@Test
public void test1() throws Exception{
　　//建立连接
    HttpHost localhost = new HttpHost("localhost", 9200);
    RestClientBuilder builder = RestClient.builder(localhost);
    RestHighLevelClient client = new RestHighLevelClient(builder);

    //查询所有
    //创建SearchRequest对象

    SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
    SearchSourceBuilder query = sourceBuilder.query(QueryBuilders.matchAllQuery());

    //将添加SearchSourceBuilder对象到SearchRequest对象中

    SearchRequest request = new SearchRequest("user");
    SearchRequest source = request.source(query);

    //查询
    SearchResponse response = client.search(request, RequestOptions.DEFAULT);

    //解析结果
    SearchHits hits = response.getHits();
    long value = hits.getTotalHits().value;
    System.out.println(value);

    SearchHit[] hits1 = hits.getHits();
    for (SearchHit hit : hits1) {
        float score = hit.getScore();
        System.out.println("文档得分："+score);
        String json = hit.getSourceAsString();
        System.out.println("JSON数据"+json);
        User user = JSON.parseObject(json, User.class);
        System.out.println("反序列化为User对象"+user);
    }
}

2.词条查询-termQuery

ElasticSearch两个数据类型

text：会分词，不支持聚合
keyword：不会分词，将全部内容作为一个词条，支持聚合

kibana测试脚本：

# 词条查询
#不会对查询条件进行分词
GET /user/_search
{
  "query": {
    "term": {
      "name": {
        "value": "小红"
      }
    }
  }
}
#会对查询条件进行分词
GET /user/_search
{
  "query": {
    "term": {
      "note": {  # note 的类型是 text
        "value": "酒馆"
      }
    }
  }
}

测试代码：

//词条查询-termQuery
@Test
public void test2() throws Exception{
    HttpHost localhost = new HttpHost("localhost", 9200);
    RestClientBuilder builder = RestClient.builder(localhost);
    RestHighLevelClient client = new RestHighLevelClient(builder);

    SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
    SearchSourceBuilder query = sourceBuilder.query(QueryBuilders.termQuery("note", "酒馆"));//核心步骤

    SearchRequest request = new SearchRequest("user");
    SearchRequest source = request.source(query);

    SearchResponse search = client.search(source, RequestOptions.DEFAULT);

    SearchHits hits = search.getHits();
    //获取总条数
    TotalHits totalHits = hits.getTotalHits();
    System.out.println("获取总条数"+totalHits.value);
    for (SearchHit hit : hits) {
        float score = hit.getScore();
        System.out.println("分数"+score);
        //获取_source数据
        String sourceAsString = hit.getSourceAsString();
        System.out.println("_source数据" + sourceAsString);
    }
}

3.分词匹配查询-matchQuery

match查询：

会对查询条件进行分词。
然后将分词后的查询条件和词条进行等值匹配
默认取并集（OR）

# match查询
GET /user/_search
{
  "query": {
    "match": {
      "name": "安度因"
    }
  }
}
# 查看分词效果
GET /_analyze
{
  "text": "安度因"
}

测试代码：

//分词匹配查询-matchQuery
    @Test
    public void test3() throws Exception{
        HttpHost localhost = new HttpHost("localhost", 9200);
        RestClientBuilder builder = RestClient.builder(localhost);
        RestHighLevelClient client = new RestHighLevelClient(builder);

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        SearchSourceBuilder query = sourceBuilder.query(QueryBuilders.matchQuery("name", "鲁班"));//核心步骤

        SearchRequest request = new SearchRequest("user");
        SearchRequest source = request.source(query);

        SearchResponse search = client.search(source, RequestOptions.DEFAULT);

        for (SearchHit hit : search.getHits()) {
            String sourceAsString = hit.getSourceAsString();
            System.out.println(sourceAsString);
        }
    }

4.模糊查询-fuzzy

fuzzy查询默认会按照一定的规则来修正当前查询的条件，修正的次数最大是 2次，可取值为：0、1、2，大于2则按照最大值计算。

# fuzzy 模糊查询
GET /user/_search
{
  "query": {
    "fuzzy": {
      "note": {
        "value": "在",  # 输入 在
        "fuzziness": 2 # 修正的次数
      }
    }
  }
}

测试代码：

//模糊查询-fuzzy
    @Test
    public void test4() throws Exception{
        HttpHost localhost = new HttpHost("localhost", 9200);
        RestClientBuilder builder = RestClient.builder(localhost);
        RestHighLevelClient client = new RestHighLevelClient(builder);

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        SearchSourceBuilder query = sourceBuilder.query(QueryBuilders.fuzzyQuery("note", "小").fuzziness(Fuzziness.ONE));//核心步骤

        SearchRequest request = new SearchRequest("user");
        SearchRequest source = request.source(query);

        SearchResponse search = client.search(source, RequestOptions.DEFAULT);

        for (SearchHit hit : search.getHits()) {
            System.out.println(hit.getSourceAsString());
        }
    }

5.范围&排序查询-range&sort

# 范围查询&排序
GET user/_search
{
  "query": {
    "range": {
      "age": {   # 范围查询字段
        "gte": 10,
        "lt": 27
      }
    }
  },
  "sort": [   # 排序，如果是多个条件则在数组中添加排序列即可
    {
      "id": {
        "order": "asc"
      }
    }
  ]
}

测试代码：

//范围&排序查询-range&sort
    @Test
    public void test5() throws Exception{
        HttpHost httpHost = new HttpHost("localhost",9200);
        RestClientBuilder builder = RestClient.builder(httpHost);
        RestHighLevelClient client = new RestHighLevelClient(builder);

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        SearchSourceBuilder age = sourceBuilder.sort("age", SortOrder.DESC)
                                                .query(QueryBuilders.rangeQuery("age").gte(18).lt(20));// 18<=age<20核心步骤

        SearchRequest request = new SearchRequest("user");
        SearchRequest source = request.source(age);

        SearchResponse search = client.search(source, RequestOptions.DEFAULT);
        for (SearchHit hit : search.getHits()) {
            System.out.println(hit.getSourceAsString());
        }
    }

6.多条件查询-queryString

queryString会对查询条件进行分词。
然后将分词后的查询条件和词条进行等值匹配
默认取并集（OR）default_operator
可以指定多个查询字段

# queryString
GET user/_search
{
  "query": {
    "query_string": {
      "fields": ["name","note"], 
      "query": "酒馆同学"  # 测试：" 酒馆AND 同学" 
    }
  }
}

测试代码：

//多条件查询-queryString
    @Test
    public void test6() throws Exception{
        HttpHost localhost = new HttpHost("localhost", 9200);
        RestClientBuilder builder = RestClient.builder(localhost);
        RestHighLevelClient client = new RestHighLevelClient(builder);

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        SearchSourceBuilder note = sourceBuilder.query(QueryBuilders.queryStringQuery("甄姬同学")
                                                .field("name")
                                                .field("note")
                                                .defaultOperator(Operator.AND));//核心步骤

        SearchRequest request = new SearchRequest("user");
        SearchRequest source = request.source(note);

        SearchResponse search = client.search(source, RequestOptions.DEFAULT);

        for (SearchHit hit : search.getHits()) {
            System.out.println(hit.getSourceAsString());
        }
    }

7.bool查询&结果过滤-boolQuery

must（and）：条件必须成立
must_not（not）：条件必须不成立
should（or）：条件可以成立
filter：条件必须成立，性能比must高。不会计算得分

得分:即条件匹配度,匹配度越高，得分越高

# boolquery
#must和filter配合使用时，max_score（得分）是显示的
#must 默认数组形式
GET user/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "note": "同学"
          }
        }
      ],
       "filter":[ 
        {
        "match": {
          "note": "在"
        }
       },
       {
         "range":{
          "age": {
            "gte": 20,
            "lte": 27
         }
         }
       }
      ]
    }
  }
}
#filter 单独使用   filter可以是单个条件，也可多个条件（数组形式）
GET user/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "name": {
              "value": "同学"
            }
          }
        }
      ]
    }
  }
}

测试代码：

//bool查询&结果过滤-boolQuery
    @Test
    public void test7() throws Exception{
        HttpHost localhost = new HttpHost("localhost", 9200);
        RestClientBuilder builder = RestClient.builder(localhost);
        RestHighLevelClient client = new RestHighLevelClient(builder);

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        SearchSourceBuilder query = sourceBuilder.query(QueryBuilders.boolQuery()//核心步骤
                .must(QueryBuilders.matchQuery("note", "同学"))
                .filter(QueryBuilders.termQuery("note", "在"))
                .filter(QueryBuilders.rangeQuery("age").gt(10).lt(30))
        );
        
        SearchRequest request = new SearchRequest("user");
        SearchRequest source = request.source(query);

        SearchResponse search = client.search(source, RequestOptions.DEFAULT);
        for (SearchHit hit : search.getHits()) {
            System.out.println(hit.getSourceAsString());
        }
    }

8.分页查询-from

默认情况下ES会设置size=10，查询10条记录。通过from和size来指定分页的开始位置及每页大小。

GET /user/_search
{
  "query": {
    "match_all": {}
  }，
  "from": 0,  
  "size": 2
}

测试代码：

//分页查询-from
    @Test
    public void test8() throws Exception{
        HttpHost localhost = new HttpHost("localhost", 9200);
        RestClientBuilder builder = RestClient.builder(localhost);
        RestHighLevelClient client = new RestHighLevelClient(builder);

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        SearchSourceBuilder query = sourceBuilder.from(0).size(3).query(QueryBuilders.matchAllQuery());//核心步骤

        SearchRequest request = new SearchRequest("user");
        SearchRequest source = request.source(query);

        SearchResponse search = client.search(source, RequestOptions.DEFAULT);
        for (SearchHit hit : search.getHits()) {
            System.out.println(hit.getSourceAsString());
        }
    }

9.高亮查询-highlight

高亮三要素：

pre_tags：前置标签，可以省略，默认是em
post_tags：后置标签，可以省略，默认是em
fields：需要高亮的字段
- title：这里声明title字段需要高亮，后面可以为这个字段设置特有配置，也可以空

# <font style="color:red">手机</font>
GET user/_search
{
  "query": {
    "match": {
      "note": "酒馆"
    }
  },
  "highlight": { # 设置高亮
    "fields": {
      "note": { # 设置高亮显示的字段
        "pre_tags": "<font color='red'>",  # 高亮显示前缀
        "post_tags": "</font>"  # 高亮显示后缀
      }
    }
  }
}

测试代码：

//高亮查询-highlight
    @Test
    public void test9() throws Exception{
        HttpHost localhost = new HttpHost("localhost", 9200);
        RestClientBuilder builder = RestClient.builder(localhost);
        RestHighLevelClient client = new RestHighLevelClient(builder);

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        SearchSourceBuilder note = sourceBuilder.query(QueryBuilders.matchQuery("note","在"))//核心步骤
                .highlighter(SearchSourceBuilder.highlight()
                .field("同学")
                .preTags("<font color='red'>")
                .postTags("</font>"));


        SearchRequest request = new SearchRequest("user");
        SearchRequest source = request.source(note);

        SearchResponse search = client.search(source, RequestOptions.DEFAULT);
        for (SearchHit hit : search.getHits()) {
            System.out.println(hit.getSourceAsString());
        }
    }

10.聚合查询-aggregation

聚合（aggregations）可以让我们极其方便的实现对数据的统计、分析。例如：

什么品牌的手机最受欢迎？
这些手机的平均价格、最高价格、最低价格？
这些手机每月的销售情况如何？

实现这些统计功能的比数据库的sql要方便的多，而且查询速度非常快，可以实现近实时搜索效果。

要注意：参与聚合的字段，必须是keyword类型。

Elasticsearch中的聚合，包含多种类型，最常用的两种，一个叫桶，一个叫度量：

桶（bucket）-- 分组

桶的作用，是按照某种方式对数据进行分组，每一组数据在ES中称为一个桶，例如我们根据国籍对人划分，可以得到中国桶、英国桶，日本桶……或者我们按照年龄段对人进行划分：0~10,10~20,20~30,30~40等。

Elasticsearch中提供的划分桶的方式有很多：

Date Histogram Aggregation：根据日期阶梯分组，例如给定阶梯为周，会自动每周分为一组
Histogram Aggregation：根据数值阶梯分组，与日期类似，需要知道分组的间隔（interval）
Terms Aggregation：根据词条内容分组，词条内容完全匹配的为一组，类似数据库group by
Range Aggregation：数值和日期的范围分组，指定开始和结束，然后按段分组
……

综上所述，我们发现bucket aggregations 只负责对数据进行分组，并不进行计算，因此往往bucket中往往会嵌套另一种聚合：metrics aggregations即度量

度量（metrics）-- 聚合

分组完成以后，我们一般会对组中的数据进行聚合运算，例如求平均值、最大、最小、求和等，这些在ES中称为度量

比较常用的一些度量聚合方式：

Avg Aggregation：求平均值
Max Aggregation：求最大值
Min Aggregation：求最小值
Percentiles Aggregation：求百分比
Stats Aggregation：同时返回avg、max、min、sum、count等
Sum Aggregation：求和
Top hits Aggregation：求前几
Value Count Aggregation：求总数

小结：

度量聚合：相当于MySQL的聚合函数。max、min、avg、sum、count、top等
桶聚合：相当于MySQL的 group by 操作。不要对text类型的数据进行分组，会失败。

# 按照年龄分组
GET /car/_search
{
  "aggs": {
    "age": {
      "terms": {
        "field": "age"
      }
    }
  },
  "size": 0
}

测试代码：

//聚合查询-aggregation
    @Test
    public void test10() throws Exception{
        HttpHost localhost = new HttpHost("localhost", 9200);
        RestClientBuilder builder = RestClient.builder(localhost);
        RestHighLevelClient client = new RestHighLevelClient(builder);

        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        SearchSourceBuilder aggregation = sourceBuilder.aggregation(AggregationBuilders.terms("age").field("age"));//核心步骤

        SearchRequest request = new SearchRequest("user");
        SearchRequest source = request.source(aggregation);

        SearchResponse search = client.search(source, RequestOptions.DEFAULT);
        for (SearchHit hit : search.getHits()) {
            System.out.println(hit.getSourceAsString());
        }
    }
}

所有的代码前三段都是建立连接。

这些实际应用时抽取公共类就行了，只需要把核心步骤builder完成就好了。

聚合查询还有很多例子，自行探索就好了。

共同学习，不足之处请多多指正。

posted @ 2021-06-08 10:42 落叶一秋风阅读(147) 评论(0) 收藏举报

刷新页面返回顶部

落叶一秋风

ElasticSearch搜索学习笔记

1. 倒排索引原理

2.ElasticSearch的强大之处就在于它具备了完善且强大的查询功能。

1.查询所有match_all

2.词条查询-termQuery

3.分词匹配查询-matchQuery

4.模糊查询-fuzzy

5.范围&排序查询-range&sort

6.多条件查询-queryString

7.bool查询&结果过滤-boolQuery

8.分页查询-from

9.高亮查询-highlight

10.聚合查询-aggregation

公告