High Level REST Client操作elasticsearch

　　Java Low Level REST Client: 低级别的REST客户端，通过http与集群交互，用户需自己编组请求JSON串，及解析响应JSON串。兼容所有ES版本。
　　Java High Level REST Client: 高级别的REST客户端，基于低级别的REST客户端，增加了编组请求JSON串、解析响应JSON串等相关api。使用的版本需要保持和ES服务端的版本一致，否则会有版本问题

Java High Level REST Client 说明

　　从6.0.0开始加入的，目的是以java面向对象的方式来进行请求、响应处理。
　　每个API 支持 同步/异步 两种方式，同步方法直接返回一个结果对象。异步的方法以async为后缀，通过listener参数来通知结果。
　　高级java REST 客户端依赖Elasticsearch core project

1、添加依赖

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.8.1</version>
        </dependency>

        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>${elasticsearch.version}</version>
        </dependency>

        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>${elasticsearch.version}</version>
        </dependency>

2、boot配置

server.port=8089
elasticsearch.port=9200
elasticsearch.host=192.168.223.145

     配置文件

@Slf4j
@Configuration
public class ElasticsearchRestClientConfig {
    private static final String HTTP_SCHEME = "http";

    @Value("${elasticsearch.host}")
    private String host;
    @Value("${elasticsearch.port}")
    private int port;

    @Bean
    public RestHighLevelClient highLevelClient() {
        RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost(host, port, HTTP_SCHEME));
        return new RestHighLevelClient(restClientBuilder);
    }

}

    创建索引

 public void createIndex() {
        Map<String, Object> jsonMap = new HashMap<>();
        jsonMap.put("user", "孔明");
        jsonMap.put("postDate", new Date());
        jsonMap.put("message", "三分天下");
        IndexRequest strategy = new IndexRequest("strategy").source(jsonMap);
        try {
            IndexResponse response = restHighLevelClient.index(strategy, RequestOptions.DEFAULT);
            log.info("{}", response.getResult());
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

     获取索引

 @Override
    public void getIndex() {
        GetRequest getRequest = new GetRequest("strategy", "Do13J28Bx_j1OnAjba1Q");
        try {
            GetResponse response = restHighLevelClient.get(getRequest, RequestOptions.DEFAULT);
            log.info("{}", response.getSource());
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

　　更新索引

 @Override
    public void updateIndex() {
        UpdateRequest updateRequest = new UpdateRequest("strategy", "Do13J28Bx_j1OnAjba1Q");
        Map<String, Object> params = new HashMap<>();
        params.put("message", "三分天下");
        updateRequest.doc(params);
        try {
            UpdateResponse updateResponse = restHighLevelClient.update(updateRequest, RequestOptions.DEFAULT);
            log.info("{}", updateResponse.getGetResult());
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

      删除索引

 @Override
    public void deleteIndex() {
        DeleteRequest deleteRequest = new DeleteRequest("strategy", "Do13J28Bx_j1OnAjba1Q");
        try {
            DeleteResponse deleteResponse = restHighLevelClient.delete(deleteRequest, RequestOptions.DEFAULT);
            log.info("{}", deleteResponse.getResult());
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

      bulk api

 public void bulkIndex() {
        BulkRequest bulkRequest = new BulkRequest();
        bulkRequest.add(new IndexRequest("leader").source(XContentType.JSON, "name", "曹操"));
        bulkRequest.add(new IndexRequest("leader").source(XContentType.JSON, "name", "刘备"));
        bulkRequest.add(new IndexRequest("leader").source(XContentType.JSON, "name", "孙权"));
//        try {
//            restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
        restHighLevelClient.bulkAsync(bulkRequest, RequestOptions.DEFAULT, new ActionListener<BulkResponse>() {

            @Override
            public void onResponse(BulkResponse bulkItemResponses) {
                log.info("{}", bulkItemResponses.getItems());
            }

            @Override
            public void onFailure(Exception e) {

            }
        });
//        } catch (IOException e) {
//            e.printStackTrace();
//        }

    }

　　bulk processor

　　BulkProcessor 简化bulk API的使用，并且使整个批量操作透明化。

       BulkProcessor 的执行需要三部分组成：

1. RestHighLevelClient :执行bulk请求并拿到响应对象。
2. BulkProcessor.Listener：在执行bulk request之前、之后和当bulk response发生错误时调用。
3. ThreadPool：bulk request在这个线程池中执行操作，这使得每个请求不会被挡住，在其他请求正在执行时，也可以接收新的请求。

@Slf4j
@Service
public class ElasticsearchUtil {

    @Autowired
    private RestHighLevelClient restHighLevelClient;

    private BulkProcessor bulkProcessor;

    @PostConstruct
    public void init() {
        BulkProcessor.Listener listener = new BulkProcessor.Listener() {

            @Override
            public void beforeBulk(long executionId, BulkRequest request) {
                //重写beforeBulk,在每次bulk request发出前执行,在这个方法里面可以知道在本次批量操作中有多少操作数
                int numberOfActions = request.numberOfActions();
                log.info("executing bulk {} with {} requests", executionId, numberOfActions);
            }

            @Override
            public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
                //重写afterBulk方法，每次批量请求结束后执行，可以在这里知道是否有错误发生。
                if (response.hasFailures()) {
                    log.error("bulk {} executed with failure,response {}", executionId, response.buildFailureMessage());
                } else {
                    log.info("bulk {} completed in {} milliseconds", executionId, response.getTook().getMillis());
                }
            }

            @Override
            public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
                //重写方法，如果发生错误就会调用
                log.error("fail to execute bulk {}", failure);
            }
        };

        BulkProcessor bulkProcessor = BulkProcessor.builder(
                (request, bulkListener) -> restHighLevelClient.bulkAsync(request, RequestOptions.DEFAULT, bulkListener),
                listener)
                // 1000条数据请求执行一次bulk
                .setBulkActions(100)
                // 5mb的数据刷新一次bulk
                .setBulkSize(new ByteSizeValue(1L, ByteSizeUnit.MB))
                // 并发请求数量, 0不并发, 1并发允许执行
                .setConcurrentRequests(0)
                // 固定1s必须刷新一次
                .setFlushInterval(TimeValue.timeValueSeconds(1L))
                // 重试5次，间隔1s
                .setBackoffPolicy(BackoffPolicy.constantBackoff(TimeValue.timeValueSeconds(1), 5)).build();
        this.bulkProcessor = bulkProcessor;
    }

    @PreDestroy
    public void destroy() {
        try {
            bulkProcessor.awaitClose(30, TimeUnit.SECONDS);
        } catch (InterruptedException e) {
            log.error("bulkProcessor fail to close!");
        }
        log.info("bulkProcessor closed");
    }

    public void update(UpdateRequest request) {
        bulkProcessor.add(request);
    }

    public void insert(IndexRequest request) {
        bulkProcessor.add(request);
    }
}

　　upsert api

　　update --当id不存在时将会抛出异常:

　　upsert--id不存在时就插入:

public void upsert() {
    Map<String, Object> jsonMap = new HashMap<>();
    jsonMap.put("name", "孔明");
    IndexRequest indexRequest = new IndexRequest("leader").source(jsonMap);
    UpdateRequest updateRequest = new UpdateRequest("leader", "2").upsert(indexRequest);
    try {
        updateRequest.doc(indexRequest);
        restHighLevelClient.update(updateRequest, RequestOptions.DEFAULT);
    } catch (IOException e) {
        e.printStackTrace();
    }
}

　　search api

public void testRestESClient() {
        SearchRequest searchRequest = new SearchRequest("person");
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        sourceBuilder.query(QueryBuilders.termQuery("name", "青春"));
        sourceBuilder.timeout(new TimeValue(60, TimeUnit.SECONDS));
        searchRequest.source(sourceBuilder);
        try {
            SearchResponse response = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            log.info("-----{}", response.getTotalShards());
        } catch (Exception e) {
            log.info("-----{}", e.getMessage());
        }
    }

　  全量搜索

 public void allMatchQuery() {
        SearchRequest searchRequest = new SearchRequest("leader");
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        sourceBuilder.query(QueryBuilders.matchAllQuery());
        searchRequest.source(sourceBuilder);
        try {
            SearchResponse search = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            log.info("match all {}", search.getHits());
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

      分页搜索

 public void scrollQuery() {
        MatchQueryBuilder queryBuilder = QueryBuilders.matchQuery("name", "青春")
                .fuzziness(Fuzziness.AUTO).prefixLength(1).maxExpansions(1);
        SearchRequest searchRequest = new SearchRequest("person");
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        sourceBuilder.query(queryBuilder);
        sourceBuilder.size(3);
        searchRequest.source(sourceBuilder);
        searchRequest.scroll(TimeValue.timeValueMinutes(1L));
        try {
            SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            String scrollId = searchResponse.getScrollId();
            SearchHit[] hits = searchResponse.getHits().getHits();

            while (hits != null && hits.length > 0) {
                SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
                scrollRequest.scroll(TimeValue.timeValueSeconds(30));
                SearchResponse response = restHighLevelClient.scroll(scrollRequest, RequestOptions.DEFAULT);
                scrollId = response.getScrollId();
                hits = response.getHits().getHits();
            }

            ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
            clearScrollRequest.addScrollId(scrollId);
            ClearScrollResponse scrollResponse = restHighLevelClient.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
            boolean succeeded = scrollResponse.isSucceeded();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

　　排序

　　SearchSourceBuilder可以添加一种或多种SortBuilder。
　　有四种特殊的排序实现：

• field
• score
• GeoDistance
• scriptSortBuilder

        sourceBuilder.sort(new ScoreSortBuilder().order(SortOrder.DESC));
        sourceBuilder.sort(new FieldSortBuilder("_uid").order(SortOrder.ASC));

• 过滤

　　默认情况下，searchRequest返回文档内容，与REST API一样，重写search行为。例如，可以完全关闭"_source"检索

sourceBuilder.fetchSource(false);

　　方法还接受一个或多个通配符模式的数组，以更细粒度地控制包含或排除哪些字段

String[] includeFields = new String[] {"title", "user", "innerObject.*"};
String[] excludeFields = new String[] {"_type"};
sourceBuilder.fetchSource(includeFields, excludeFields);

       聚合

　　配置适当的 AggregationBuilder ，再将它传入SearchSourceBuilder里,就可以完成聚合请求

　　聚合请求进行分组

GET /person/_search?pretty
{
  "size": 0,
  "aggs":{
    "group_by_state":{
        "terms":{
          "field":"country"
        }
    }
  }
}

　　响应

{
  "took" : 443,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 5,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "group_by_state" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "china",
          "doc_count" : 4
        },
        {
          "key" : "usa",
          "doc_count" : 1
        }
      ]
    }
  }
}

　　java实现

 public void aggsQuery() {
        SearchRequest searchRequest = new SearchRequest("person");
        TermsAggregationBuilder aggregation = AggregationBuilders.terms("group_by_state").field("country");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.aggregation(aggregation);
        searchSourceBuilder.size(0);
        searchRequest.source(searchSourceBuilder);
        try {
            SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            System.out.println(searchResponse.getHits());
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

　　SearchResponse

　　查询的返回结果、使用分片情况、文档数据,HTTP状态码

SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            RestStatus status = searchResponse.status();
            TimeValue took = searchResponse.getTook();
            Boolean terminatedEarly = searchResponse.isTerminatedEarly();
            boolean timedOut = searchResponse.isTimedOut();

            SearchHits hits = searchResponse.getHits();

　　为了取回文档数据，我们要从search response的返回对象里先得到searchHit对象

 RestStatus status = searchResponse.status();
            TimeValue took = searchResponse.getTook();
            Boolean terminatedEarly = searchResponse.isTerminatedEarly();
            boolean timedOut = searchResponse.isTimedOut();
            SearchHit[] searchHits = searchResponse.getHits().getHits();
            for (SearchHit hit : searchHits) {
                System.out.println(hit.getSourceAsString());
                Map<String, Object> sourceAsMap = hit.getSourceAsMap();
                System.out.println("map:" + sourceAsMap);
            }

　　取回聚合数据 

       参考:  https://www.cnblogs.com/leeSmall/p/9218779.html

            Aggregations aggregations = searchResponse.getAggregations();
            Terms groupByState = aggregations.get("group_by_state");
            List<? extends Terms.Bucket> buckets = groupByState.getBuckets();
            for (Terms.Bucket b : buckets) {
                System.out.println(b.getKeyAsString() + ":" + b.getDocCount());
            }