elasticsearch学习笔记001

《Elasticsearch 核心技术与实战》课程Github代码 https://github.com/onebirdrocks/geektime-ELK

运行的环境: windows

安装了 PowerShell-7.0.0

下载 elasticsearch和Kibana 都可以在华为云 https://mirrors.huaweicloud.com/ 上下载到

我安装的 elasticsearch 7.4.0 和 kibana 7.4.0

安装是解压之后，就可以使用的，安装了jdk 1.8.221，在环境变量中，配置了 JAVA_HOME并加入到 path中去了。

启动 powershell,进入 elasticsearch解压的目录,启动 elasticsearch

./bin/elasticsearch

浏览器可通过http://localhost:9200 访问，查看elasticsearch的基本信息

查看安装的插件

 ./bin/elasticsearch-plugin list

安装一个 analysis-icu

./bin/elasticsearch-plugin install analysis-icu

可用 ./bin/elasticsearch-plugin list再次查看安装的插件，然后启动 elasticsearch,

可通过http://localhost:9200/_cat/plugins 查看安装的列表

_cat/ 是 elasticsearch 提供的要给 api

plugins 查看安装的插件
nodes 查看运行的节点

如何在开发机上运行多个 Elasticsearch实例

./bin/elasticsearch -E node.name=node1 -E cluster.name=geektime -E path.data=node1_data -d 
./bin/elasticsearch -E node.name=node2 -E cluster.name=geektime -E path.data=node2_data -d 
./bin/elasticsearch -E node.name=node3 -E cluster.name=geektime -E path.data=node3_data -d

node.name 节点的名称

cluster.name 一个相同的集群的名称

pat.data 为每个节点设置存放数据的地址

注: 在 windows下的 powershell 好像不需要最后的 -d 参数，我是打开三个 powershell的窗口，单独运行上面的命令的

删除进程 ps grep|elasticsearch / kill pid

kibana

启动

./bin/kibana

默认是 5601 端口

http://localhost:5601

在首页有一个 Add sample data 下面有一个链接，点击进去，可以进行导入 kibana 提供的简单数据

kinana console

dev tools
Search Profiler
Help + 一些快捷键
- cmd + / (查看API帮助文档)
- cmd + option + l
- cmd + option + 0
- cmd + option + shift + 0

kibana plugins

bin/kibana-plugin install plugin_location
bin/kibana-plugin list
bin/kibana remove

Apps:

LogTrail
Own Home
Shard Allocation
Corweyor
Indices View
Analyze UI
Cleaner Setting index ttl
ElastAlert Kibana Plugin

Visualizations:

3D Chars
3D Graph
Bmap
C3JS Visualzations
Calendar Visualzation
Cohort analysis
Collored Metilc Visualzation
Dendrogram
Dotplot
Dropdown
Enhanced Table
Enhanced Themap
Extended Metric

学习在本机Docker环境中云高兴 ELK Stack

Docker-compose 相关命令
- 运行 docker-compose up [-d]
- docker-compose down
- docker-compose down -v
- docker stop / rm containerID

Demo

运行 Docker-compose 本地构建 Elasticsearch 分布式特性
集成 Cerobro，方便查看集群状态，默认运行在 9000端口，可通过http://localhost:9000

logstash

2.4-Logstash安装与导入数据

参考文档说明导入 movices.csv 数据，启动 logstash

bin/logstash -f path_logstash.conf

input {
  file {
    path => "D:/dev/data/movies.csv"
    start_position => "beginning"
	sincedb_path => "D:/dev/logstash/mydata/moive.txt"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {

    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "movies"
     document_id => "%{id}"
   }
  stdout {}
}

[2019-10-23T09:58:45,519][ERROR][logstash.javapipeline    ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<ArgumentError: The "sincedb_path" argument must point to a file, received a directory: "D:/dev/logstash/mydata/moive">,

sincedb_path 需要是一个文件，而不是一个目录

文檔CURD

//create document.自动生成 _id
POSt users/_doc
{
  "user": "Mkie",
  "post_date": "2019-10-23 00:01:22",
  "message": "hello"
}

//create document,指定id,如果id存在，报错

PUT users/_doc/1?op_type=create
{
  "user": "kangkang",
  "post_date": "2019-10-23 02:01:22",
  "message": "hello2"
}


//create document,指定id已经存在，就报错
PUT users/_create/1
{
  "user": "lulu",
  "post_date": "2019-10-23 03:01:22",
  "message": "hello3"
}

//get document by id
GET users/_doc/1


PUT users/_doc/1
{
  "user": "my mike"
}

//在原文档增加字段


POST users/_update/1
{
  "doc":{
    "post_date" : "2019-10-10 23:56:56",
    "message": "just message"
  }
}


DELETE usrs/_doc/1

bulk 操作

每个操作的失败不会影响其它操作

批量读取 - mget

GEt /_mget
{
  "docs":[
  	{
  	   "_index" : "user",
  	   "_id": 1,
  	},
  	{
  	   "_index": "comments",
  	   "_id": 1
  	}
  ]
}

msearch _msearch

正排索引和倒排索引

analysis与analyzer

analyzer 分词器

使用 analyzer api

get /_analyze

POST books/_analyzer
{
 "filed": "title"
}

Standard Analyzer

默认分词器
按词划分
小写处理

Simple Analyzer

按照非字母，非字母都被去除

Whitespace Analyzer

按照空格划分

Stop Analyzer

相比Simple Analyzer

language analyzer

icu analyzer

需要按照插件

elasticsearch-plugin install analysis-icu

提供了Unicode的支持，更好的支持亚洲语音

更多的中文分词器

IK
THULAC

//standard
GET _analyze
{
  "analyzer": "standard",
  "text": "He my polite be object oh change. Consider no mr am overcame yourself throwing sociable children. Hastily her totally conduct may. "
}

//simple
GET _analyze
{
  "analyzer": "simple",
  "text": "He my polite be object oh change. Consider no mr am overcame yourself throwing sociable children. Hastily her totally conduct may. "
}

//stop
GET _analyze
{
  "analyzer": "stop",
  "text": "Six started far placing saw respect females old. Civilly why how end viewing attempt related enquire visitor. Man particular insensible celebrated conviction stimulated principles day. "
}

//whitespace
GET _analyze
{
  "analyzer": "whitespace",
  "text": "Six started far placing saw respect females old. Civilly why how end viewing attempt related enquire visitor. Man particular insensible celebrated conviction stimulated principles day."
}

//keyword
GET _analyze
{
  "analyzer": "keyword",
  "text": "Six started far placing saw respect females old. Civilly why how end viewing attempt related enquire visitor. Man particular insensible celebrated conviction stimulated principles day."
}

//pattern,default \w
GET _analyze
{
  "analyzer": "pattern",
  "text": "2 Six started far-placing saw respect females old. Civilly why how end viewing attempt related enquire visitor. Man particular insensible celebrated conviction stimulated principles day."
}

GET _analyze
{
  "analyzer": "english",
  "text": "2 Six started far-placing saw respect females old. Civilly why how end viewing attempt related enquire visitor. Man particular insensible celebrated conviction stimulated principles day."
}

GET _analyze
{
  "analyzer": "icu_analyzer",
  "text": "他說的的確在理“"
}

GET _analyze
{
  "analyzer": "standard",
  "text": "他說的的確在理“"
}

search api

URL Search 使用 q
Request Body search

/_search

/index1/_search

URL Search

q
df 默认字段
sort
profile

GET /movies/_search?q=2012&df=title
{
  "profile":"true"
}

//泛查询
GET /movies/_search?q=2012
{
  "profile":"true"
}

//指定查询
GET /movies/_search?q=title:2012
{
  "profile":"true"
}

//使用引号
GET /movies/_search?q=title:"Beautiful Mind"
{
  "profile":"true"
}

//查找美丽心灵，Mind 泛查询
GET /movies/_search?q=title:Beautiful Mind
{
  "profile":"true"
}

//分组，bool查询
GET /movies/_search?q=title:(Beautiful Mind)
{
  "profile":"true"
}


//查找美丽心灵
GET /movies/_search?q=title:(Beautiful AND Mind)
{
  "profile":"true"
}


GET /movies/_search?q=title:(Beautiful NOT Mind)
{
  "profile":"true"
}

//%2B 加号
GET /movies/_search?q=title:(Beautiful %2BMind)
{
  "profile":"true"
}

GET /movies/_search?q=year:>=1990
{
  "profile":"true"
}

GET /movies/_search?q=title:b*
{
  "profile":"true"
}

GET /movies/_search?q=title:beautiful~1
{
  "profile":"true"
}

GET /movies/_search?q=title:"Lord Rings"~2
{
  "profile":"true"
}

Request body search

POST kibana_sample_data_ecommerce/_search
{
  "_source": ["order_date"],
  "sort": [
    {
      "order_date": {
        "order": "desc"
      }
    }
  ],
  "query": {
    "match_all": {}
  }
}

//脚本字段
GET kibana_sample_data_ecommerce/_search
{
  "script_fields": {
    "new_field": {
      "script": {
        "lang":"painless",
        "source": "doc['order_date'].value+'_hello'"
      }
    }
  }, 
  "query": {
    "match_all": {}
  }
}

POST movies/_search
{
  "query": {
    "match": {
      "title": "Last Christmas"
    }
  }
}

POST movies/_search
{
  "query": {
    "match": {
      "title": {
        "query": "Last Christmas", 
        "operator": "AND"
      }
      
    }
  }
}

POST movies/_search
{
  "query": {
    "match_phrase": {
      "title": {
        "query": "one love"
      }
      
    }
  }
}

POST movies/_search
{
  "query": {
    "match_phrase": {
      "title": {
        "query": "one love",
        "slop": 1
      }
      
    }
  }
}

query_string

simple query string query

posted @ 2019-10-25 22:27 五毛钱的饼阅读(410) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

wakasann

elasticsearch学习笔记001

如何在开发机上运行多个 Elasticsearch实例

kibana

学习在本机Docker环境中云高兴 ELK Stack

logstash

公告