ES 字符过滤器&令牌过滤器

1、字符过滤器 官方文档

其作用主要是在调用分词器进行分词之前，进行一些无用字符的过滤,字符过滤器主要分为以下三种

(1)、Html strip 官方文档

过滤html标签，主要参数escaped_tags保留哪些html标签,示例代码如下:

PUT test_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer":{
          //指定分词器
          "tokenizer":"keyword",
          //指定分析器的字符串过滤器
          "char_filter":"custom_char_filter"
        }
      },
      //字符过滤器
      "char_filter": {
        "custom_char_filter":{
          //字符过滤器的类型
          "type":"html_strip",
          //跳过过滤的html标签
          "escaped_tags": [
            "a"
          ]
        }
      }
    }
  }
}

测试过滤器代码:

GET test_index/_analyze
{
  "analyzer": "custom_analyzer",
  "text": ["this is address of baidu<a>baidu</a><p>baidu content</p>"]
}

执行结果如下:

{
  "tokens" : [
    {
      "token" : """this is address of baidu<a>baidu</a>
baidu content
""",
      "start_offset" : 0,
      "end_offset" : 56,
      "type" : "word",
      "position" : 0
    }
  ]
}

从结果中可以看出过滤了除a标签之外的所有html标签.

 

(2)、Mapping 官方文档

常用于敏感词过滤,示例代码如下:

PUT test_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer":{
          "tokenizer":"keyword",
          "char_filter":["custom_char_filter","custom_mapping_filter"]
        }
      },
      "char_filter": {
        "custom_char_filter":{
          "type":"html_strip",
          "escaped_tags": [
            "a"
          ]
        },
        "custom_mapping_filter":{
          "type": "mapping",
          //当内容出现baidu或者is 全都用**替换
          "mappings": [
            "baidu=>**",
            "is=>**"
          ]
        }
      }
    }
  }
}

执行搜索代码如下:

GET test_index/_analyze
{
  "analyzer": "custom_analyzer",
  "text": ["this is address of baidu<a>baidu</a><p>baidu content</p>"]
}

执行结果如下:

{
  "tokens" : [
    {
      "token" : """th** ** address of **<a>**</a>
** content
""",
      "start_offset" : 0,
      "end_offset" : 56,
      "type" : "word",
      "position" : 0
    }
  ]
}

在html_strip的基础上,通过mapping完成了baidu和is的敏感词过滤.

 

(3)、Pattern Replace 官方文档

主要用于一些结构化的内容(可以用正则表达式检索到的)的替换,示例代码如下:

PUT test_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer":{
          "tokenizer":"keyword",
          "char_filter":["custom_char_filter","custom_mapping_filter","custom_pattern_replace_filter"]
        }
      },
      "char_filter": {
        "custom_char_filter":{
          "type":"html_strip",
          "escaped_tags": [
            "a"
          ]
        },
        "custom_mapping_filter":{
          "type": "mapping",
          "mappings": [
            "baidu=>**",
            "is=>**"
          ]
        },
        "custom_pattern_replace_filter":{
          "type":"pattern_replace",
          "pattern": "(\\d{3})\\d{4}(\\d{4})",
          "replacement": "$1****$2"
        }
      }
    }
  }
}

在(1)、(2)的基础上增加了custom_pattern_replace_filter用于正则替换内容,主要作用是手机号脱敏

检索代码如下:

GET test_index/_analyze
{
  "analyzer": "custom_analyzer",
  "text": ["this is address of baidu<a>baidu</a><p>baidu content</p>telphone:13311112222"]
}

执行结果如下:

{
  "tokens" : [
    {
      "token" : """th** ** address of **<a>**</a>
** content
telphone:133****2222""",
      "start_offset" : 0,
      "end_offset" : 76,
      "type" : "word",
      "position" : 0
    }
  ]
}

手机号13311112222被替换成了133****2222

 

2、令牌过滤器 官方文档

令牌过滤器包含的内容过多,参考官方文档,这里分析几种常用的令牌过滤器

(1)、同义词过滤器 synonym

第一步向运行目录的config文件夹下添加analysis文件夹,再到此文件夹下添加synonym.txt文件,集群下所有节点重复此操作,内容如下:

 

 第二步编写相关的设置指向同义词文件

PUT test_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer":{
          "tokenizer":"ik_max_word",
          "filter":["synonym"]
        }
      },
      "filter": {
        "synonym":{
          "type": "synonym",
          "synonyms_path": "analysis/synonym.txt"
        }
      }
    }
  }
}

这里用了ik分词器,不明白参考ES 中文分词器ik

新增索引后,执行搜索代码

GET test_index/_analyze
{
  "analyzer": "custom_analyzer",
  "text":"啦啦啦,呵呵呵,啧啧啧"
}

结果如下:

{
  "tokens" : [
    {
      "token" : "嘻嘻",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "SYNONYM",
      "position" : 0
    },
    {
      "token" : "嘻嘻",
      "start_offset" : 4,
      "end_offset" : 6,
      "type" : "SYNONYM",
      "position" : 1
    },
    {
      "token" : "嘎嘎",
      "start_offset" : 4,
      "end_offset" : 6,
      "type" : "SYNONYM",
      "position" : 1
    },
    {
      "token" : "嘎嘎",
      "start_offset" : 5,
      "end_offset" : 7,
      "type" : "SYNONYM",
      "position" : 2
    },
    {
      "token" : "么",
      "start_offset" : 8,
      "end_offset" : 10,
      "type" : "SYNONYM",
      "position" : 3
    },
    {
      "token" : "么",
      "start_offset" : 9,
      "end_offset" : 11,
      "type" : "SYNONYM",
      "position" : 4
    },
    {
      "token" : "么",
      "start_offset" : 9,
      "end_offset" : 11,
      "type" : "SYNONYM",
      "position" : 5
    }
  ]
}

结果中所有的相关词汇都被执行的词汇替换

 

(2)、停用词stop 官方文档

在设置中指定的停用词,将不会创建倒排索引.

PUT test_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer":{
          "tokenizer":"ik_max_word",
          "filter":["custom_stop_filter"]
        }
      },
      "filter": {
        "custom_stop_filter":{
          "type": "stop",
          "ignore_case": true,
          "stopwords": [ "and", "is","friend" ]
        }
      }
    }
  }
}

执行以上代码,并执行以下搜索语句

GET test_index/_analyze
{
  "analyzer": "custom_analyzer",
  "text":"You and me IS friend"
}

执行结果如下:

{
  "tokens" : [
    {
      "token" : "you",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "ENGLISH",
      "position" : 0
    },
    {
      "token" : "me",
      "start_offset" : 8,
      "end_offset" : 10,
      "type" : "ENGLISH",
      "position" : 1
    }
  ]
}

注：也可以指定停用词文件路劲,和ik分词器类似.具体参考官方文档.