elasticsearch因同义词分词组合引起的短语匹配顺序一致性问题(ik_max_word与拼音插件带来的插入错位)

1、环境说明

elasticsearch版本8.17.2，启用ik分词器的远程热更新自定义词汇和停用词词汇。

[root@xsh-test config]# cat analysis-ik/IKAnalyzer.cfg.xml 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
        <comment>IK Analyzer 扩展配置</comment>
        <!--用户可以在这里配置自己的扩展字典 -->
        <!--<entry key="ext_dict">

        </entry>-->
         <!--用户可以在这里配置自己的扩展停止词字典-->
        <!--<entry key="ext_stopwords">

        </entry>-->
        <!--用户可以在这里配置远程扩展字典 -->
        <entry key="remote_ext_dict">https://**********/***********/file/system/es/dict/no_cut_word.dic</entry>
        <!--用户可以在这里配置远程扩展停止词字典-->
        <entry key="remote_ext_stopwords">https://**********/**************/file/system/es/dict/stop_word.dic</entry>
</properties>

docker配置，让数据、插件、配置都能持久化

elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.17.2
    container_name: elasticsearch
    restart: always
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms1g -Xmx2g
      - xpack.security.enabled="false"
      - TZ=Asia/Shanghai
    volumes:
      - /docker/continew-admin-prod/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - /docker/continew-admin-prod/elasticsearch/config/analysis:/usr/share/elasticsearch/config/analysis
      - /docker/continew-admin-prod/elasticsearch/config/analysis-ik:/usr/share/elasticsearch/config/analysis-ik
      - /docker/continew-admin-prod/elasticsearch/data:/usr/share/elasticsearch/data
      - /docker/continew-admin-prod/elasticsearch/plugins:/usr/share/elasticsearch/plugins
    privileged: true
    ports:
      - "9200:9200"
      - "9300:9300"
    #内存限制
    deploy:
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 50M

2、问题重现

DELETE dev_second_hand_category

PUT dev_second_hand_category
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "content_ik_max_word_pinyin": {
            "type": "custom",
            "tokenizer": "ik_max_word",
            "filter": [
              "my_pinyin_filter"
            ]
          },
          "keyword_ik_max_word_synonym": {
            "type": "custom",
            "tokenizer": "ik_max_word",
            "filter": [
              "my_synonym_filter"
            ]
          },
          "content_ik_smart_pinyin": {
            "type": "custom",
            "tokenizer": "ik_smart",
            "filter": [
              "my_pinyin_filter"
            ]
          },
          "keyword_ik_smart_synonym": {
            "type": "custom",
            "tokenizer": "ik_smart",
            "filter": [
              "my_synonym_filter"
            ]
          }
        },
        "filter": {
          "my_synonym_filter": {
            "type": "synonym_graph",
            "updateable": true,
            "synonyms_path": "analysis/synonyms.txt"
          },
          "my_pinyin_filter": {
            "type": "pinyin",
            "keep_separate_first_letter": false,
            "keep_full_pinyin": true,
            "keep_joined_full_pinyin": true,
            "keep_original": true,
            "limit_first_letter_length": 16,
            "lowercase": true,
            "remove_duplicated_term": true
          }
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "id": { "type": "keyword" },
      "name": {
        "type": "text",
        "analyzer": "content_ik_smart_pinyin",#索引使用ik_smart_pinyin来分词扩展
        "search_analyzer": "keyword_ik_smart_synonym" #搜索用同义词扩展并ik_smart分词
      },
      "description": {
        "type": "text",
        "analyzer": "content_ik_smart_pinyin",
        "search_analyzer": "keyword_ik_smart_synonym"
      },
      "parentId": { "type": "keyword" },
      "level": { "type": "integer" },
      "sort": { "type": "integer" },
      "status": { "type": "keyword" }
    }
  }

同义词表有一个“智能手环,运动手环,fitnessband”的词条组

此时

GET dev_second_hand_category/_validate/query?explain
{
  "query": {
    "match": {
      "description": "智能手环"
    }
  }
}

结果

{
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "valid": true,
  "explanations": [
    {
      "index": "dev_second_hand_category",
      "valid": true,
      "explanation": """(description:"运动 手环" description:fitnessband description:"智能 手环")"""
    }
  ]
}

输入：智能手环

↓

同义词扩展：

智能手环 OR 运动手环 OR fitnessband

↓

再经过 ik_smart 分词：

智能手环 → 智能、手环
运动手环 → 运动、手环
fitnessband → fitnessband

最终组合成查询语句：

(description:"运动手环" description:fitnessband description:"智能手环")

此时的match查询变成了三个or合理，但是注意"智能手环"变成了短语匹配需要必须智能和手环连在一起

3、原因

然后我们看索引里面的内容

POST /dev_second_hand_category/_analyze
{
  "field": "name",
  "text": "智能手环"
}

结果是

{
  "tokens": [
    {
      "token": "zhi",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "智能",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "zhineng",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "zn",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "neng",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 1
    },
    {
      "token": "shou",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 2
    },
    {
      "token": "huan",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 3
    },
    {
      "token": "手环",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 3
    },
    {
      "token": "shouhuan",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 3
    },
    {
      "token": "sh",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 3
    }
  ]
}

智能和手环的位置变成了 0和3，也就是明明看着是挨着的，但是死活查不出来

GET dev_second_hand_category/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "status": {
              "value": "ENABLE"
            }
          }
        },
        {
          "term": {
            "level": {
              "value": 3
            }
          }
        }
      ],
      "minimum_should_match": "1",
      "should": [
        {
          "match": {
            "name": {
              "boost": 5,
              "query": "智能手环"

            }
          }
        },
        {
          "match": {
            "description": {
              "boost": 1,
              "query": "智能手环"

            }
          }
        }
      ]
    }
  },
  "highlight": {
    "fields": {
      "name": {},
      "description": {}
    },
    "pre_tags": [
      "<red>"
    ],
    "post_tags": [
      "</red>"
    ]
  }
}

结果就是空

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

4、最终配置

发现了问题以后，我的解决方案是将拼音单独放在一个子字段，然后这个子字段的搜索和索引都用ik_smart+拼音来分词扩展，查询的时候同时匹配name和name.pinyin既能保证同义词又能保证正常分词匹配

{
    "settings": {
        "index": {
            "analysis": {
                "analyzer": {
                    "index_ik_max_word_pinyin": {
                        "type": "custom",
                        "tokenizer": "ik_max_word",
                        "filter": [
                            "my_pinyin_filter"
                        ]
                    },
                    "index_ik_smart_pinyin": {
                        "type": "custom",
                        "tokenizer": "ik_smart",
                        "filter": [
                            "my_pinyin_filter"
                        ]
                    },
                    "keyword_ik_smart_synonym": {
                        "type": "custom",
                        "tokenizer": "ik_smart",
                        "filter": [
                            "my_synonym_filter"
                        ]
                    }
                },
                "filter": {
                    "my_synonym_filter": {
                        "type": "synonym_graph",
                        "updateable": true,
                        "synonyms_path": "analysis/synonyms.txt"
                    },
                    "my_pinyin_filter": {
                        "type": "pinyin",
                        "keep_separate_first_letter": false,
                        "keep_full_pinyin": true,
                        "keep_joined_full_pinyin": true,
                        "keep_original": false,
                        "limit_first_letter_length": 16,
                        "lowercase": true,
                        "remove_duplicated_term": true
                    }
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "id": {
                "type": "keyword"
            },
            "name": {
                "type": "text",
                "analyzer": "ik_smart",
                "search_analyzer": "keyword_ik_smart_synonym",
                "fields": {
                    "pinyin": {
                        "type": "text",
                        "analyzer": "index_ik_smart_pinyin",
                        "search_analyzer": "index_ik_smart_pinyin"
                    }
                }
            },
            "description": {
                "type": "text",
                "analyzer": "ik_max_word",
                "search_analyzer": "keyword_ik_smart_synonym",
                "fields": {
                    "pinyin": {
                        "type": "text",
                        "analyzer": "index_ik_smart_pinyin",
                        "search_analyzer": "index_ik_smart_pinyin"
                    }
                }
            },
            "parentId": {
                "type": "keyword"
            },
            "level": {
                "type": "integer"
            },
            "sort": {
                "type": "integer"
            },
            "status": {
                "type": "keyword"
            }
        }
    }
}

再看name索引的结果

POST /dev_second_hand_category/_analyze
{
  "field": "name",
  "text": "智能手环"
}

{
  "tokens": [
    {
      "token": "智能",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "手环",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 1
    }
  ]
}

POST /dev_second_hand_category/_analyze
{
  "field": "name.pinyin",
  "text": "智能手环"
}

{
  "tokens": [
    {
      "token": "zhi",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "zhineng",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "zn",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "neng",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 1
    },
    {
      "token": "shou",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 2
    },
    {
      "token": "huan",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 3
    },
    {
      "token": "shouhuan",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 3
    },
    {
      "token": "sh",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 3
    }
  ]
}

再看搜索结果

GET dev_second_hand_category/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "status": {
              "value": "ENABLE"
            }
          }
        },
        {
          "term": {
            "level": {
              "value": 3
            }
          }
        }
      ],
      "minimum_should_match": "1",
      "should": [
        {
          "match": {
            "name": {
              "boost": 50,
              "query": "智能手环"
            }
          }
        },
        {
          "match": {
            "description": {
              "boost": 10,
              "query": "智能手环"
            }
          }
        },
        {
          "match": {
            "name.pinyin": {
              "boost": 5,
              "query": "智能手环"
            }
          }
        },
        {
          "match": {
            "description.pinyin": {
              "boost": 1,
              "query": "智能手环"
            }
          }
        }
      ]
    }
  },
  "highlight": {
    "fields": {
      "name": {},
      "description": {},
      "name.pinyin": {},
      "description.pinyin": {}
    },
    "pre_tags": [
      "<red>"
    ],
    "post_tags": [
      "</red>"
    ]
  }
}

{
  "took": 23,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 465,
      "relation": "eq"
    },
    "max_score": 635.59906,
    "hits": [
      {
        "_index": "dev_second_hand_category",
        "_id": "35",
        "_score": 635.59906,
        "_source": {
          "_class": "top.continew.admin.system.model.es.SecondHandCategoryES",
          "id": 35,
          "name": "智能手环",
          "description": "智能手环 小米手环9 华为手环 Fitbit 手环 运动手环 健康监测 心率 睡眠 计步 智能穿戴 血氧手环 苹果 华为 小米 三星 Fitbit 石头 科沃斯 大疆 Meta PICO",
          "parentId": 32,
          "level": 3,
          "sort": 2,
          "status": "ENABLE"
        },
        "highlight": {
          "name.pinyin": [
            "<red>智能</red><red>手环</red>"
          ],
          "name": [
            "<red>智能手环</red>"
          ],
          "description.pinyin": [
            "<red>智能</red><red>手环</red> 小米<red>手环</red>9 华为<red>手环</red> Fitbit <red>手环</red> 运动<red>手环</red> 健康监测 心率 睡眠 计步 <red>智能</red>穿戴 血氧<red>手环</red> 苹果 华为 小米 三星 Fitbit 石头 科沃斯 大疆 Meta PICO"
          ]
        }
      },
      {
        "_index": "dev_second_hand_category",
        "_id": "33",
        "_score": 77.008995,
        "_source": {
          "_class": "top.continew.admin.system.model.es.SecondHandCategoryES",
          "id": 33,
          "name": "智能手表",
          "description": "智能手表 AppleWatchSeries11 AppleWatchSE3 华为WATCHGT6 华为WatchUltimate 小米手表 三星GalaxyWatch8 运动手表 健康监测 心率 血氧 GPS 通话 智能穿戴 eSIM 睡眠监测智能手表 苹果 华为 小米 三星 Fitbit 石头 科沃斯 大疆 Meta PICO",
          "parentId": 32,
          "level": 3,
          "sort": 0,
          "status": "ENABLE"
        },
        "highlight": {
          "name.pinyin": [
            "<red>智能</red><red>手表</red>"
          ],
          "description.pinyin": [
            "<red>智能</red><red>手表</red> AppleWatchSeries11 AppleWatchSE3 华为WATCHGT6 华为WatchUltimate 小米<red>手表</red> 三星GalaxyWatch8 运动<red>手表</red> 健康监测 心率 血氧",
            "GPS 通话 <red>智能</red>穿戴 eSIM 睡眠监测<red>智能</red><red>手表</red> 苹果 华为 小米 三星 Fitbit 石头 科沃斯 大疆 Meta PICO"
          ]
        }
      },
      {
        "_index": "dev_second_hand_category",
        "_id": "34",
        "_score": 59.924656,
        "_source": {
          "_class": "top.continew.admin.system.model.es.SecondHandCategoryES",
          "id": 34,
          "name": "智能音箱",
          "description": "智能音箱 小爱同学 小度 天猫精灵 小米音箱 华为音箱 苹果HomePod 智能语音 智能家居 语音助手 蓝牙音箱智能音箱 苹果 华为 小米 三星 Fitbit 石头 科沃斯 大疆 Meta PICO",
          "parentId": 32,
          "level": 3,
          "sort": 1,
          "status": "ENABLE"
        },
        "highlight": {
          "name.pinyin": [
            "<red>智能</red>音箱"
          ],
          "description.pinyin": [
            "<red>智能</red>音箱 小爱同学 小度 天猫精灵 小米音箱 华为音箱 苹果HomePod <red>智能</red>语音 <red>智能家居</red> 语音<red>助手</red> 蓝牙音箱<red>智能</red>音箱 苹果 华为 小米 三星 Fitbit 石头 科沃斯 大疆 Meta PICO"
          ]
        }
      },
      {
        "_index": "dev_second_hand_category",
        "_id": "482",
        "_score": 55.29284,
        "_source": {
          "_class": "top.continew.admin.system.model.es.SecondHandCategoryES",
          "id": 482,
          "name": "智能马桶盖",
          "description": "智能马桶盖 智能马桶坐便器盖板 加热马桶盖 洁身器 即热式马桶盖 储热式马桶盖 松下马桶盖 TOTO智能盖板 洗之朗 智洁马桶盖 暖风烘干马桶盖 臀洗女性清洗 自动除臭马桶盖智能马桶盖 卫浴智能设备 苹果 华为 小米 三星 Fitbit 石头 科沃斯 大疆 Meta PICO",
          "parentId": 477,
          "level": 3,
          "sort": 4,
          "status": "ENABLE"
        },
        "highlight": {
          "name.pinyin": [
            "<red>智能</red>马桶盖"
          ],
          "description.pinyin": [
            "<red>智能</red>马桶盖 <red>智能</red>马桶坐便器盖板 加热马桶盖 洁身器 即热式马桶盖 储热式马桶盖 松下马桶盖 TOTO<red>智能</red>盖板 洗之朗 <red>智</red>洁马桶盖 暖风烘干马桶盖 臀洗女性清洗 自动除臭马桶盖<red>智能</red>马桶盖 卫浴<red>智能</red>设备 苹果"
          ]
        }
      },
      {
        "_index": "dev_second_hand_category",
        "_id": "38",
        "_score": 54.728516,
        "_source": {
          "_class": "top.continew.admin.system.model.es.SecondHandCategoryES",
          "id": 38,
          "name": "智能机器人",
          "description": "智能机器人 机器人 教育机器人 编程机器人 陪聊机器人 扫地机器人 AI机器人 智能陪伴 石头 科沃斯机器人 苹果 华为 小米 三星 Fitbit 石头 科沃斯 大疆 Meta PICO",
          "parentId": 32,
          "level": 3,
          "sort": 5,
          "status": "ENABLE"
        },
        "highlight": {
          "name.pinyin": [
            "<red>智能</red>机器人"
          ],
          "description.pinyin": [
            "<red>智能</red>机器人 机器人 教育机器人 编程机器人 陪聊机器人 扫地机器人 AI机器人 <red>智能</red>陪伴 石头 科沃斯机器人 苹果 华为 小米 三星 Fitbit 石头 科沃斯 大疆 Meta PICO"
          ]
        }
      },
      {
        "_index": "dev_second_hand_category",
        "_id": "364",
        "_score": 43.638157,
        "_source": {
          "_class": "top.continew.admin.system.model.es.SecondHandCategoryES",
          "id": 364,
          "name": "腮红",
          "description": "腮红 腮红粉 腮红膏 气质腮红 腮红盘 花知晓 3CE 彩妆 化妆品",
          "parentId": 359,
          "level": 3,
          "sort": 4,
          "status": "ENABLE"
        },
        "highlight": {
          "name.pinyin": [
            "<red>腮红</red>"
          ],
          "description.pinyin": [
            "<red>腮红</red> 腮红粉 <red>腮红</red>膏 <red>气质</red><red>腮红</red> 腮红盘 花<red>知晓</red> 3CE 彩妆 化妆品"
          ]
        }
      },
      {
        "_index": "dev_second_hand_category",
        "_id": "165",
        "_score": 42.126217,
        "_source": {
          "_class": "top.continew.admin.system.model.es.SecondHandCategoryES",
          "id": 165,
          "name": "耳环",
          "description": "耳环 金属耳环 珍珠耳环 水晶耳环 长款耳环 耳饰耳环",
          "parentId": 154,
          "level": 3,
          "sort": 10,
          "status": "ENABLE"
        },
        "highlight": {
          "name.pinyin": [
            "<red>耳环</red>"
          ],
          "description.pinyin": [
            "<red>耳环</red> 金属<red>耳环</red> 珍珠<red>耳环</red> 水晶<red>耳环</red> 长款<red>耳环</red> 耳饰<red>耳环</red>"
          ]
        }
      },
      {
        "_index": "dev_second_hand_category",
        "_id": "876",
        "_score": 41.609035,
        "_source": {
          "_class": "top.continew.admin.system.model.es.SecondHandCategoryES",
          "id": 876,
          "name": "转换器",
          "description": "转换器 电源转换 插头转换器 电压转换 电气配件 万能转换插头 出国转换器 220V转110V 110V转220V DC-DC转换器 AC-DC转换器 欧标转换器 美标转换器 英标转换器转换器 电气配件",
          "parentId": 867,
          "level": 3,
          "sort": 8,
          "status": "ENABLE"
        },
        "highlight": {
          "name.pinyin": [
            "<red>转换器</red>"
          ],
          "description.pinyin": [
            "<red>转换器</red> 电源<red>转换</red> 插头<red>转换器</red> 电压<red>转换</red> 电气配件 <red>万能</red><red>转换</red>插头 出国<red>转换器</red> 220V转110V 110V转220V DC-DC<red>转换器</red> AC-DC<red>转换器</red> 欧标<red>转换器</red> 美标<red>转换器</red> 英标<red>转换器</red><red>转换器</red> 电气配件"
          ]
        }
      },
      {
        "_index": "dev_second_hand_category",
        "_id": "91",
        "_score": 38.72568,
        "_source": {
          "_class": "top.continew.admin.system.model.es.SecondHandCategoryES",
          "id": 91,
          "name": "交换机",
          "description": "交换机 网络交换机 千兆交换机 万兆交换机 TP-Link 华为 H3C 锐捷 企业交换机 家用交换机 PoE交换机 网络设备交换机 TP-Link 华为 小米 锐捷 ASUS 思科 H3C 群晖 威联通",
          "parentId": 87,
          "level": 3,
          "sort": 3,
          "status": "ENABLE"
        },
        "highlight": {
          "name.pinyin": [
            "<red>交换机</red>"
          ],
          "description.pinyin": [
            "<red>交换机</red> 网络<red>交换机</red> 千兆<red>交换机</red> 万兆<red>交换机</red> TP-Link 华为 H3C 锐捷 企业<red>交换机</red> 家用<red>交换机</red> PoE<red>交换机</red> 网络设备<red>交换机</red> TP-Link 华为 小米 锐捷 ASUS 思科 H3C 群晖 威联通"
          ]
        }
      },
      {
        "_index": "dev_second_hand_category",
        "_id": "676",
        "_score": 32.750076,
        "_source": {
          "_class": "top.continew.admin.system.model.es.SecondHandCategoryES",
          "id": 676,
          "name": "功能饮料",
          "description": "功能饮料 红牛 脉动 维他命水 能量饮料 运动饮料 宝矿力水特 尖叫 动力怪兽 魔爪 东鹏特饮 乐虎 战马 电解质水 运动补水 抗疲劳饮料功能饮料 饮品",
          "parentId": 673,
          "level": 3,
          "sort": 2,
          "status": "ENABLE"
        },
        "highlight": {
          "name.pinyin": [
            "<red>功能</red>饮料"
          ],
          "description.pinyin": [
            "<red>功能</red>饮料 红牛 脉动 维他命水 <red>能量</red>饮料 运动饮料 宝矿力水特 尖叫 动力<red>怪兽</red> 魔爪 东鹏特饮 乐虎 战马 <red>电解质</red>水 运动补水 抗疲劳饮料<red>功能</red>饮料 饮品"
          ]
        }
      }
    ]
  }
}

完美匹配

posted @ 2026-04-23 22:05 大背头阅读(16) 评论(0) 收藏举报

刷新页面返回顶部

大背头

elasticsearch因同义词分词组合引起的短语匹配顺序一致性问题(ik_max_word与拼音插件带来的插入错位)

公告