java.lang.IllegalStateException in ES + Zipkin Environment

java.lang.IllegalStateException in ES + Zipkin Environment

Issue

When I integrate Spring Cloud Slueth + Kafka + OpenZipkin + Elasticsearch, ES is the storage of OpenZipkin, I met this error when Openzipkin write trace log to ES using "zipkin-autoconfigure-storage-elasticsearch-http".

"error":{"type":"mapper_parsing_exception","reason":"failed to parse [timestamp]","caused_by":{"type":"json_parse_exception","reason":"Numeric value (1523409146837000) out of range of int\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@781cb877; line: 1, column: 616]"}}}}]}

at zipkin2.elasticsearch.internal.HttpBulkIndexer$CheckForErrors.convert(HttpBulkIndexer.java:74)
at zipkin2.elasticsearch.internal.HttpBulkIndexer$CheckForErrors.convert(HttpBulkIndexer.java:69)
at zipkin2.elasticsearch.internal.client.HttpCall.parseResponse(HttpCall.java:148)
at zipkin2.elasticsearch.internal.client.HttpCall$V2CallbackAdapter.onResponse(HttpCall.java:125)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

This is caused by ES convert JSON integer/long number field to ES integer field in my environment.

By default, ES dynamic field mapping is enabled, some simple rules are used to determine which datatype the field should have:

JSON datatype	Elasticsearch datatype
null	No field is added.
true or false	boolean field
floating point number	float field
integer	long field
object	object field
array	Depends on the first non-null value in the array.
string	Either a date field (if the value passes date detection), a double or long field (if the value passes numeric detection) or a text field, with a keyword sub-field.

Please refer https://www.elastic.co/guide/en/elasticsearch/reference/master/dynamic-field-mapping.html#numeric-detection

So, by default, ES will map the JSON integer/long to ES long, the default rule must be override in my invironment.

Through checking the Dynamic templates configuration by "GET /_template" in kibana dev tool console, I find there is a template for other usage as below 

{
  "log-index-template": {
    "order": 0,
    "version": 1,
    "template": "*",
    "settings": {
      "index": {
        "number_of_shards": "6",
        "number_of_replicas": "2",
        "refresh_interval": "5s"
      }
    },
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "integers": {
              "match_mapping_type": "long",
              "mapping": {
                "type": "integer"
              }
            }
          },
          {
            "strings": {
              "match_mapping_type": "string",
              "mapping": {
                "type": "text",
                "fields": {
                  "raw": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "include_in_all": false,
            "type": "date"
          },
          "@version": {
            "include_in_all": false,
            "type": "keyword"
          }
        },
        "_all": {
          "enabled": true,
          "norms": false
        }
      }
    },
    "aliases": {}
  }
}

It is this template caused the issue.

Fix　　

Option1:  Change "template": "*" to "template": "log*" to make it only affect log* index so that zipkin* index not affected by it.

Option2: Remove the defined mapping "long" to "int" setting.