Prometheus断电后启动异常 Error on ingesting samples that are too old or are too far into the future

错误日志

level=warn ts=2020-10-09T02:29:06.700Z caller=scrape.go:1216 component="scrape manager" scrape_pool=service.server target=http://192.168.1.22:9100/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=929 level=warn ts=2020-10-09T02:29:06.700Z caller=scrape.go:987 component="scrape manager" scrape_pool=service.server target=http://192.168.1.22:9100/metrics msg="appending scrape report failed" err="out of bounds"

 

日志的时间戳与当前时间戳提前了8个小时： level=warn ts=2020-10-09T02:29:06.700Z

服务器长时间断电导致的 tsdb 记录日期太旧

# 创建一个新的 tsdb 数据目录 mkdir /root/prometheus-2.16.0/tsdb_data/ # 启动时指定 tsdb 数据目录为刚刚创建的 /root/prometheus-2.16.0/prometheus \ --config.file="/root/prometheus-2.16.0/prometheus.yml" \ --storage.tsdb.path="/root/prometheus-2.16.0/tsdb_data" \ --web.enable-lifecycle \ --web.enable-admin-api &