[OLAP/Doris] Doris FAQ

FAQ for Doris

Q: Doris数据写入失败，日志报SQLException: errCode = 2, detailMessage = disk /opt/software/apache-doris-2.0.4-bin-x64/be/storage on backend xxx exceed limit usage, path hash: xxxxxxxxx

  • 问题描述

Flink 向 Doris 集群写数据失败，日志报: SQLException: errCode = 2, detailMessage = disk /opt/software/apache-doris-2.0.4-bin-x64/be/storage on backend xxx exceed limit usage, path hash: xxxxxxxxx

详情日志:

/tmp/tm_flink-313956-taskmanager-1-1-735c5e4c-840d-45f9-923d-9ef169d19548/tmp/job_d74af21325bcc0ba60c74bbf55465edd_op_StreamSink_f860a740dbef102e1cd269b1641573dd__2_2__uuid_adc01b95-2722-4c5c-8b4d-4726ddc78155.
2095-10-17 15:20:50,248 INFO  cn.xxxxx.bd.datasource.datasourcex.connector.AbstractConnector 464 [Sink: DeviceHistoryStatusSignals-BigdataDoris-Sink (2/2)#5]  - success to close/release this connection. currentInstanceId: 1
2095-10-17 15:20:50,248 WARN  org.apache.flink.runtime.taskmanager.Task                    1132 [Sink: DeviceHistoryStatusSignals-BigdataDoris-Sink (2/2)#5]  - Sink: DeviceHistoryStatusSignals-BigdataDoris-Sink (2/2)#5 (ce9496be69c2a98259e8eb61ed0cf3da) switched from RUNNING to FAILED with failure cause:
java.lang.RuntimeException: Write the history status signals record to doris fail!deviceId:xxxxxxxxxx, samplingTime:1760434219596, reportTime:1760434229652, executeOperationResult:null, exception.message:java.sql.SQLException: errCode = 2, detailMessage = disk /opt/software/apache-doris-2.0.4-bin-x64/be/storage on backend 10146 exceed limit usage, path hash: 1955778581179018148, insertSql:INSERT INTO bdp_xhx.dwd_device_status_signal_ri(
	sys_code, device_id, sampling_time
	, xms_info, xhx_info, ext_data
	, sampling_interval, sampling_date, report_time
	, trace_id, gateway_message_time
	, ingest_time, parse_time
) VALUES
(  'xxxxx', 'xxxxxxxxxx', 1760434219596
	, '{"xmsMinorVersionNumber":257,"xmsMajorVersionNumber":16973824,"xmsPackCnt":1,"xmsPackInfo":[{"id":"0","isPrimary":true,"xmsState":0,"xmsIPack":4.16,"chargeEndNo":null,"xmsCellCnt":13,"xmsModuleXCellBlock":[3393.0,3392.0,3394.0,3394.0,3395.0,3391.0,3392.0,3393.0,3393.0,3392.0,3394.0,3394.0,3395.0],"xmsCellThremistorCnt":6,"xmsModuleXTempSensor":[38.0,37.0,37.0,37.0,36.0,35.0],"xmsMosTemp":null,"xmsPPack":null,"xmsSoh":100.0,"xmsChgAmount":null,"xmsIsolationResistance":null,"xmsVPackBmu":43.92,"xmsBalanceSwCellBlock":[0,0],"xmsCycleCnt":4,"ratedCurrent":null,"peakCurrent":null,"ratedVolt":null,"ratedCapacity":9.0,"currentCapacity":2.45,"xmsSoc":27.0}]}', 
	'{"speakerState":"...", ... }', 
	'{"simulateSignals":{"xmsInfo.xmsPackInfo":[{"xmsCellVoltAvg":3393.2307,"xmsCellVoltMax":3395.0,"xmsCellVoltMaxIndex":4,"xmsCellVoltMin":3391.0,"xmsCellVoltMinIndex":5,"xmsPackTempAvg":36.666668,"xmsPackTempMax":38.0,"xmsPackTempMaxIdx":0,"xmsPackTempMin":35.0,"xmsPackTempMinIdx":5,"dataType":"simulate","index":0}]}}'
	, 10000, '2095-10-14 09:30:19.596', 1760434229652 
	, '583708e5-eaf9-488e-97c8-3e1bd2e55bad', 1760685646046
	, 1760685649822 , 1760685650235
)
	at cn.xxxxx.bd.dataintegration.flinkapi.sink.xhx.DeviceHistoryStatusSignalDorisSink.invoke(DeviceHistoryStatusSignalDorisSink.java:143) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at cn.xxxxx.bd.dataintegration.flinkapi.sink.xhx.DeviceHistoryStatusSignalDorisSink.invoke(DeviceHistoryStatusSignalDorisSink.java:44) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54) ~[flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233) ~[flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:539) ~[flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:216) ~[flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:829) ~[flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:778) ~[flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) [flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751) [flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573) [flink-dist-1.15.0-h0.cbu.dli.330.20241021.r34.jar:1.15.0-h0.cbu.dli.330.20241021.r34]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422]
Caused by: java.lang.RuntimeException: java.sql.SQLException: errCode = 2, detailMessage = disk /opt/software/apache-doris-2.0.4-bin-x64/be/storage on backend 10146 exceed limit usage, path hash: 1955778581179018148
	at cn.xxxxx.bd.datasource.datasourcex.connector.doris.DorisConnector.executeAndGetAffectedRows(DorisConnector.java:273) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at cn.xxxxx.bd.datasource.datasourcex.connector.AbstractConnector.execute(AbstractConnector.java:311) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at cn.xxxxx.bd.datasource.datasourcex.connector.AbstractConnector.execute(AbstractConnector.java:332) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at cn.xxxxx.bd.dataintegration.flinkapi.sink.xhx.DeviceHistoryStatusSignalDorisSink.invoke(DeviceHistoryStatusSignalDorisSink.java:136) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	... 15 more
Caused by: java.sql.SQLException: errCode = 2, detailMessage = disk /opt/software/apache-doris-2.0.4-bin-x64/be/storage on backend 10146 exceed limit usage, path hash: 1955778581179018148
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at com.mysql.cj.jdbc.StatementImpl.executeUpdateInternal(StatementImpl.java:1335) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at com.mysql.cj.jdbc.StatementImpl.executeLargeUpdate(StatementImpl.java:2084) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at com.mysql.cj.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1246) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at cn.xxxxx.bd.datasource.datasourcex.connector.doris.DorisConnector.executeAndGetAffectedRows(DorisConnector.java:270) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at cn.xxxxx.bd.datasource.datasourcex.connector.AbstractConnector.execute(AbstractConnector.java:311) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at cn.xxxxx.bd.datasource.datasourcex.connector.AbstractConnector.execute(AbstractConnector.java:332) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	at cn.xxxxx.bd.dataintegration.flinkapi.sink.xhx.DeviceHistoryStatusSignalDorisSink.invoke(DeviceHistoryStatusSignalDorisSink.java:136) ~[blob_p-9aebb8bfb056a43c39ba55dade84ab20c1f66207-0d862062367058233c8b80c469908047:?]
	... 15 more
2095-10-17 15:20:50,248 WARN  org.apache.flink.runtime.taskmanager.Task                    1139 [Sink: DeviceHistoryStatusSignals-BigdataDoris-Sink (2/2)#5]  - Call stack:
    at java.lang.Thread.getStackTrace(Thread.java:1564)
    at org.apache.flink.runtime.taskmanager.Task.transitionState(Task.java:1139)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:801)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573)
    at java.lang.Thread.run(Thread.java:750)

...
  • 问题分析
  • 日志提示： backend服务为10146的磁盘空间(disk)超过使用阈值(exceed limit usage) =>即：磁盘空间不足

此阈值，默认为 95%

  • 排查方法

可通过show backends检查磁盘使用情况，重点关注指标字段： AvailCapacity / TotalCapacity / UsedPct / Alive

# SHOW PROC '/backends' 或 show backends

/**
BackendId|Host        |HeartbeatPort|BePort|HttpPort|BrpcPort|ArrowFlightSqlPort|LastStartTime      |LastHeartbeat      |Alive|SystemDecommissioned|TabletNum|DataUsedCapacity|TrashUsedCapcacity|AvailCapacity|TotalCapacity|UsedPct|MaxDiskUsedPct|RemoteUsedCapacity|Tag                     |ErrMsg|Version                    |Status                                                                                                                       |HeartbeatFailureCounter|NodeRole|
---------+------------+-------------+------+--------+--------+------------------+-------------------+-------------------+-----+--------------------+---------+----------------+------------------+-------------+-------------+-------+--------------+------------------+------------------------+------+---------------------------+-----------------------------------------------------------------------------------------------------------------------------+-----------------------+--------+
10201    |xx.xx.xx.1|9050         |9060  |8040    |8060    |-1                |2095-06-26 15:55:25|2095-10-17 15:40:43|true |false               |91646    |63.722 GB       |6.538 MB          |28.531 GB    |98.304 GB    |70.98 %|70.98 %       |0.000             |{"location" : "default"}|      |doris-2.1.3-rc09-2dc65ce356|{"lastSuccessReportTabletsTime":"2095-10-17 15:40:33","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}|0                      |mix     |
10146    |xx.xx.xx.2 |9050         |9060  |8040    |8060    |-1                |2095-06-26 15:51:48|2095-10-17 15:40:43|true |false               |31051    |1.748 GB        |0.000             |1002.211 MB  |39.246 GB    |97.51 %|97.51 %       |0.000             |{"location" : "default"}|      |doris-2.1.3-rc09-2dc65ce356|{"lastSuccessReportTabletsTime":"2095-10-17 15:39:46","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}|0                      |mix     |
10200    |xx.xx.xx.3 |9050         |9060  |8040    |8060    |-1                |2095-06-26 15:57:28|2095-10-17 15:40:43|true |false               |93031    |61.051 GB       |6.784 MB          |31.189 GB    |98.304 GB    |68.27 %|68.27 %       |0.000             |{"location" : "default"}|      |doris-2.1.3-rc09-2dc65ce356|{"lastSuccessReportTabletsTime":"2095-10-17 15:39:58","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}|0                      |mix     |
  • 解决方法

磁盘扩容 or 清理数据（如 日志数据 或 不用的表数据）

【总结】
当DorisBE端磁盘使用超过95%阈值时，系统会报错。可通过`show backends`检查磁盘使用情况，清理trash和snapshot目录，或调整数据和阈值策略来释放或扩大空间。
  • 参考文献

该问题出现在导入数据到Doris数据库中，doris be端存放date位置空间使用不足5%，doris的阈值模式使用为95%。

SHOW TABLE STATUS; -- 查看当前数据库下所有表的信息

SHOW TABLE STATUS FROM db LIKE "%exam%"; -- 查看指定数据库下，名称包含 exam 的表的信息

/**
+----------+--------+---------+------------+-----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------+----------+----------------+---------+
| Name     | Engine | Version | Row_format | Rows      | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time         | Update_time         | Check_time | Collation | Checksum | Create_options | Comment |
+----------+--------+---------+------------+-----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------+----------+----------------+---------+
| exam151 | Doris  |    NULL | NULL       | 200000000 |            349 | 69872320591 |            NULL |         NULL |      NULL |  NULL | 2023-03-06 16:19:18 | 2023-03-14 11:05:38 | NULL       | utf-8     |     NULL | NULL           | 表151   |
| exam152 | Doris  |    NULL | NULL       | 200000000 |            174 | 34935641063 |            NULL |         NULL |      NULL |  NULL | 2023-03-14 11:10:43 | 2023-03-14 11:14:35 | NULL       | utf-8     |     NULL | NULL           | 表152   |
+----------+--------+---------+------------+-----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------+----------+----------------+---------+
2 rows in set (0.01 sec)

Data_length:
    从数值来看该列显示的是整张表占用的空间大小，如果是双备份，则是双备份的大小

**/

Y 推荐文献

X 参考文献

posted @ 2025-10-17 16:20  千千寰宇  阅读(6)  评论(0)    收藏  举报