1次Nexus9508电源模块故障，导致机柜电源跳闸issue
Nexus9508电源模块故障，导致机柜电源跳闸issue
背景
出现的情况是机柜整个PDU掉电了，这个PDU上还有其他网络设备，所以这一路全都掉电。
现场人员在机房尝试着把这个PDU再推上去，但马上又跳闸，
尝试把9508的电源线拔掉，再次推闸上去就可以了，
这个9508之前还出过一回，当时是电模块坏的问题。
这台9508有一共安装了3个电源模块，
目前是其中2个电源模块状态异常，共有8个电源槽位。
------- ------------------- ---------- ---------- ---------- --------------
1 N9K-PAC-3000W-B 0 W 0 W 0 W Shutdown
2 N9K-PAC-3000W-B 0 W 0 W 0 W Shutdown
3 ------------ 0 W 0 W 0 W Absent
4 ------------ 0 W 0 W 0 W Absent
5 ------------ 0 W 0 W 0 W Absent
6 ------------ 0 W 0 W 0 W Absent
7 ------------ 0 W 0 W 0 W Absent
8 N9K-PAC-3000W-B 1024 W 1090 W 3000 W Ok
Nexus 9508面板外观如下：
电源模块长这样
2 检查过程
2.1 Logging
Nexus9508# sh logging | in "2021 Jul 1"
2021 Jul 1 11:05:20 Nexus9508 %PLATFORM-2-PS_CAPACITY_CHANGE: Power supply PS2 changed its capacity. possibly due to On/Off or power cable removal/in
sertion (Serial number FDO12345678)
2021 Jul 1 11:05:21 Nexus9508 %PLATFORM-5-PS_STATUS: PowerSupply 2 current-status is PS_FAIL
2021 Jul 1 11:05:21 Nexus9508 %PLATFORM-2-PS_FAIL: Power supply 2 failed or shut down (Serial number FDO12345678)
2021 Jul 1 11:05:22 Nexus9508 %PLATFORM-2-PS_CAPACITY_CHANGE: Power supply PS2 changed its capacity. possibly due to On/Off or power cable removal/in
sertion (Serial number FDO12345678)
2021 Jul 1 11:05:22 Nexus9508 %PLATFORM-2-PS_ABSENT: Power supply 2 is absent/shutdown, ps-redundancy might be affected
2021 Jul 1 11:06:00 Nexus9508 %PLATFORM-5-PS_STATUS: PowerSupply 1 current-status is PS_FAIL
2021 Jul 1 11:06:00 Nexus9508 %PLATFORM-2-PS_FAIL: Power supply 1 failed or shut down (Serial number FDO88885443)
2021 Jul 1 11:06:05 Nexus9508 %PLATFORM-2-PS_CAPACITY_CHANGE: Power supply PS1 changed its capacity. possibly due to On/Off or power cable removal/in
sertion (Serial number FDO88885443)
2021 Jul 1 11:06:05 Nexus9508 %PLATFORM-2-PS_ABSENT: Power supply 1 is absent/shutdown, ps-redundancy might be affected
Nexus9508#
2.2 Show environment
Nexus9508# show environment
Power Supply:
Voltage: 12 Volts
Power Actual Actual Total
Supply Model Output Input Capacity Status
(Watts ) (Watts ) (Watts )
------- ------------------- ---------- ---------- ---------- --------------
1 N9K-PAC-3000W-B 0 W 0 W 0 W Shutdown
2 N9K-PAC-3000W-B 0 W 0 W 0 W Shutdown
3 ------------ 0 W 0 W 0 W Absent
4 ------------ 0 W 0 W 0 W Absent
5 ------------ 0 W 0 W 0 W Absent
6 ------------ 0 W 0 W 0 W Absent
7 ------------ 0 W 0 W 0 W Absent
8 N9K-PAC-3000W-B 1024 W 1090 W 3000 W Ok
Actual Power
Module Model Draw Allocated Status
(Watts ) (Watts )
------- ------------------- ----------- ----------- --------------
1 N9K-X9464TX2 118.00 W 360.00 W Powered-Up
3 N9K-X9432PQ 188.00 W 300.00 W Powered-Up
5 N9K-X9464PX 100.00 W 240.00 W Powered-Up
Xb21 xbar N/A 0.00 W Absent
Xb22 N9K-C9508-FM 128.00 W 250.80 W Powered-Up
Xb23 N9K-C9508-FM 135.00 W 250.80 W Powered-Up
Xb24 N9K-C9508-FM 126.00 W 250.80 W Powered-Up
Xb25 xbar N/A 0.00 W Absent
Xb26 N9K-C9508-FM 128.00 W 250.80 W Powered-Up
27 N9K-SUP-A 59.00 W 79.92 W Powered-Up
28 supervisor N/A 0.00 W Absent
29 N9K-SC-A 14.00 W 25.20 W Powered-Up
30 N9K-SC-A 14.00 W 25.20 W Powered-Up
fan1 N9K-C9508-FAN 38.00 W 249.00 W Powered-Up
fan2 N9K-C9508-FAN 37.00 W 249.00 W Powered-Up
fan3 N9K-C9508-FAN 38.00 W 249.00 W Powered-Up
N/A - Per module power not available
Power Usage Summary:
--------------------
Power Supply redundancy mode (configured) PS-Redundant
Power Supply redundancy mode (operational) Non-Redundant
Total Power Capacity (based on configured mode) 3000.00 W
Total Grid-A (first half of PS slots) Power Capacity 0.00 W
Total Grid-B (second half of PS slots) Power Capacity 3000.00 W
Total Power of all Inputs (cumulative) 3000.00 W
Total Power Output (actual draw) 1024.00 W
Total Power Input (actual draw) 1090.00 W
Total Power Allocated (budget) 2862.24 W
Total Power Available for additional modules 137.76 W
Fan:
---------------------------------------------------------------------------
Fan Model Hw Direction Status
---------------------------------------------------------------------------
Fan1(sys_fan1) N9K-C9508-FAN 1.0 front-to-back Ok
Fan2(sys_fan2) N9K-C9508-FAN 1.0 front-to-back Ok
Fan3(sys_fan3) N9K-C9508-FAN 1.0 front-to-back Ok
Fan_in_PS1 -- front-to-back Shutdown
Fan_in_PS2 -- front-to-back Shutdown
Fan_in_PS3 -- front-to-back None
Fan_in_PS4 -- front-to-back None
Fan_in_PS5 -- front-to-back None
Fan_in_PS6 -- front-to-back None
Fan_in_PS7 -- front-to-back None
Fan_in_PS8 -- front-to-back Ok
Fan Zone Speed: Zone 1: 0x0
Fan Air Filter : NotSupported
Temperature:
--------------------------------------------------------------------
Module Sensor MajorThresh MinorThres CurTemp Status
(Celsius) (Celsius) (Celsius)
--------------------------------------------------------------------
1 CPU 105 95 52 Ok
1 TD2-1 105 95 57 Ok
1 VRM-3 110 100 45 Ok
3 CPU 105 95 47 Ok
3 TD2-1 105 95 53 Ok
3 TD2-2 105 95 52 Ok
3 VRM-1 110 100 46 Ok
3 VRM-2 110 100 47 Ok
5 CPU 105 95 38 Ok
5 TD2-1 105 95 48 Ok
5 VRM-1 110 100 42 Ok
22 CPU 105 95 49 Ok
22 TD2-1 105 95 46 Ok
22 TD2-2 105 95 41 Ok
22 VRM-1 110 100 50 Ok
22 VRM-2 110 100 41 Ok
23 CPU 105 95 43 Ok
23 TD2-1 105 95 46 Ok
23 TD2-2 105 95 41 Ok
23 VRM-1 110 100 49 Ok
23 VRM-2 110 100 40 Ok
24 CPU 105 95 41 Ok
24 TD2-1 105 95 42 Ok
24 TD2-2 105 95 40 Ok
24 VRM-1 110 100 47 Ok
24 VRM-2 110 100 40 Ok
26 CPU 105 95 45 Ok
26 TD2-1 105 95 44 Ok
26 TD2-2 105 95 41 Ok
26 VRM-1 110 100 47 Ok
26 VRM-2 110 100 39 Ok
27 OUTLET 75 55 33 Ok
27 INLET 60 42 25 Ok
27 CPU 90 80 33 Ok
29 CPU 105 95 52 Ok
30 CPU 105 95 53 Ok
2.3 结论
判定电源模块故障
2.4 处理方法：电源模块替换
电源模块备件到达后，替换后电源正常
电源模块长这样
Nexus9508# show environment power
Power Supply:
Voltage: 12 Volts
Power Actual Actual Total
Supply Model Output Input Capacity Status
(Watts ) (Watts ) (Watts )
------- ------------------- ---------- ---------- ---------- --------------
1 N9K-PAC-3000W-B 351 W 373 W 3000 W Ok
2 N9K-PAC-3000W-B 355 W 383 W 3000 W Ok
3 ------------ 0 W 0 W 0 W Absent
4 ------------ 0 W 0 W 0 W Absent
5 ------------ 0 W 0 W 0 W Absent
6 ------------ 0 W 0 W 0 W Absent
7 ------------ 0 W 0 W 0 W Absent
8 N9K-PAC-3000W-B 344 W 365 W 3000 W Ok
Actual Power
Module Model Draw Allocated Status
(Watts ) (Watts )
------- ------------------- ----------- ----------- --------------
1 N9K-X9464TX2 118.00 W 360.00 W Powered-Up
3 N9K-X9432PQ 186.00 W 300.00 W Powered-Up
5 N9K-X9464PX 99.00 W 240.00 W Powered-Up
Xb21 xbar N/A 0.00 W Absent
Xb22 N9K-C9508-FM 139.00 W 250.80 W Powered-Up
Xb23 N9K-C9508-FM 132.00 W 250.80 W Powered-Up
Xb24 N9K-C9508-FM 135.00 W 250.80 W Powered-Up
Xb25 xbar N/A 0.00 W Absent
Xb26 N9K-C9508-FM 130.00 W 250.80 W Powered-Up
27 N9K-SUP-A 56.00 W 79.92 W Powered-Up
28 supervisor N/A 0.00 W Absent
29 N9K-SC-A 15.00 W 25.20 W Powered-Up
30 N9K-SC-A 14.00 W 25.20 W Powered-Up
fan1 N9K-C9508-FAN 39.00 W 249.00 W Powered-Up
fan2 N9K-C9508-FAN 39.00 W 249.00 W Powered-Up
fan3 N9K-C9508-FAN 38.00 W 249.00 W Powered-Up
N/A - Per module power not available
Power Usage Summary:
--------------------
Power Supply redundancy mode (configured) PS-Redundant
Power Supply redundancy mode (operational) PS-Redundant
Total Power Capacity (based on configured mode) 6000.00 W
Total Grid-A (first half of PS slots) Power Capacity 6000.00 W
Total Grid-B (second half of PS slots) Power Capacity 3000.00 W
Total Power of all Inputs (cumulative) 9000.00 W
Total Power Output (actual draw) 1052.00 W
Total Power Input (actual draw) 1122.00 W
Total Power Allocated (budget) 2862.24 W
Total Power Available for additional modules 3137.76 W
Nexus9508#
浙公网安备 33010602011771号