转：Red Hat Enterprise Linux systems using the TSC clock source reboots or panics when 'sched_clock()' overflows after an uptime of 208.5 days

虽说是老贴了，但是最近还接触到

 

原文：https://access.redhat.com/site/solutions/68466

 

 

问题

  • Linux Kernel panics when sched_clock() overflows after an uptime of around 208.5 days
  • Red Hat Enterprise Linux system reboots with sched_clock() overflow after an uptime of around 208.5 days
  • This symptom may happen on a system using the Time Stamp Counter (TSC) clock source
  • Some processes may generate following log message:
BUG: soft lockup - CPU#N stuck for 4278190091s!

环境

  • Red Hat Enterprise Linux (RHEL) 6
    • Red Hat Enterprise Linux 6.0 (some kernels)
    • Red Hat Enterprise Linux 6.1 (some kernels)
    • Red Hat Enterprise Linux 6.2 (some kernels)
    • With TSC clock source
  • Red Hat Enterprise Linux (RHEL) 5
    • Red Hat Enterprise Linux 5.0
    • Red Hat Enterprise Linux 5.1
    • Red Hat Enterprise Linux 5.2
    • Red Hat Enterprise Linux 5.3 (some kernels)
    • Red Hat Enterprise Linux 5.4
    • Red Hat Enterprise Linux 5.5
    • Red Hat Enterprise Linux 5.6 (some kernels)
    • Red Hat Enterprise Linux 5.7
    • Red Hat Enterprise Linux 5.8 (some kernels)
    • With TSC clock source
  • Red Hat Enterprise MRG 1.3 Realtime kernel
  • An approximate uptime of around 208.5 days

决议

  • Red Hat Enterprise Linux 6
    • RHEL 6.x
      • Update to kernel-2.6.32-279.el6 or later (RHSA-2012-0862)
      • This kernel is already part of Red Hat Enterprise Linux 6.3 GA
    • RHEL 6.2
    • RHEL 6.1 Extended Update Support
  • Red Hat Enterprise Linux 5 x86_64 (64bit)
    • RHEL 5.x
      • Update to kernel-2.6.18-348.el5 or later (RHBA-2013-0006)
      • Red Hat Enterprise Linux 5.9 GA and later already contain this fix
    • RHEL 5.8.z
    • RHEL 5.6.z
    • RHEL 5.3.z
  • Red Hat Enterprise Linux 5 x86 (32bit)
    • RHEL 5.x
      • Update to kernel-2.6.18-348.el5 or later (RHBA-2013-0006)
      • Red Hat Enterprise Linux 5.9 GA and later already contain this fix
    • RHEL 5.8.z
    • RHEL 5.6.z
    • RHEL 5.3.z
  • Red Hat Enterprise 1.3 Realtime kernel
    • Red Hat Enterprise MRG 1.3 Realtime kernel
      • Update to kernel kernel-rt-2.6.33.9-rt31.86.el5rt or later (RHBA-2013:0927)

根源

  • An insufficiently designed calculation in the CPU accelerator in the previous kernel caused an arithmetic overflow in the sched_clock() function

    • This overflow led to a kernel panic or any other unpredictable trouble on the systems using the TSC clock source
    • This problem will occur only when system uptime reaches or exceeds 208.5 days
    • This update corrects the aforementioned calculation so that this arithmetic overflow and kernel panic can no longer occur under these circumstances

  • On RHEL5, this problem is a timing issue and is very unlikely to be encountered.

  • Switching to another clocksource is usually not a workaround for most workloads

    • The TSC is a fast access clock, whereas the HPET and PMTimer are both slow access clocks
    • Using notsc would be a significant performance hit
    • In RHEL5, the affected sched_clock() uses the TSC regardless of clock source selection.
    • Also, in some situation, the system may hit this issue even if you set notsc to current_clocksource.

诊断步骤

  • Note: this issue could occur in numerous locations that deal with time in the kernel

    • For example, a user running a non-Red Hat kernel could have a kernel panic with a soft lockup in __ticket_spin_lock

  • The system must be booting a kernel that is a version prior to releases mentioned in the "Resolution" field

posted @ 2014-04-29 15:44  4admin2root  阅读(682)  评论(0)    收藏  举报