家里的那台LINUX AS4.0已经有大概1个多月没关机了,今天回家
吃完饭准备做几个experiment,大概在20:18分的样子机器掉死了，
还记得最后一个查询是select * from temp.卡死在那里，出现
硬盘狂闪，后来客户端报错，通道文件结束，键盘啥都没反应，
TELNET也中断了，不得不冷启动机器，机器启动大概花了4分钟，
包括后台FSCK等等，非法启动需要同步FS,INODE节点，以确保文件
系统于Inode一致，所以比平常要慢点，于是开始找问题发生的根源，
一开始就查看syslogd的日志，发现没什么可用信息，都是启动的日志，
后来只能查看ORACLE debug ，core dump 没有 ，但是数据库日志有
一个奇怪的东西(如下)
WARNING: EINVAL creating segment of size 0x0000000010400000
fix shm parameters in /etc/system or equivalent
这个也不是很明显，于是开始怀疑是Operating System 信号量参数有
问题，查看了一下内核参数，修改了一下，启动数据库日志正常了。
kernel.shmmax = 200000000
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 250 32000 100 128(这个参数貌似有问题)
OS内核版本是：
Linux oradb02 2.6.9-34.EL #1 Fri Feb 24 16:44:51
EST 2006 i686 i686 i386 GNU/Linux
ORACLE版本
Oracle9i Enterprise Edition Release 9.2.0.4.0 - Production
PL/SQL Release 9.2.0.4.0 - Production
CORE 9.2.0.3.0 Production
TNS for Linux: Version 9.2.0.4.0 - Production
NLSRTL Version 9.2.0.4.0 - Production
但是为了我的数据，还是决定年底割接完C网以后，升级一下我的内核，
DB也打一下OPATCH吧。
挂了之前的日志：
WARNING: EINVAL creating segment of size 0x0000000010400000
fix shm parameters in /etc/system or equivalent
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 2
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.4.0.
System parameters with non-default values:
processes = 150
timed_statistics = TRUE
shared_pool_size = 50331648
large_pool_size = 33554432
java_pool_size = 33554432
control_files = /opt/ora9/oradata/oradb02/control01.ctl, /opt/ora9/oradata/oradb02/control02.ctl, /opt/ora9/oradata/oradb02/control03.ctl
db_block_size = 8192
db_cache_size = 117440512
compatible = 9.2.0.0.0
db_file_multiblock_read_count= 64
fast_start_mttr_target = 300
undo_management = AUTO
undo_tablespace = UNDOTBS1
undo_retention = 10800
remote_login_passwordfile= EXCLUSIVE
db_domain =
instance_name = oradb02
dispatchers = (PROTOCOL=TCP) (SERVICE=oradb02XDB)
job_queue_processes = 10
hash_join_enabled = TRUE
hash_area_size = 1048576
background_dump_dest = /opt/ora9/admin/oradb02/bdump
user_dump_dest = /opt/ora9/admin/oradb02/udump
core_dump_dest = /opt/ora9/admin/oradb02/cdump
sort_area_size = 2048576
db_name = oradb02
open_cursors = 300
star_transformation_enabled= TRUE
query_rewrite_enabled = TRUE
pga_aggregate_target = 47185920
aq_tm_processes = 1
PMON started with pid=2
DBW0 started with pid=3
LGWR started with pid=4
CKPT started with pid=5
SMON started with pid=6
RECO started with pid=7
Wed Dec 3 20:35:27 2008
starting up 1 shared server(s) ...
QMN0 started with pid=9
Wed Dec 3 20:35:27 2008
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
CJQ0 started with pid=8
Wed Dec 3 20:35:28 2008
ALTER DATABASE MOUNT
Wed Dec 3 20:35:32 2008
Successful mount of redo thread 1, with mount id 1457397904.
Wed Dec 3 20:35:32 2008
Database mounted in Exclusive Mode.
Completed: ALTER DATABASE MOUNT
Wed Dec 3 20:35:32 2008
ALTER DATABASE OPEN
Wed Dec 3 20:35:32 2008
Beginning crash recovery of 1 threads
Wed Dec 3 20:35:32 2008
Started first pass scan
Wed Dec 3 20:35:33 2008
Completed first pass scan
3691 redo blocks read, 68 data blocks need recovery
Wed Dec 3 20:35:33 2008
Started recovery at
Thread 1: logseq 203, block 70152, scn 0.0
Recovery of Online Redo Log: Thread 1 Group 1 Seq 203 Reading mem 0
Mem# 0 errs 0: /opt/ora9/oradata/oradb02/redo01.log
Wed Dec 3 20:35:34 2008
Completed redo application
Wed Dec 3 20:35:34 2008
Ended recovery at
Thread 1: logseq 203, block 73843, scn 0.7374774
68 data blocks read, 68 data blocks written, 3691 redo blocks read
Crash recovery completed successfully
Wed Dec 3 20:35:34 2008
Thread 1 advanced to log sequence 204
Thread 1 opened at log sequence 204
Current log# 2 seq# 204 mem# 0: /opt/ora9/oradata/oradb02/redo02.log
Successful open of redo thread 1.
Wed Dec 3 20:35:34 2008
SMON: enabling cache recovery
Wed Dec 3 20:35:35 2008
Undo Segment 1 Onlined
Undo Segment 2 Onlined
Undo Segment 3 Onlined
Undo Segment 4 Onlined
Undo Segment 5 Onlined
Undo Segment 6 Onlined
Undo Segment 7 Onlined
Undo Segment 8 Onlined
Undo Segment 9 Onlined
Undo Segment 10 Onlined
Successfully onlined Undo Tablespace 1.
Wed Dec 3 20:35:35 2008
SMON: enabling tx recovery
Wed Dec 3 20:35:35 2008
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: ALTER DATABASE OPEN
Dec 3 20:17:13 oradb02 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 3 20:17:13 oradb02 kernel: security: 3 users, 4 roles, 354 types, 26 bools
Dec 3 20:17:13 oradb02 kernel: security: 55 classes, 21833 rules
Dec 3 20:17:13 oradb02 kernel: SELinux: Completing initialization.
Dec 3 20:17:13 oradb02 kernel: SELinux: Setting up existing superblocks.
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev dm-0, type ext3), uses xattr
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Dec 3 20:17:13 oradb02 rpcidmapd: rpc.idmapd startup succeeded
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev mqueue, type mqueue), not configured for labeling
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses genfs_contexts
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev devpts, type devpts), uses transition SIDs
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev eventpollfs, type eventpollfs), uses genfs_contexts
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev futexfs, type futexfs), uses genfs_contexts
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev proc, type proc), uses genfs_contexts
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
Dec 3 20:17:13 oradb02 kernel: SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
Dec 3 20:17:13 oradb02 kernel: inserting floppy driver for 2.6.9-34.EL
Dec 3 20:17:13 oradb02 kernel: FDC 0 is a post-1991 82077
Dec 3 20:17:13 oradb02 kernel: 8139too Fast Ethernet driver 0.9.27
Dec 3 20:17:13 oradb02 kernel: PCI: Found IRQ 5 for device 0000:00:09.0
Dec 3 20:17:13 oradb02 kernel: eth0: RealTek RTL8139 at 0xe400, 00:e0:4c:e7:26:49, IRQ 5
Dec 3 20:17:14 oradb02 kernel: pciehp: acpi_pciehprm:get_device PCI ROOT HID fail=0x1001
Dec 3 20:17:14 oradb02 kernel: USB Universal Host Controller Interface driver v2.2
Dec 3 20:17:14 oradb02 kernel: uhci_hcd 0000:00:07.2: UHCI Host Controller
Dec 3 20:17:14 oradb02 kernel: uhci_hcd 0000:00:07.2: irq 10, io base 0000e000
Dec 3 20:17:14 oradb02 kernel: SELinux: initialized (dev usbdevfs, type usbdevfs), uses genfs_contexts
Dec 3 20:17:14 oradb02 kernel: uhci_hcd 0000:00:07.2: new USB bus registered, assigned bus number 1
Dec 3 20:17:14 oradb02 kernel: hub 1-0:1.0: USB hub found
Dec 3 20:17:14 oradb02 kernel: hub 1-0:1.0: 2 ports detected
Dec 3 20:17:14 oradb02 kernel: md: Autodetecting RAID arrays.
Dec 3 20:17:14 oradb02 kernel: md: autorun ...
Dec 3 20:17:14 oradb02 kernel: md: ... autorun DONE.
Dec 3 20:17:14 oradb02 kernel: ip_tables: (C) 2000-2002 Netfilter core team
Dec 3 20:17:14 oradb02 kernel: eth0: link up, 100Mbps, full-duplex, lpa 0x45E1
Dec 3 20:17:14 oradb02 kernel: usb 1-2: new full speed USB device using address 2
Dec 3 20:17:14 oradb02 kernel: hub 1-2:1.0: USB hub found
Dec 3 20:17:14 oradb02 kernel: hub 1-2:1.0: 4 ports detected
Dec 3 20:17:14 oradb02 kernel: EXT3 FS on dm-0, internal journal
Dec 3 20:17:14 oradb02 kernel: kjournald starting. Commit interval 5 seconds
Dec 3 20:17:14 oradb02 kernel: EXT3 FS on hda1, internal journal
Dec 3 20:17:14 oradb02 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 3 20:17:14 oradb02 kernel: SELinux: initialized (dev hda1, type ext3), uses xattr
Dec 3 20:17:14 oradb02 kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Dec 3 20:17:14 oradb02 kernel: Adding 786424k swap on /dev/VolGroup00/LogVol01. Priority:-1 extents:1
Dec 3 20:17:14 oradb02 kernel: SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
Dec 3 20:17:14 oradb02 kernel: IA-32 Microcode Update Driver: v1.14 <tigran@veritas.com>
Dec 3 20:17:14 oradb02 kernel: microcode: CPU0 already at revision 0x8 (current=0x8)
Dec 3 20:17:14 oradb02 kernel: microcode: No new microdata for cpu 0
Dec 3 20:17:14 oradb02 kernel: IA-32 Microcode Update Driver v1.14 unregistered
Dec 3 20:17:14 oradb02 kernel: parport0: PC-style at 0x378 [PCSPP,TRISTATE]
Dec 3 20:17:14 oradb02 kernel: ip_tables: (C) 2000-2002 Netfilter core team
Dec 3 20:17:14 oradb02 kernel: ip_tables: (C) 2000-2002 Netfilter core team
Dec 3 20:17:14 oradb02 kernel: SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
Dec 3 20:17:14 oradb02 netfs: Mounting other filesystems: succeeded
Dec 3 20:17:14 oradb02 apmd[1844]: Version 3.0.2 (APM BIOS 1.2, Linux driver 1.16ac)
Dec 3 20:17:14 oradb02 apmd: apmd startup succeeded
Dec 3 20:17:14 oradb02 kernel: i2c /dev entries driver
Dec 3 20:17:14 oradb02 rc: Starting lm_sensors: succeeded
Dec 3 20:17:14 oradb02 autofs: automount startup succeeded
Dec 4 04:16:35 oradb02 rc.sysinit: -e
Dec 4 04:16:39 oradb02 start_udev: Starting udev: succeeded
Dec 4 04:16:42 oradb02 udevsend[982]: starting udevd daemon
Dec 4 04:16:43 oradb02 rc.sysinit: -e
Dec 4 04:16:47 oradb02 sysctl: net.ipv4.ip_forward = 0
Dec 4 04:16:47 oradb02 sysctl: net.ipv4.conf.default.rp_filter = 1
Dec 4 04:16:47 oradb02 sysctl: net.ipv4.conf.default.accept_source_route = 0
Dec 4 04:16:47 oradb02 sysctl: kernel.sysrq = 0
Dec 4 04:16:47 oradb02 sysctl: kernel.shmmax = 200000000
Dec 4 04:16:47 oradb02 sysctl: kernel.shmmni = 4096
Dec 4 04:16:47 oradb02 sysctl: kernel.shmall = 2097152
Dec 4 04:16:47 oradb02 sysctl: kernel.sem = 250 32000 100 128
Dec 4 04:16:47 oradb02 sysctl: fs.file-max = 65536
Dec 4 04:16:47 oradb02 sysctl: net.ipv4.ip_local_port_range = 1024 65000
Dec 4 04:16:47 oradb02 rc.sysinit: Configuring kernel parameters: succeeded
Dec 3 20:16:47 oradb02 date: 三 12月 3 20:16:47 CST 2008
Dec 3 20:16:47 oradb02 rc.sysinit: Setting clock (localtime): 三 12月 3 20:16:47 CST 2008 succeeded
Dec 3 20:16:47 oradb02 rc.sysinit: Loading default keymap succeeded
Dec 3 20:16:47 oradb02 rc.sysinit: Setting hostname oradb02: succeeded
Dec 3 20:16:52 oradb02 fsck: /dev/VolGroup00/LogVol00: clean, 278397/4767744 files, 8349152/9535488 blocks
Dec 3 20:16:52 oradb02 rc.sysinit: Checking root filesystem succeeded
Dec 3 20:16:52 oradb02 rc.sysinit: Remounting root filesystem in read-write mode: succeeded
Dec 3 20:16:53 oradb02 lvm.static: 2 logical volume(s) in volume group VolGroup00 now active
Dec 3 20:16:53 oradb02 rc.sysinit: Setting up Logical Volume Management: succeeded
Dec 3 20:16:53 oradb02 fsck: /boot: recovering journal
Dec 3 20:16:53 oradb02 fsck: /boot: clean, 34/26104 files, 12424/104388 blocks
Dec 3 20:16:53 oradb02 rc.sysinit: Checking filesystems succeeded
Dec 3 20:16:53 oradb02 rc.sysinit: Mounting local filesystems: succeeded
Dec 3 20:16:53 oradb02 rc.sysinit: Enabling local filesystem quotas: succeeded
Dec 3 20:16:54 oradb02 rc.sysinit: Enabling swap space: succeeded
Dec 3 20:17:14 oradb02 smartd[1892]: smartd version 5.33 [i386-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Dec 3 20:16:55 oradb02 init: Entering runlevel: 3
Dec 3 20:17:14 oradb02 smartd[1892]: Home page is http://smartmontools.sourceforge.net/
Dec 3 20:16:56 oradb02 microcode_ctl: microcode_ctl startup succeeded
Dec 3 20:17:14 oradb02 smartd[1892]: Opened configuration file /etc/smartd.conf
Dec 3 20:17:07 oradb02 kudzu: succeeded
Dec 3 20:17:15 oradb02 smartd[1892]: Configuration file /etc/smartd.conf parsed.
Dec 3 20:17:07 oradb02 rc: Starting pcmcia: succeeded
Dec 3 20:17:15 oradb02 smartd[1892]: Device: /dev/hda, opened
Dec 3 20:17:07 oradb02 sysctl: net.ipv4.ip_forward = 0
Dec 3 20:17:07 oradb02 sysctl: net.ipv4.conf.default.rp_filter = 1
Dec 3 20:17:07 oradb02 sysctl: net.ipv4.conf.default.accept_source_route = 0
Dec 3 20:17:07 oradb02 sysctl: kernel.sysrq = 0
Dec 3 20:17:07 oradb02 sysctl: kernel.shmmax = 200000000
Dec 3 20:17:07 oradb02 sysctl: kernel.shmmni = 4096
Dec 3 20:17:15 oradb02 smartd[1892]: Device: /dev/hda, found in smartd database.
Dec 3 20:17:07 oradb02 sysctl: kernel.shmall = 2097152
Dec 3 20:17:07 oradb02 sysctl: kernel.sem = 250 32000 100 128
Dec 3 20:17:15 oradb02 apmd[1844]: Charge: * * * (-1% unknown)
Dec 3 20:17:15 oradb02 smartd[1892]: Device: /dev/hda, is SMART capable. Adding to "monitor" list.
Dec 3 20:17:07 oradb02 sysctl: fs.file-max = 65536
Dec 3 20:17:15 oradb02 smartd[1892]: Monitoring 1 ATA and 0 SCSI devices
Dec 3 20:17:07 oradb02 sysctl: net.ipv4.ip_local_port_range = 1024 65000
Dec 3 20:17:07 oradb02 network: Setting network parameters: succeeded
Dec 3 20:17:15 oradb02 smartd[1894]: smartd has fork()ed into background mode. New PID=1894.
Dec 3 20:17:15 oradb02 smartd: smartd startup succeeded
Dec 3 20:17:07 oradb02 network: Bringing up loopback interface: succeeded
Dec 3 20:17:12 oradb02 network: Bringing up interface eth0: succeeded
Dec 3 20:17:17 oradb02 kernel: parport0: PC-style at 0x378 [PCSPP,TRISTATE]
Dec 3 20:17:17 oradb02 kernel: lp0: using parport0 (polling).
Dec 3 20:17:17 oradb02 kernel: lp0: console ready
Dec 3 20:17:18 oradb02 cups: cupsd startup succeeded
Dec 3 20:17:18 oradb02 kernel: NET: Registered protocol family 10
Dec 3 20:17:18 oradb02 kernel: Disabled Privacy Extensions on device c0378f60(lo)
Dec 3 20:17:18 oradb02 kernel: IPv6 over IPv4 tunneling driver
Dec 3 20:17:18 oradb02 sshd: succeeded
Dec 3 20:17:18 oradb02 xinetd: xinetd startup succeeded
Dec 3 20:17:19 oradb02 xinetd[1991]: xinetd Version 2.3.13 started with libwrap loadavg options compiled in.
Dec 3 20:17:19 oradb02 xinetd[1991]: Started working: 1 available service
Dec 3 20:17:19 oradb02 sendmail[2010]: sql_select option missing
Dec 3 20:17:19 oradb02 sendmail[2010]: auxpropfunc error no mechanism available
Dec 3 20:17:20 oradb02 sendmail: sendmail startup succeeded
Dec 3 20:17:20 oradb02 sendmail: sm-client startup succeeded
Dec 3 20:17:20 oradb02 gpm[2030]: *** info [startup.c(95)]:
Dec 3 20:17:20 oradb02 gpm[2030]: Started gpm successfully. Entered daemon mode.
Dec 3 20:17:20 oradb02 gpm[2030]: *** info [mice.c(1766)]:
Dec 3 20:17:20 oradb02 gpm[2030]: imps2: Auto-detected intellimouse PS/2
Dec 3 20:17:20 oradb02 gpm: gpm startup succeeded
Dec 3 20:17:22 oradb02 iiim: htt startup succeeded
Dec 3 20:17:22 oradb02 crond: crond startup succeeded
Dec 3 20:17:23 oradb02 htt_server[2077]: started.
Dec 3 20:17:25 oradb02 xfs: xfs startup succeeded
Dec 3 20:17:26 oradb02 smb: smbd startup succeeded
Dec 3 20:17:26 oradb02 smb: nmbd startup succeeded
Dec 3 20:17:26 oradb02 anacron: anacron startup succeeded
Dec 3 20:17:26 oradb02 atd: atd startup succeeded
Dec 3 20:17:26 oradb02 messagebus: messagebus startup succeeded
Dec 3 20:17:27 oradb02 cups-config-daemon: cups-config-daemon startup succeeded
Dec 3 20:17:27 oradb02 haldaemon: haldaemon startup succeeded
Dec 3 20:17:30 oradb02 fstab-sync[2517]: removed all generated mount points
Dec 3 20:17:33 oradb02 login(pam_unix)[2190]: session opened for user root by LOGIN(uid=0)
Dec 3 20:17:33 oradb02 -- root[2190]: ROOT LOGIN ON tty1
Dec 3 20:18:17 oradb02 remote(pam_unix)[2647]: session opened for user oracle by (uid=0)
Dec 3 20:18:17 oradb02 -- oracle[2647]: LOGIN ON pts/0 BY oracle FROM 192.168.1.100
Dec 3 20:19:40 oradb02 su(pam_unix)[2698]: authentication failure; logname=oracle uid=501 euid=0 tty=pts/0 ruser=oracle rhost= user=root
Dec 3 20:19:45 oradb02 su(pam_unix)[2699]: session opened for user root by oracle(uid=501)
Dec 3 20:23:16 oradb02 nmbd[2127]: [2008/12/03 20:23:16, 0] nmbd/nmbd_become_lmb.c:become_local_master_stage2(396)
Dec 3 20:23:16 oradb02 nmbd[2127]: *****
Dec 3 20:23:16 oradb02 nmbd[2127]:
Dec 3 20:23:16 oradb02 nmbd[2127]: Samba name server ORADB02 is now a local master browser for workgroup MYGROUP on subnet 192.168.1.2
Dec 3 20:23:16 oradb02 nmbd[2127]:
Dec 3 20:23:16 oradb02 nmbd[2127]: *****
Dec 3 20:34:34 oradb02 su(pam_unix)[2856]: session opened for user oracle by oracle(uid=0)
Dec 3 20:34:59 oradb02 remote(pam_unix)[2903]: session opened for user oracle by (uid=0)
Dec 3 20:34:59 oradb02 -- oracle[2903]: LOGIN ON pts/1 BY oracle FROM 192.168.1.100
Dec 3 20:41:39 oradb02 su(pam_unix)[2856]: session closed for user oracle
Dec 3 20:41:40 oradb02 su(pam_unix)[2699]: session closed for user root
Dec 3 20:41:41 oradb02 remote(pam_unix)[2647]: session closed for user oracle
Dec 3 20:41:51 oradb02 remote(pam_unix)[3001]: bad username []
Dec 3 20:41:51 oradb02 login[3001]: FAILED LOGIN 1 FROM 192.168.1.100 FOR , Authentication failure
Dec 3 20:42:02 oradb02 remote(pam_unix)[3001]: session opened for user oracle by (uid=0)
Dec 3 20:42:02 oradb02 -- oracle[3001]: LOGIN ON pts/0 BY oracle FROM 192.168.1.100
Dec 3 20:42:06 oradb02 su(pam_unix)[3043]: authentication failure; logname=oracle uid=501 euid=0 tty=pts/0 ruser=oracle rhost= user=root
Dec 3 20:42:13 oradb02 su(pam_unix)[3044]: session opened for user root by oracle(uid=501)
Dec 3 21:01:01 oradb02 crond(pam_unix)[3100]: session opened for user root by (uid=0)
Dec 3 21:01:01 oradb02 crond(pam_unix)[3100]: session closed for user root
12月7日
丫丫的，晚上真TMD倒霉，那台AS4 怎么又挂了，启动日志又没报错，郁闷，拆掉了一根ECC，现在2根ECC运行，不知道是不是那根
条子的问题，先让他跑几天试试。郁闷。。。。。
