oracle安装目录满导致数据库down一例

查看oracle按安装目录使用100%，登陆进去开始删除无用的trace文件，以及audit文件。

$ df -k
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4         10485760         0 100%    46115    99% /
/dev/hd2         20971520 16020784   24%    87004     3% /usr
/dev/hd9var      10485760 10466276    1%      594     1% /var
/dev/hd3          8388608   8263424    2%      544     1% /tmp
/dev/hd1         10485760   9233808   12%       33     1% /home
/proc                   -         -    -         -     - /proc
/dev/hd10opt     10485760 10126932    4%     4254     1% /opt
/dev/lv00          262144    253872    4%       18     1% /var/adm/csd
/dev/export_home    20971520 14427768   32%     7093     1% /export/home
/dev/archlv     456130560 449946632    2%       10     1% /export/home/archivelog
$ pwd
/oracle/product/10.2
$ cd rdbms
$ ls
admin       demo        install     label.info lib32       mesg        public
audit       doc         jlib        lib         log         notes       xml
$ cd audit
$ ls -l
total 177320

-rw-r----- 1 oracle oinstall 1939 May 10 02:23 ora_1003520.aud

......

$ rm ora*.aud
ksh: /usr/bin/rm: 0403-027 The parameter list is too long.

http://blog.chinaunix.net/u/12960/showart_83004.html

这个错误的根本原因是因为　/usr/include/limits.h 定义系统核心 LINE_BUFSZ 限制．如果有大量文件数需要删除，可以考虑用 xargs 的 -n 参数进行批量删除.参考如下示例:

# df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4          10.00      0.01 100%    43184    88% /
/dev/hd2          20.00     15.28   24%    87004     3% /usr
/dev/hd9var       10.00      9.98    1%      594     1% /var
/dev/hd3           8.00      7.88    2%      544     1% /tmp
/dev/hd1          10.00      8.81   12%       33     1% /home
/proc                 -         -    -         -     - /proc
/dev/hd10opt      10.00      9.66    4%     4254     1% /opt
/dev/lv00          0.25      0.24    4%       18     1% /var/adm/csd
/dev/export_home     20.00     13.76   32%     7093     1% /export/home
/dev/archlv      435.00    429.10    2%       10     1% /export/home/archivelog

之后，启动数据库还是无法启动:

$ sqlplus /nolog

SQL*Plus: Release 10.2.0.4.0 - Production on Fri May 28 10:18:05 2010

SQL> conn /as sysdba
Connected to an idle instance.
SQL> statup;
SP2-0042: unknown command "statup" - rest of line ignored.
SQL> startup;
ORA-01081: cannot start already-running ORACLE - shut it down first
SQL> shutdown immediate
ORA-24324: service handle not initialized
ORA-24323: value not allowed
ORA-01089: immediate shutdown in progress - no operations are permitted
SQL> startup
ORA-01031: insufficient privileges
SQL> conn /as sysdba
Connected to an idle instance.
SQL> startup
ORA-01081: cannot start already-running ORACLE - shut it down first
SQL> exit
Disconnected
$

$ tail -100 *log

ORACLE Instance NOAP - Archival Error
Fri May 21 21:09:25 2010
ORA-16038: log 2 sequence# 6338 cannot be archived
ORA-00312: online log 2 thread 1: '+DGROUP1/noap/onlinelog/group_2.260.704095729'
ORA-00312: online log 2 thread 1: '+DGROUP1/noap/onlinelog/group_2.261.704095731'
Fri May 21 21:09:25 2010
Errors in file /oracle/admin/NOAP/bdump/noap_arc1_1278076.trc:
ORA-16038: log 2 sequence# 6338 cannot be archived
ORA-00312: online log 2 thread 1: '+DGROUP1/noap/onlinelog/group_2.260.704095729'
ORA-00312: online log 2 thread 1: '+DGROUP1/noap/onlinelog/group_2.261.704095731'
Fri May 21 21:10:01 2010
Archiver process freed from errors. No longer stopped
Fri May 21 21:14:57 2010
Thread 1 advanced to log sequence 6340 (LGWR switch)
Current log# 4 seq# 6340 mem# 0: +DGROUP1/noap/onlinelog/group_4.264.704095741
Current log# 4 seq# 6340 mem# 1: +DGROUP1/noap/onlinelog/group_4.265.704095745
Fri May 21 21:35:01 2010
Thread 1 advanced to log sequenFri May 28 02:14:15 2010
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/export/home/archivelog/1_6731_704095717.dbf' (error 19502)
(NOAP)
Fri May 28 02:14:16 2010
Errors in file /oracle/admin/NOAP/bdump/noap_arc0_1355806.trc:
ORA-19502: write error on file "/export/home/archivelog/1_6731_704095717.dbf", blockno 1077249 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
IBM AIX RISC System/6000 Error: 28: No space left on device
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "/export/home/archivelog/1_6731_704095717.dbf", blockno 1077249 (blocksize=512)
Fri May 28 02:14:16 2010
ARCH: Archival stopped, error occurred. Will continue retrying
Fri May 28 02:14:16 2010
ORACLE Instance NOAP - Archival Error
Fri May 28 02:14:16 2010
ORA-16038: log 5 sequence# 6731 cannot be archived
ORA-19502: write error on file "", blockno (blocksize=)
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.266.704095749'
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.267.704095751'
Fri May 28 02:14:16 2010
Errors in file /oracle/admin/NOAP/bdump/noap_arc0_1355806.trc:
ORA-16038: log 5 sequence# 6731 cannot be archived
ORA-19502: write error on file "", blockno (blocksize=)
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.266.704095749'
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.267.704095751'
ARCH: Archival stopped, error occurred. Will continue retrying
Fri May 28 02:14:16 2010
ORACLE Instance NOAP - Archival Error
Fri May 28 02:14:16 2010
ORA-16014: log 5 sequence# 6731 not archived, no available destinations
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.266.704095749'
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.267.704095751'
Fri May 28 02:14:16 2010
Errors in file /oracle/admin/NOAP/bdump/noap_arc0_1355806.trc:
ORA-16014: log 5 sequence# 6731 not archived, no available destinations
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.266.704095749'
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.267.704095751'
Fri May 28 02:24:21 2010
Errors in file /oracle/admin/NOAP/bdump/noap_arc0_1355806.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '+DGROUP1/noap/onlinelog/group_1.258.704095721'
ORA-17503: ksfdopn:2 Failed to open file +DGROUP1/noap/onlinelog/group_1.258.704095721
ORA-09925: Unable to create audit trail file
IBM AIX RISC System/6000 Error: 28: No space left on device
Additional information: 9925
Fri May 28 02:24:41 2010
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/export/home/archivelog/1_6731_704095717.dbf' (error 19502)
(NOAP)
Fri May 28 02:24:43 2010
Errors in file /oracle/admin/NOAP/bdump/noap_arc0_1355806.trc:
ORA-19502: write error on file "/export/home/archivelog/1_6731_704095717.dbf", blockno 1077249 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
IBM AIX RISC System/6000 Error: 28: No space left on device
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "/export/home/archivelog/1_6731_704095717.dbf", blockno 1077249 (blocksize=512)
Fri May 28 02:24:43 2010
ARCH: Archival stopped, error occurred. Will continue retrying
Fri May 28 02:24:43 2010
ORACLE Instance NOAP - Archival Error
Fri May 28 02:24:43 2010
ORA-16038: log 5 sequence# 6731 cannot be archived
ORA-19502: write error on file "", blockno (blocksize=)
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.266.704095749'
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.267.704095751'
Fri May 28 02:24:43 2010
Errors in file /oracle/admin/NOAP/bdump/noap_arc0_1355806.trc:
ORA-16038: log 5 sequence# 6731 cannot be archived
ORA-19502: write error on file "", blockno (blocksize=)
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.266.704095749'
ORA-00312: online log 5 thread 1: '+DGROUP1/noap/onlinelog/group_5.267.704095751'
Fri May 28 02:24:54 2010
ARCH: Archival stopFri May 28 10:18:31 2010
Starting ORACLE instance (normal)
Fri May 28 10:19:41 2010
Starting ORACLE instance (normal)
Fri May 28 10:22:30 2010
Starting ORACLE instance (normal)
Fri May 28 10:23:43 2010
Starting ORACLE instance (normal)
Fri May 28 10:27:12 2010
Starting ORACLE instance (normal)

查看asm状态：

$ export ORACLE_SID=+ASM
$
$ sqlplus /nolog

SQL*Plus: Release 10.2.0.4.0 - Production on Fri May 28 10:32:45 2010

SQL> conn /as sysdba
Connected.
SQL> show sga

Total System Global Area 130023424 bytes
Fixed Size                  2082312 bytes
Variable Size             102775288 bytes
ASM Cache                  25165824 bytes
SQL> desc v$adm_diskgroups
ERROR:
ORA-04043: object v$adm_diskgroups does not exist

SQL> ;
SP2-0223: No lines in SQL buffer.
SQL> desc v$asm_diskgroup;
Name                                      Null?    Type
----------------------------------------- -------- ----------------------------
GROUP_NUMBER                                       NUMBER
NAME                                               VARCHAR2(30)
SECTOR_SIZE                                        NUMBER
BLOCK_SIZE                                         NUMBER
ALLOCATION_UNIT_SIZE                               NUMBER
STATE                                              VARCHAR2(11)
TYPE                                               VARCHAR2(6)
TOTAL_MB                                           NUMBER
FREE_MB                                            NUMBER
REQUIRED_MIRROR_FREE_MB                            NUMBER
USABLE_FILE_MB                                     NUMBER
OFFLINE_DISKS                                      NUMBER
UNBALANCED                                         VARCHAR2(1)
COMPATIBILITY                                      VARCHAR2(60)
DATABASE_COMPATIBILITY                             VARCHAR2(60)

SQL> select name from v$asm_diskgroup;

NAME
------------------------------
DGROUP1

SQL> select * from v$asm_diskgroup;

GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB
-------------------- ----------- ------ ---------- ----------
REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS U
----------------------- -------------- ------------- -
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY
------------------------------------------------------------
           1 DGROUP1                                512       4096
             1048576 MOUNTED     EXTERN   12339912    9254389
                      0        9254389             0 N

GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE
------------ ------------------------------ ----------- ----------
ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB
-------------------- ----------- ------ ---------- ----------
REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS U
----------------------- -------------- ------------- -
COMPATIBILITY
------------------------------------------------------------
DATABASE_COMPATIBILITY
------------------------------------------------------------
10.1.0.0.0
10.1.0.0.0

SQL> select state from v$asm_diskgroup;

STATE
-----------
MOUNTED

SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
$

重试：

$ export ORACLE_SID=NOAP
$ sqlplus /nolog

SQL*Plus: Release 10.2.0.4.0 - Production on Fri May 28 10:35:28 2010

SQL> conn /as sysdba
Connected to an idle instance.
SQL> shutdown immediate
ORA-24324: service handle not initialized
ORA-24323: value not allowed
ORA-01089: immediate shutdown in progress - no operations are permitted
SQL> conn /as sysdba
Connected to an idle instance.
SQL> startup
ORA-01081: cannot start already-running ORACLE - shut it down first
SQL> conn /as sysdba
Connected to an idle instance.
SQL> alter database mount;
alter database mount
*
ERROR at line 1:
ORA-01012: not logged on

SQL> conn /as sysdba
Connected to an idle instance.
SQL> select * from v$instance;
select * from v$instance
*
ERROR at line 1:
ORA-01012: not logged on

SQL> exit
Disconnected
$

查看oracle进程：

$ ps -ef | grep ora
oracle 131100       1   0   May 10      - 0:34 asm_ckpt_+ASM
    root 295110       1   0   Dec 30      - 24:36 /oracle/product/10.2/bin/ocssd.bin
oracle 303186       1   0   May 10      - 0:22 asm_lgwr_+ASM
oracle 327882       1   0   May 10      - 0:20 asm_dbw0_+ASM
oracle 335956       1   0   May 10      - 0:19 asm_smon_+ASM
oracle 360672       1   0   May 10      - 0:18 asm_mman_+ASM
oracle 381018       1   0   May 10      - 0:16 asm_psp0_+ASM
oracle 393306       1   0   May 10      - 2:04 asm_pmon_+ASM
oracle 450782       1   0   May 10      - 1:21 asm_gmon_+ASM
oracle 503820       1   0   May 13      - 21:28 ora_smon_NOAP
oracle 544776 1577164   0 09:39:15 pts/0 0:02 -ksh
oracle 635026       1   0   Feb 01   vty0 13:08 /oracle/product/10.2/perl/bin/perl /oracle/product/10.2/bin/emwd.pl dbconsole /oracle/product/10.2/GSdb1_NOAP/sysman/log/emdb.nohup
oracle 753892       1   0   May 13      - 0:19 ora_psp0_NOAP
oracle 757946 291050   0   Feb 01   vty0 0:00 -ksh
oracle 831634 635026   0   Feb 01   vty0 210:38 /oracle/product/10.2/bin/emagent
oracle 843980       1   0   May 13      - 9:57 ora_ckpt_NOAP
oracle 860286       1   0   Apr 17      - 180:57 /oracle/product/10.2/bin/tnslsnr LISTENER -inherit
oracle 987364       1   0   May 10      - 0:18 asm_rbal_+ASM
oracle 1007626       1   0   May 13      - 0:00 ora_reco_NOAP
oracle 1024068       1   0   May 13      - 78:27 ora_lgwr_NOAP
oracle 1040572       1   0   May 13      - 7:35 ora_dbw7_NOAP
oracle 1089678       1   0   May 13      - 7:08 ora_dbw6_NOAP
oracle 1093632 635026   0 10:14:17   vty0 0:10 /oracle/product/10.2/jdk/bin/java -Xmx256M -DORACLE_HOME=/oracle/product/10.2 -Doracle.home=/oracle/product/10.2/oc4j -Doracle.oc4j.localhome=/oracle/product/10.2/GSdb1_NOAP/sysman -DEMSTATE=/oracle/product/10.2/GSdb1_NOAP -Doracle.j2ee.dont.use.memory.archive=true -Djava.protocol.handler.pkgs=HTTPClient -Doracle.security.jazn.config=/oracle/product/10.2/oc4j/j2ee/OC4J_DBConsole_GSdb1_NOAP/config/jazn.xml -Djava.security.policy=/oracle/product/10.2/oc4j/j2ee/OC4J_DBConsole_GSdb1_NOAP/config/java2.policy -Djava.security.properties=/oracle/product/10.2/oc4j/j2ee/home/config/jazn.security.props -DEMDROOT=/oracle/product/10.2/GSdb1_NOAP -Dsysman.md5password=true -Drepapi.oracle.home=/oracle/product/10.2 -Ddisable.checkForUpdate=true -Djava.awt.headless=true -jar /oracle/product/10.2/oc4j/j2ee/home/oc4j.jar -config /oracle/product/10.2/oc4j/j2ee/OC4J_DBConsole_GSdb1_NOAP/config/server.xml
oracle 1097742       1   0   May 13      - 4:38 ora_pmon_NOAP
oracle 1110168       1   0   May 13      - 6:22 ora_dbw5_NOAP
oracle 1130524       1   0   May 13      - 6:23 ora_dbw4_NOAP
oracle 1155318       1   0   May 13      - 0:39 oracle+ASM (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 1175752       1   0   May 13      - 6:47 ora_dbw3_NOAP
oracle 1220778       1   0   May 13      - 6:57 ora_dbw2_NOAP
oracle 1233094       1   0   May 13      - 6:51 ora_dbw1_NOAP
oracle 1253534       1   0   May 13      - 6:42 ora_dbw0_NOAP
oracle 1278076       1   0   May 13      - 9:39 ora_arc1_NOAP
oracle 1355806       1   0   May 13      - 9:13 ora_arc0_NOAP
oracle 1429652 544776   0 10:38:50 pts/0 0:00 ps -ef
oracle 1433734       1   0   May 13      - 0:12 ora_rbal_NOAP
oracle 1437898       1   0   May 13      - 0:35 ora_asmb_NOAP
oracle 1482942       1   0   May 13      - 24:48 ora_mman_NOAP
oracle 1593522 544776   0 10:38:50 pts/0 0:00 grep ora
$

发现有Oraxxxnoap进程，杀掉试试吧：

$ kill -9 1097742
$ ps -ef | grep ora
oracle 131100       1   0   May 10      - 0:34 asm_ckpt_+ASM
    root 295110       1   0   Dec 30      - 24:36 /oracle/product/10.2/bin/ocssd.bin
oracle 303186       1   0   May 10      - 0:22 asm_lgwr_+ASM
oracle 327882       1   0   May 10      - 0:20 asm_dbw0_+ASM
oracle 335956       1   0   May 10      - 0:19 asm_smon_+ASM
oracle 360672       1   0   May 10      - 0:18 asm_mman_+ASM
oracle 381018       1   0   May 10      - 0:16 asm_psp0_+ASM
oracle 393306       1   0   May 10      - 2:04 asm_pmon_+ASM
oracle 450782       1   0   May 10      - 1:21 asm_gmon_+ASM
oracle 544776 1577164   0 09:39:15 pts/0 0:02 -ksh
oracle 635026       1   0   Feb 01   vty0 13:08 /oracle/product/10.2/perl/bin/perl /oracle/product/10.2/bin/emwd.pl dbconsole /oracle/product/10.2/GSdb1_NOAP/sysman/log/emdb.nohup
oracle 757946 291050   0   Feb 01   vty0 0:00 -ksh
oracle 831634 635026   0   Feb 01   vty0 210:38 /oracle/product/10.2/bin/emagent
oracle 860286       1   0   Apr 17      - 180:57 /oracle/product/10.2/bin/tnslsnr LISTENER -inherit
oracle 987364       1   0   May 10      - 0:18 asm_rbal_+ASM
oracle 1093632 635026   1 10:14:17   vty0 0:10 /oracle/product/10.2/jdk/bin/java -Xmx256M -DORACLE_HOME=/oracle/product/10.2 -Doracle.home=/oracle/product/10.2/oc4j -Doracle.oc4j.localhome=/oracle/product/10.2/GSdb1_NOAP/sysman -DEMSTATE=/oracle/product/10.2/GSdb1_NOAP -Doracle.j2ee.dont.use.memory.archive=true -Djava.protocol.handler.pkgs=HTTPClient -Doracle.security.jazn.config=/oracle/product/10.2/oc4j/j2ee/OC4J_DBConsole_GSdb1_NOAP/config/jazn.xml -Djava.security.policy=/oracle/product/10.2/oc4j/j2ee/OC4J_DBConsole_GSdb1_NOAP/config/java2.policy -Djava.security.properties=/oracle/product/10.2/oc4j/j2ee/home/config/jazn.security.props -DEMDROOT=/oracle/product/10.2/GSdb1_NOAP -Dsysman.md5password=true -Drepapi.oracle.home=/oracle/product/10.2 -Ddisable.checkForUpdate=true -Djava.awt.headless=true -jar /oracle/product/10.2/oc4j/j2ee/home/oc4j.jar -config /oracle/product/10.2/oc4j/j2ee/OC4J_DBConsole_GSdb1_NOAP/config/server.xml
oracle 1130524       1 120   May 13      - 6:26 ora_dbw4_NOAP
oracle 1155318       1   0   May 13      - 0:39 oracle+ASM (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 1437904 544776   0 10:40:22 pts/0 0:00 grep ora
oracle 1482946 544776   0 10:40:22 pts/0 0:00 ps -ef
$
$

重新登录启动：

$ sqlplus /nolog

SQL*Plus: Release 10.2.0.4.0 - Production on Fri May 28 10:40:37 2010

SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.

Total System Global Area 1.7180E+10 bytes
Fixed Size                  2114248 bytes
Variable Size            7247760696 bytes
Database Buffers         9915334656 bytes
Redo Buffers               14659584 bytes
Database mounted.
Database opened.
SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
NOAP

SQL>

终于ok！

oracle安装目录满导致数据库down一例

导航

公告