one troubleshooting case about em access issue
Today when trying to open my Oracle EM url, I can not open it. So I thought may be the network is having issue. I use the tenlet to try to identifiy if the network is ok. The command is telnet racnode1 1158 and it shows the network is ok. So I go to the the rac server to check.
First I use emctl status dbconsole. The output is wired because it stuck here
[oracle@racnode1 ~]$ emctl status dbconsole Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. https://racnode1:1158/em/console/aboutApplication
after I issue ctrl+c to stop the command. The who output came out.
^COracle Enterprise Manager 11g is running. ------------------------------------------------------------------ Logs are generated in directory /u01/app/oracle/product/11.2.0/dbhome_1/racnode1_orcl/sysman/log
As it said the dbconsole is running. So I think I might need to restart it first.
[oracle@racnode1 log]$ emctl stop dbconsole Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. https://racnode1:1158/em/console/aboutApplication Stopping Oracle Enterprise Manager 11g Database Control ... ... Stopped.
[oracle@racnode1 log]$ emctl start dbconsole Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. https://racnode1:1158/em/console/aboutApplication Starting Oracle Enterprise Manager 11g Database Control ..... .....^C started. ------------------------------------------------------------------
As you can see, the stop command workds fine but start command stuck.
I try to read the logs to find out why. So I go to the log directory
/u01/app/oracle/product/11.2.0/dbhome_1/racnode1_orcl/sysman/log
There are logs below
[oracle@racnode1 log]$ ls -ltr total 15008 -rw-r-----. 1 oracle oinstall 0 Sep 12 11:54 emdctl.log -rw-r-----. 1 oracle oinstall 0 Sep 12 11:55 patchAgtStPlugin.log -rw-r-----. 1 oracle oinstall 0 Sep 12 11:55 nfsPatchPlug.log -rw-r-----. 1 oracle oinstall 0 Sep 12 11:56 emagentfetchlet.log -rw-r-----. 1 oracle oinstall 0 Sep 12 11:56 emagentfetchlet.trc drwxr-----. 2 oracle oinstall 4096 Sep 12 11:57 pafLogs -rw-r-----. 1 oracle oinstall 10662 Sep 12 11:58 secure.log -rw-r-----. 1 oracle oinstall 5000093 Sep 13 14:29 emoms.trc.1 -rw-r-----. 1 oracle oinstall 364 Sep 21 22:09 emctl.msg -rw-r-----. 1 oracle oinstall 358202 Sep 23 13:53 emdctl.trc -rw-r-----. 1 oracle oinstall 241587 Sep 23 13:53 emdb.nohup -rw-r-----. 1 oracle oinstall 346675 Sep 23 13:59 emagent_perl.trc -rw-r-----. 1 oracle oinstall 14309 Sep 23 13:59 emagent.log -rw-r-----. 1 oracle oinstall 324490 Sep 23 14:03 emoms.trc -rw-r-----. 1 oracle oinstall 5324583 Sep 23 14:03 emoms.log -rw-r-----. 1 oracle oinstall 3700315 Sep 23 14:03 emagent.trc
I tail -100f emdb.nohup found below information.
----- Mon Sep 23 13:41:27 2013::Checking status of DBConsole : 21820 ----- ----- Mon Sep 23 13:41:27 2013::DBConsole exited at Mon Sep 23 13:41:27 2013 with return value 7. ----- ----- Mon Sep 23 13:41:27 2013::Restarting DBConsole. ----- ----- Mon Sep 23 13:41:27 2013::Console Launched with PID 22258 at time Mon Sep 23 13:41:27 2013 ----- ----- Mon Sep 23 13:41:27 2013::osname is linux ----- ----- Mon Sep 23 13:41:27 2013::heap size used is -1 ----- ----- Mon Sep 23 13:41:27 2013::perm size used is -1 ----- ----- Mon Sep 23 13:41:27 2013::joptions used are -server -Xmx192M -XX:MaxPermSize=200M -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 ----- 13/09/23 13:41:33 Error starting ORMI server racnode1 port 5,520: Address already in use 13/09/23 13:41:33 Shutting down OC4J...
It said the 5,520 port are being used. So I try to use netstat to find out what is the process using this port.
[oracle@racnode1 log]$ netstat -ap | grep 5520 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 *:5520 *:* LISTEN 22545/java udp 0 0 169.254.166.63:55203 *:* 7842/asm_psp0_+ASM1 [oracle@racnode1 log]$ ps -ef | grep 22545 oracle 22545 1 0 Sep21 ? 00:03:15 /u01/app/oracle/product/11.2.0/dbhome_1/jdk/bin/java -server -Xmx192M -XX:MaxPermSize=200M -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -DORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 -Doracle.home=/u01/app/oracle/product/11.2.0/dbhome_1/oc4j -Doracle.oc4j.localhome=/u01/app/oracle/product/11.2.0/dbhome_1/racnode1_orcl/sysman -DEMSTATE=/u01/app/oracle/product/11.2.0/dbhome_1/racnode1_orcl -Doracle.j2ee.dont.use.memory.archive=true -Djava.protocol.handler.pkgs=HTTPClient -Doracle.security.jazn.config=/u01/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/OC4J_DBConsole_racnode1_orcl/config/jazn.xml -Djava.security.policy=/u01/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/OC4J_DBConsole_racnode1_orcl/config/java2.policy -Djavax.net.ssl.KeyStore=/u01/app/oracle/product/11.2.0/dbhome_1/sysman/config/OCMTrustedCerts.txt-Djava.security.properties=/u01/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/home/config/jazn.security.props -DEMDROOT=/u01/app/oracle/product/11.2.0/dbhome_1/racnode1_orcl -Dsysman.md5password=true -Drepapi.oracle.home=/u01/app/oracle/product/11.2.0/dbhome_1 -Ddisable.checkForUpdate=true -Doracle. sysman.ccr.ocmSDK.websvc.keystore=/u01/app/oracle/product/11.2.0/dbhome_1/jlib/emocmclnt.ks -Dice.pilots.html4.ignoreNonGenericFonts=true -Djava.awt.headless=true -jar /u01/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/home/oc4j.jar -config /u01/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/OC4J_DBConsole_racnode1_orcl/config/server.xml
There do have one process using this port. One thing you need to be careful about is that if you are not root. The netstat may not return you all the result. Ok I checked the process found it is an OC4J process. And the parent process id is 1. Means this process should have be exited. I killed it. Try to start em again.
[oracle@racnode1 log]$ emctl start dbconsole Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. https://racnode1:1158/em/console/aboutApplication Starting Oracle Enterprise Manager 11g Database Control ............ started. ------------------------------------------------------------------ Logs are generated in directory /u01/app/oracle/product/11.2.0/dbhome_1/racnode1_orcl/sysman/log
Very smoothly. I started it without any problem.
