Archive for April 2011
crashing jrockit while starting WebLogic
Just today I spent some time working on a JVM (jrockit) while starting WebLogic with the default startWebLogic.sh script. The enviroment is rather straight forward:
- Jrockit (jrockit-jdk1.6.0_22-R28.1.1-4.0.1)
- WebLogic 10.3.4
- SOA Suite PS3 (11.1.1.4)
Now whenever I started WebLogic I ran into a rather lengthy core dump, which you can view: here (just rename to .txt). The interesting part of this core dump is this:
Read the rest of this entry »
OHS 11 and configuration errors
While trying to work out some rewrite rules for UCM on OHS (Apache) I noticed a new feature that I didn’t notice before. Back in the good old OAS days (when the air was clean and sex was dirty) I was used to check my apache configuration like this:
apachectl configtest
This would prevent Apache from crashing upon restart from opmn. When moving to 11 (and WebLogic) I stopped doing this (could get the configtest to work), which ment I have to doublecheck my configurations before setting them online.
Anyway, constructing rewrite rules is difficult (to me), chances are that you make a mistake while constructing the rule. So when I made a configuration error I noticed this:
$ /u01/app/oracle/admin/instance1/bin/opmnctl restartproc process-type=OHS
opmnctl restartproc: restarting opmn managed processes...
================================================================================
opmn id=testbox.area51.local:6701
0 of 1 processes restarted.
ias-instance id=instance1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
——————————————————————————–
ias-component/process-type/process-set:
ohs1/OHS/OHS/
Error
–> Process (index=1,uid=893024996,pid=14351)
Process restart failed.
Process is still alive — check configuration.
Log:
/u01/app/oracle/admin/instance1/diagnostics/logs/OHS/ohs1/console~OHS~1.log
Now that’s interesting. Oracle added a new feature: check the configuration before restarting Apache. The “Process is still alive — check configuration.” is new to me. Didn’t read this is the docs, still nice to know.
Hope this helps.
OID11 on Windows 2008R2 abnormality
One of the organizations I work for is running their middleware on Windows (on VMWare), to be precize on Windows 2008 R2. Last week service pack 1 was applied on their boxes and somehow OID died on me. When I checked opmn I noticed that oid was really down:
opmnctl status
Processes in Instance: asinst_1
--------------+--------------+------+---------
ias-component | process-type | pid | status
--------------+--------------+------+---------
oid1 | oidldapd | N/A | Down
oid1 | oidldapd | N/A | Down
oid1 | oidmon | N/A | Down
EMAGENT | EMAGENT | 3868 | Alive
Now when I called opmn to start all processes, I didn’t see an error, but somehow OID still didn’t start:
opmnctl startall
opmnctl startall: starting opmn and all managed processes...
opmnctl status
Processes in Instance: asinst_1
--------------+--------------+------+---------
ias-component | process-type | pid | status
--------------+--------------+------+---------
oid1 | oidldapd | N/A | Down
oid1 | oidldapd | N/A | Down
oid1 | oidmon | N/A | Down
EMAGENT | EMAGENT | 3868 | Alive
When I checked the logfiles I noticed that the .oidmonstdout had a different filesize from normal. The contents was this:
ORA-24550: signal received: Unhandled exception: Code=c0000005 Flags=0
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
kpeDbgCrash()+83 CALL??? kpedbg_dmp_stack()+ 0774AB328 00168DD86 000000000
0 005CFD9D0
kpeDbgSignalHandler CALL??? kpeDbgCrash()+0 00168E026 000000000 005CFD9D0
()+122 000000002
skgesig_Win_Unhandl CALL??? kpeDbgSignalHandler 000000000 000000000 000000000
edExceptionFilter() ()+0 000000000
+171
0000000077439490 CALL??? skgesig_Win_Unhandl 000000001 000000000 000000001
edExceptionFilter() 000000000
+0
00000000776543B8 CALL??? 0000000077439330 005CFDB40 000000006 000000000
000000001
00000000775D85A8 CALL??? 0000000077654374 000000000 000000000 000000000
000000000
00000000775E9D0D CALL??? 00000000775D850C 005D00000 005CFFF90 005CFFF90
077702DE8
00000000775D91AF CALL??? 00000000775E9D00 005D00000 0774ADF08 000012F24
00369C680
0000000077611278 CALL??? 00000000775D8E20 005CFE780 005CFE290 000000000
000000000
0000000000402877 CALL??? 000000007761124A 000000000 00056B5BC 005CFE730
005CFE6A8
0000000010011A80 CALL??? 00000000004027B4 000000000 000000000 004FCD010
003790000
0000000010005E3E CALL??? 0000000010011A24 000000000 7FEFD8E1413
004FCD010 004FCD010
000007FEFD92C387 CALL??? 0000000010005E08 00504E900 000000000 000000000
000000000
000007FEFD92C424 CALL??? 000007FEFD92C370 7FEFD971EA0 004FCD010
000000000 000000000
00000000773B652D CALL??? 000007FEFD92C3A8 000000000 000000000 000000000
000000000
00000000775EC521 CALL??? 00000000773B6520 000000000 000000000 000000000
000000000
----- End of Call Stack Trace -----
Some googleing tought me that this has to do with some sqlnet settings for diagnostics. Funny how diagnostics can actually break your stuff on Windows… Anyway, placing a sqlnet.ora in C:\Oracle\asinst_1\config (yes, that’s the config directory of your instance, or any other location if you have modified your opmn.xml manually) with this contents fixed the issue:
DIAG_ADR_ENABLED=OFF
DIAG_SIGHANDLER_ENABLED=FALSE
DIAG_DDE_ENABLED=FALSE
Hope this helps.
rlwrap, wlst and the nodemanager
Lately I have seen a couple of blogposts about wlst and rlwrap (e.g. this one). This blogpost is a friendly warning to all of you who have followed this tip.
If you happen to start your nodemanager from wlst, i.e. like this:
$ rlwrap java weblogic.WLST
Initializing WebLogic Scripting Tool (WLST) …
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
wls:/offline>
wls:/offline>
wls:/offline> startNodeManager(NodeManagerHome=’/u01/app/oracle/nodemanager’,PropertiesFile=’/u01/app/oracle/nodemanager/nodemanager.properties’)
Launching NodeManager …
<>
Node Manager starting in the background
wls:/offline>
your nodemanager gets started just like expected. If you check your process tree you might find something you won’t like:
wls_user 21785 19630 0 17:35 pts/0 00:00:00 rlwrap java weblogic.WLST
wls_user 21786 21785 4 17:35 pts/1 00:00:09 java weblogic.WLST
wls_user 22740 21786 15 17:38 pts/1 00:00:06 /u01/app/oracle/jrmc-4.0.0-1.6.0/jre/bin/java -classpath /u01/app/oracle/jrmc-4.0.0-1.6.0/jre/lib/rt.jar:/u01/app/oracle/jrmc-4.0.0-1.6.0/jre/lib/i18n.jar:/u01/app/oracle/middleware/wlserver_10.3/server/ext/jdbc/oracle/11g/ojdbc6dms.jar:/u01/app/oracle/middleware/patch_wls1033/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/oracle/middleware/patch_ocp353/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/oracle/jrmc-4.0.0-1.6.0/lib/tools.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.jar:/u01/app/oracle/middleware/modules/features/weblogic.server.modules_10.3.3.0.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/webservices.jar:/u01/app/oracle/middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/oracle/middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.jar:/u01/app/oracle/middleware/oracle_common/soa/modules/commons-cli-1.1.jar:/u01/app/oracle/middleware/oracle_common/soa/modules/oracle.soa.mgmt_11.1.1/soa-infra-mgmt.jar:/u01/app/oracle/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrf.jar:/u01/app/oracle/middleware/wlserver_10.3/common/derby/lib/derbyclient.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/xqrl.jar -DPropertiesFile=/u01/app/oracle/middleware/wlserver_10.3/common/nodemanager/nodemanager.properties -DNodeManagerHome=/u01/app/oracle/middleware/wlserver_10.3/common/nodemanager -DQuitEnabled=true weblogic.NodeManager -v
as you can see, the user wls_user owns the pid that started rlwrap. This pid is the parent of the wlst session, which is the parent of the nodemanager pid. Now guess what happens if you exit out of your wlst session?
To make this worse, guess what process is the parent of the startWebLogic.sh script that starts your managed server?
wls_user 23727 22740 0 17:41 ? 00:00:00 /bin/sh /u01/app/oracle/middleware/user_projects/domains/ooid_domain/bin/startWebLogic.sh
So, here’s my friendly advice: do not start the nodemanager with a rl-wrapped wlst session or you will be finding yourself banging your head against the wall.
Oracle MVA 