OTD-62015 An error occurred while creating server certificates

<UPDATE !!!>
One of my colleagues asked for help creating an OTD configuration on an engineered system. For some reason the creation of the administration server failed. Here’s the command he issued:

-bash-3.2$ export ORACLE_HOME=/u01/app/oracle/product/otd
-bash-3.2$ export PATH=$ORACLE_HOME/bin:$PATH
-bash-3.2$ $ORACLE_HOME/bin/tadm configure-server --host=my_host --java-home=$ORACLE_HOME/jdk --port=8989 --user=admin --instance-home=/u01/app/oracle/admin/otd/otdadmin --server-user=oracle --port 8989 --verbose
This command will create the administration server. The password that is provided will be required to access the administration server.
Enter admin-user-password>
Enter admin-user-password again>
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ConfigureServer validateRuntimeUser
FINEST: Checking availability of valid runtime user...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance init
FINEST: Initing AdminServerInstance
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: Initing ServerInstance...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareDirsAndFiles
FINEST: AdminServerInstance.prepareDirsAndFiles()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareInstanceNameAndDir
FINEST: AdminServerInstance.prepareInstanceNameAndDir()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareTokens
FINEST: AdminServerInstance.prepareTokens()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance prepareTokens
FINEST: ServerInstance.prepareTokens()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: isWindows = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: oracleHome = /u01/app/oracle/product/otd
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: instanceHome = /u01/app/oracle/admin/otd/otdadmin
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: cfgTmplPath = /u01/app/oracle/product/otd/lib/templates/config
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: scriptsTmplPath = /u01/app/oracle/product/otd/lib/templates/scripts
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: configName = admin-server
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: unixUser = null
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: isZip = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: createService = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance
FINEST: In AdminServerInstance constructor :: after calling super
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance
FINEST: 		 logger is null = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance createInstance
FINEST: Starting to create server instance...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance createDirectories
FINEST: Starting to create instance directory structure...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance setupSecurityDB
FINEST: AdminServerInstance.setupSecurityDB
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance setupSecurityDB
FINEST: dbDir = /u01/app/oracle/admin/otd/otdadmin/admin-server/config
Jan 14, 2014 11:06:54 AM com.sun.web.admin.configurator.AdminServerInstance createAdminCerts
FINEST: Starting to setup the administration self-signed certificates
Jan 14, 2014 11:06:55 AM com.sun.web.admin.configurator.AdminServerInstance createAdminCerts
FINEST: java.lang.SecurityException: Unable to initialize security library
com.sun.web.admin.security.NSSDBException: java.lang.SecurityException: Unable to initialize security library
	at com.sun.web.admin.security.SecurityUtil.initDB(SecurityUtil.java:69)
	at com.sun.web.admin.configurator.AdminServerInstance.createAdminCerts(AdminServerInstance.java:161)
	at com.sun.web.admin.configurator.AdminServerInstance.setupSecurityDB(AdminServerInstance.java:101)
	at com.sun.web.admin.configurator.ServerInstance.createInstance(ServerInstance.java:604)
	at com.sun.web.admin.configurator.ConfigureServer.configureServer(ConfigureServer.java:111)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.configure(ConfigureServerCommand.java:93)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.configureServer(ConfigureServerCommand.java:48)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.runCommand(ConfigureServerCommand.java:29)
	at com.sun.enterprise.cli.framework.CLIMain.invokeCommand(CLIMain.java:171)
	at com.sun.web.admin.cli.shelladapter.WSadminShell.invokeFramework(WSadminShell.java:162)
	at com.sun.web.admin.cli.shelladapter.WSadminShell.main(WSadminShell.java:79)
Caused by: java.lang.SecurityException: Unable to initialize security library
	at org.mozilla.jss.CryptoManager.initializeAllNative(Native Method)
	at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:919)
	at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:885)
	at com.sun.web.admin.security.SecurityUtil.initDB(SecurityUtil.java:62)
	... 10 more

OTD-62015 An error occurred while creating server certificates: java.lang.SecurityException: Unable to initialize security library

Now this seemed interesting to me, since I never had this error before. So, fond of tracing as I am I started an strace

strace -f -o /tmp/tadm.trc $ORACLE_HOME/bin/tadm configure-server --host=my_host --java-home=$ORACLE_HOME/jdk --port=8989 --user=admin --instance-home=/u01/app/oracle/admin/otd/otdadmin --server-user=oracle --port 8989 --verbose

This gave me a rather extensive trace file (close to 12k lines) which I won’t bother you with. One of the relevant lines that draw my attention was:

fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824, len=1}) = -1 ENOLCK (No locks available)

So, it is a NFS locking issue! Checking /etc/mtab showed me that the instance home was on a NFS mount: /u01/app/oracle/admin/otd . I changed the mountoptions to include noac,nolock and this instantly solved the error.

Hope this helps.

<UPDATE>
Well, that noac option caused some severe performance issues. Seems that this database best practice doesn’t work so much on Exalogic.

The nolock option should be handled with care. If you are absolutely sure that files can only be opened from one location this could solve the issues, but I was told by experts to avoid this as much as possible. Removing the nolock option did bring me back to a crashing tadm though. Back to the drawing board….

yum exclude list for Exalogic vServers

Recently I have been doing some work on Exalogic. While building a template for vServers on Exalogic I ran into an issue. After executing yum update following by a reboot, I wasn’t able to connect to the vServers anymore. This is caused by an issue with the network stack which, in the end, is caused by an documentation error.

It seems that the yum exclude list for vServers is not correctly documented , also Oracle Support Document 1594674.1 (Exalogic Virtual Environment – Guest vServer Upgrade to Oracle Linux v5.10 ) seems to be off.  The exclusion list that didn’t break the operating system after a yum update is:

exclude=kernel* compat-dapl* dapl* ib-bonding* ibacm* ibutils* ibsim* infiniband-diags* kmod-ovmapi-uek* libibcm* libibmad* libibumad* libibverbs* libmlx4* libovmapi* librdmacm* libsdp* mpi-selector* mpitests_openmpi_gcc* mstflint* mvapich* ofa* ofed* openmpi_gcc* opensm* ovm-template-config* ovmd* perftest* qperf* rds-tools* sdpnetstat* srptools* exalogic* infinibus* xenstoreprovider* initscripts* nfs-utils*

rlwrap, wlst and the nodemanager

Lately I have seen a couple of blogposts about wlst and rlwrap (e.g. this one). This blogpost is a friendly warning to all of you who have followed this tip.

If you happen to start your nodemanager from wlst, i.e. like this:

$ rlwrap java weblogic.WLST

Initializing WebLogic Scripting Tool (WLST) …

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline>
wls:/offline>
wls:/offline> startNodeManager(NodeManagerHome=’/u01/app/oracle/nodemanager’,PropertiesFile=’/u01/app/oracle/nodemanager/nodemanager.properties’)
Launching NodeManager …

<>

Node Manager starting in the background
wls:/offline>

your nodemanager gets started just like expected. If you check your process tree you might find something you won’t like:

wls_user 21785 19630 0 17:35 pts/0 00:00:00 rlwrap java weblogic.WLST
wls_user 21786 21785 4 17:35 pts/1 00:00:09 java weblogic.WLST
wls_user 22740 21786 15 17:38 pts/1 00:00:06 /u01/app/oracle/jrmc-4.0.0-1.6.0/jre/bin/java -classpath /u01/app/oracle/jrmc-4.0.0-1.6.0/jre/lib/rt.jar:/u01/app/oracle/jrmc-4.0.0-1.6.0/jre/lib/i18n.jar:/u01/app/oracle/middleware/wlserver_10.3/server/ext/jdbc/oracle/11g/ojdbc6dms.jar:/u01/app/oracle/middleware/patch_wls1033/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/oracle/middleware/patch_ocp353/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/oracle/jrmc-4.0.0-1.6.0/lib/tools.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.jar:/u01/app/oracle/middleware/modules/features/weblogic.server.modules_10.3.3.0.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/webservices.jar:/u01/app/oracle/middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/oracle/middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.jar:/u01/app/oracle/middleware/oracle_common/soa/modules/commons-cli-1.1.jar:/u01/app/oracle/middleware/oracle_common/soa/modules/oracle.soa.mgmt_11.1.1/soa-infra-mgmt.jar:/u01/app/oracle/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrf.jar:/u01/app/oracle/middleware/wlserver_10.3/common/derby/lib/derbyclient.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/xqrl.jar -DPropertiesFile=/u01/app/oracle/middleware/wlserver_10.3/common/nodemanager/nodemanager.properties -DNodeManagerHome=/u01/app/oracle/middleware/wlserver_10.3/common/nodemanager -DQuitEnabled=true weblogic.NodeManager -v


as you can see, the user wls_user owns the pid that started rlwrap. This pid is the parent of the wlst session, which is the parent of the nodemanager pid. Now guess what happens if you exit out of your wlst session?

To make this worse, guess what process is the parent of the startWebLogic.sh script that starts your managed server?

wls_user 23727 22740 0 17:41 ? 00:00:00 /bin/sh /u01/app/oracle/middleware/user_projects/domains/ooid_domain/bin/startWebLogic.sh

So, here’s my friendly advice: do not start the nodemanager with a rl-wrapped wlst session or you will be finding yourself banging your head against the wall.

iscsi-targets

I am build a new environment on my testing-kit. Instead of downloading OpenFiler, I decided to build my own ISCSI device on OEL 5. The main reason for this exercise is that I want this box to be DNS server and some more.

Anyway, configuring ISCSI is not an average DBA’s job. I don’t like to type in commands on a prompt when I don’t know what they mean. Every how-to I find keeps on calling difficult commands to create a ISCSI LUN, which made me spent lots of time in man-pages last night. In the end this was a waste of time, since all you need to do is:

• add a disk to your VM (let’s say /dev/sdb)
• install perl-Config-General and scsi-target-utils rpm’s from the ClusterStorage directory on the DVD with your installation media
• edit /etc/tgt/targets.conf and make it look like this:
  ASM1>
backing-store /dev/sdb
</target>
where area51.local is my domain, ASM1 is my LUN and /dev/sdb is the disk just added to the VM
• make the tgtd daemon start
  chkconfig 345 tgtd on; service tgtd start

Now whenever you restart your server, you will still have the same ISCSI LUN presented to the world. No big man-page needed, just a simple configuration file. How about that….

Obviously, when you want to check the LUN, you do need the tgtadm command. This should do the trick:

tgtadm --lld iscsi --op show --mode target

Group existence

Usually I work on Linux and I love it. For some sort of reason it just took me an hour to find out if a group existed and what the gid was (ldap was configured). Therefore I make this not to myself: getent is cool!

The easiest way I found to check for group existence is:

$ getent group dba
dba:x:4006:

And, other way around, if you have the gid here’s how you find the group name :

$ getent group 4006
dba:x:4006:


</end reminder>

EUS and asmcmd

I have been working a lot with EUS lately at a big customer. My personal account is able to login to databases (EUS) and also on to OEL (OAS4OS). This combined with some chown/chmod commands on OEL enables me to do my job with my personal account.

Since this customers also uses ASM, I figured I would like to use my personal account for asmcmd too. First I tested the process with a local account, baby steps usually works best for me. I created an account jhl

# useradd -g asmadmin -G dba jhl

Next i su’d to jhl and tested the procedure:

$ id
uid=10238(jhl) gid=4007(asmadmin) groups=4006(dba),4007(asmadmin)

$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/asm_200 is /u01/app/oracle

$ asmcmd
ASMCMD> ls

This looks promising, all needed to be done next was repeating the steps only now with an account from the OID. First I had to add the group to the OID, here’s the ldif I used:

startup scripts

Every now and then there’s another discussion about how to create startup scripts for a database. Frits Hoogland just made an interesting post, somehow I get the feeling this information is new for most people. Being brought up with Linux instead of an old unix starts getting more and more useful 😉