Archive for January 2014
OTD-62015 An error occurred while creating server certificates
<UPDATE !!!>
One of my colleagues asked for help creating an OTD configuration on an engineered system. For some reason the creation of the administration server failed. Here’s the command he issued:
-bash-3.2$ export ORACLE_HOME=/u01/app/oracle/product/otd -bash-3.2$ export PATH=$ORACLE_HOME/bin:$PATH -bash-3.2$ $ORACLE_HOME/bin/tadm configure-server --host=my_host --java-home=$ORACLE_HOME/jdk --port=8989 --user=admin --instance-home=/u01/app/oracle/admin/otd/otdadmin --server-user=oracle --port 8989 --verbose This command will create the administration server. The password that is provided will be required to access the administration server. Enter admin-user-password> Enter admin-user-password again> Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ConfigureServer validateRuntimeUser FINEST: Checking availability of valid runtime user... Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance init FINEST: Initing AdminServerInstance Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init FINEST: Initing ServerInstance... Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareDirsAndFiles FINEST: AdminServerInstance.prepareDirsAndFiles() Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareInstanceNameAndDir FINEST: AdminServerInstance.prepareInstanceNameAndDir() Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareTokens FINEST: AdminServerInstance.prepareTokens() Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance prepareTokens FINEST: ServerInstance.prepareTokens() Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init FINEST: isWindows = false Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init FINEST: oracleHome = /u01/app/oracle/product/otd Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init FINEST: instanceHome = /u01/app/oracle/admin/otd/otdadmin Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init FINEST: cfgTmplPath = /u01/app/oracle/product/otd/lib/templates/config Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init FINEST: scriptsTmplPath = /u01/app/oracle/product/otd/lib/templates/scripts Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init FINEST: configName = admin-server Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init FINEST: unixUser = null Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init FINEST: isZip = false Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init FINEST: createService = false Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance FINEST: In AdminServerInstance constructor :: after calling super Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance FINEST: logger is null = false Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance createInstance FINEST: Starting to create server instance... Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance createDirectories FINEST: Starting to create instance directory structure... Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance setupSecurityDB FINEST: AdminServerInstance.setupSecurityDB Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance setupSecurityDB FINEST: dbDir = /u01/app/oracle/admin/otd/otdadmin/admin-server/config Jan 14, 2014 11:06:54 AM com.sun.web.admin.configurator.AdminServerInstance createAdminCerts FINEST: Starting to setup the administration self-signed certificates Jan 14, 2014 11:06:55 AM com.sun.web.admin.configurator.AdminServerInstance createAdminCerts FINEST: java.lang.SecurityException: Unable to initialize security library com.sun.web.admin.security.NSSDBException: java.lang.SecurityException: Unable to initialize security library at com.sun.web.admin.security.SecurityUtil.initDB(SecurityUtil.java:69) at com.sun.web.admin.configurator.AdminServerInstance.createAdminCerts(AdminServerInstance.java:161) at com.sun.web.admin.configurator.AdminServerInstance.setupSecurityDB(AdminServerInstance.java:101) at com.sun.web.admin.configurator.ServerInstance.createInstance(ServerInstance.java:604) at com.sun.web.admin.configurator.ConfigureServer.configureServer(ConfigureServer.java:111) at com.sun.web.admin.cli.commands.ConfigureServerCommand.configure(ConfigureServerCommand.java:93) at com.sun.web.admin.cli.commands.ConfigureServerCommand.configureServer(ConfigureServerCommand.java:48) at com.sun.web.admin.cli.commands.ConfigureServerCommand.runCommand(ConfigureServerCommand.java:29) at com.sun.enterprise.cli.framework.CLIMain.invokeCommand(CLIMain.java:171) at com.sun.web.admin.cli.shelladapter.WSadminShell.invokeFramework(WSadminShell.java:162) at com.sun.web.admin.cli.shelladapter.WSadminShell.main(WSadminShell.java:79) Caused by: java.lang.SecurityException: Unable to initialize security library at org.mozilla.jss.CryptoManager.initializeAllNative(Native Method) at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:919) at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:885) at com.sun.web.admin.security.SecurityUtil.initDB(SecurityUtil.java:62) ... 10 more OTD-62015 An error occurred while creating server certificates: java.lang.SecurityException: Unable to initialize security library
Now this seemed interesting to me, since I never had this error before. So, fond of tracing as I am I started an strace
strace -f -o /tmp/tadm.trc $ORACLE_HOME/bin/tadm configure-server --host=my_host --java-home=$ORACLE_HOME/jdk --port=8989 --user=admin --instance-home=/u01/app/oracle/admin/otd/otdadmin --server-user=oracle --port 8989 --verbose
This gave me a rather extensive trace file (close to 12k lines) which I won’t bother you with. One of the relevant lines that draw my attention was:
fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824, len=1}) = -1 ENOLCK (No locks available)
So, it is a NFS locking issue! Checking /etc/mtab showed me that the instance home was on a NFS mount: /u01/app/oracle/admin/otd . I changed the mountoptions to include noac,nolock and this instantly solved the error.
Hope this helps.
<UPDATE>
Well, that noac option caused some severe performance issues. Seems that this database best practice doesn’t work so much on Exalogic.
The nolock option should be handled with care. If you are absolutely sure that files can only be opened from one location this could solve the issues, but I was told by experts to avoid this as much as possible. Removing the nolock option did bring me back to a crashing tadm though. Back to the drawing board….
yum exclude list for Exalogic vServers
Recently I have been doing some work on Exalogic. While building a template for vServers on Exalogic I ran into an issue. After executing yum update following by a reboot, I wasn’t able to connect to the vServers anymore. This is caused by an issue with the network stack which, in the end, is caused by an documentation error.
It seems that the yum exclude list for vServers is not correctly documented , also Oracle Support Document 1594674.1 (Exalogic Virtual Environment – Guest vServer Upgrade to Oracle Linux v5.10 ) seems to be off. The exclusion list that didn’t break the operating system after a yum update is:
exclude=kernel* compat-dapl* dapl* ib-bonding* ibacm* ibutils* ibsim* infiniband-diags* kmod-ovmapi-uek* libibcm* libibmad* libibumad* libibverbs* libmlx4* libovmapi* librdmacm* libsdp* mpi-selector* mpitests_openmpi_gcc* mstflint* mvapich* ofa* ofed* openmpi_gcc* opensm* ovm-template-config* ovmd* perftest* qperf* rds-tools* sdpnetstat* srptools* exalogic* infinibus* xenstoreprovider* initscripts* nfs-utils*
Oracle MVA 