Oracle MVA

Tales from a Jack of all trades

OTD-62015 An error occurred while creating server certificates

leave a comment »

<UPDATE !!!>
One of my colleagues asked for help creating an OTD configuration on an engineered system. For some reason the creation of the administration server failed. Here’s the command he issued:

-bash-3.2$ export ORACLE_HOME=/u01/app/oracle/product/otd
-bash-3.2$ export PATH=$ORACLE_HOME/bin:$PATH
-bash-3.2$ $ORACLE_HOME/bin/tadm configure-server --host=my_host --java-home=$ORACLE_HOME/jdk --port=8989 --user=admin --instance-home=/u01/app/oracle/admin/otd/otdadmin --server-user=oracle --port 8989 --verbose
This command will create the administration server. The password that is provided will be required to access the administration server.
Enter admin-user-password>
Enter admin-user-password again>
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ConfigureServer validateRuntimeUser
FINEST: Checking availability of valid runtime user...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance init
FINEST: Initing AdminServerInstance
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: Initing ServerInstance...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareDirsAndFiles
FINEST: AdminServerInstance.prepareDirsAndFiles()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareInstanceNameAndDir
FINEST: AdminServerInstance.prepareInstanceNameAndDir()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareTokens
FINEST: AdminServerInstance.prepareTokens()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance prepareTokens
FINEST: ServerInstance.prepareTokens()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: isWindows = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: oracleHome = /u01/app/oracle/product/otd
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: instanceHome = /u01/app/oracle/admin/otd/otdadmin
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: cfgTmplPath = /u01/app/oracle/product/otd/lib/templates/config
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: scriptsTmplPath = /u01/app/oracle/product/otd/lib/templates/scripts
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: configName = admin-server
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: unixUser = null
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: isZip = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: createService = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance
FINEST: In AdminServerInstance constructor :: after calling super
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance
FINEST: 		 logger is null = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance createInstance
FINEST: Starting to create server instance...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance createDirectories
FINEST: Starting to create instance directory structure...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance setupSecurityDB
FINEST: AdminServerInstance.setupSecurityDB
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance setupSecurityDB
FINEST: dbDir = /u01/app/oracle/admin/otd/otdadmin/admin-server/config
Jan 14, 2014 11:06:54 AM com.sun.web.admin.configurator.AdminServerInstance createAdminCerts
FINEST: Starting to setup the administration self-signed certificates
Jan 14, 2014 11:06:55 AM com.sun.web.admin.configurator.AdminServerInstance createAdminCerts
FINEST: java.lang.SecurityException: Unable to initialize security library
com.sun.web.admin.security.NSSDBException: java.lang.SecurityException: Unable to initialize security library
	at com.sun.web.admin.security.SecurityUtil.initDB(SecurityUtil.java:69)
	at com.sun.web.admin.configurator.AdminServerInstance.createAdminCerts(AdminServerInstance.java:161)
	at com.sun.web.admin.configurator.AdminServerInstance.setupSecurityDB(AdminServerInstance.java:101)
	at com.sun.web.admin.configurator.ServerInstance.createInstance(ServerInstance.java:604)
	at com.sun.web.admin.configurator.ConfigureServer.configureServer(ConfigureServer.java:111)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.configure(ConfigureServerCommand.java:93)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.configureServer(ConfigureServerCommand.java:48)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.runCommand(ConfigureServerCommand.java:29)
	at com.sun.enterprise.cli.framework.CLIMain.invokeCommand(CLIMain.java:171)
	at com.sun.web.admin.cli.shelladapter.WSadminShell.invokeFramework(WSadminShell.java:162)
	at com.sun.web.admin.cli.shelladapter.WSadminShell.main(WSadminShell.java:79)
Caused by: java.lang.SecurityException: Unable to initialize security library
	at org.mozilla.jss.CryptoManager.initializeAllNative(Native Method)
	at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:919)
	at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:885)
	at com.sun.web.admin.security.SecurityUtil.initDB(SecurityUtil.java:62)
	... 10 more

OTD-62015 An error occurred while creating server certificates: java.lang.SecurityException: Unable to initialize security library

Now this seemed interesting to me, since I never had this error before. So, fond of tracing as I am I started an strace

strace -f -o /tmp/tadm.trc $ORACLE_HOME/bin/tadm configure-server --host=my_host --java-home=$ORACLE_HOME/jdk --port=8989 --user=admin --instance-home=/u01/app/oracle/admin/otd/otdadmin --server-user=oracle --port 8989 --verbose

This gave me a rather extensive trace file (close to 12k lines) which I won’t bother you with. One of the relevant lines that draw my attention was:

fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824, len=1}) = -1 ENOLCK (No locks available)

So, it is a NFS locking issue! Checking /etc/mtab showed me that the instance home was on a NFS mount: /u01/app/oracle/admin/otd . I changed the mountoptions to include noac,nolock and this instantly solved the error.

Hope this helps.

<UPDATE>
Well, that noac option caused some severe performance issues. Seems that this database best practice doesn’t work so much on Exalogic.

The nolock option should be handled with care. If you are absolutely sure that files can only be opened from one location this could solve the issues, but I was told by experts to avoid this as much as possible. Removing the nolock option did bring me back to a crashing tadm though. Back to the drawing board….

Advertisements

Written by Jacco H. Landlust

January 14, 2014 at 12:32 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: