Oracle MVA

Tales from a Jack of all trades

OTD-62015 An error occurred while creating server certificates

leave a comment »

<UPDATE !!!>
One of my colleagues asked for help creating an OTD configuration on an engineered system. For some reason the creation of the administration server failed. Here’s the command he issued:

-bash-3.2$ export ORACLE_HOME=/u01/app/oracle/product/otd
-bash-3.2$ export PATH=$ORACLE_HOME/bin:$PATH
-bash-3.2$ $ORACLE_HOME/bin/tadm configure-server --host=my_host --java-home=$ORACLE_HOME/jdk --port=8989 --user=admin --instance-home=/u01/app/oracle/admin/otd/otdadmin --server-user=oracle --port 8989 --verbose
This command will create the administration server. The password that is provided will be required to access the administration server.
Enter admin-user-password>
Enter admin-user-password again>
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ConfigureServer validateRuntimeUser
FINEST: Checking availability of valid runtime user...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance init
FINEST: Initing AdminServerInstance
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: Initing ServerInstance...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareDirsAndFiles
FINEST: AdminServerInstance.prepareDirsAndFiles()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareInstanceNameAndDir
FINEST: AdminServerInstance.prepareInstanceNameAndDir()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareTokens
FINEST: AdminServerInstance.prepareTokens()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance prepareTokens
FINEST: ServerInstance.prepareTokens()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: isWindows = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: oracleHome = /u01/app/oracle/product/otd
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: instanceHome = /u01/app/oracle/admin/otd/otdadmin
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: cfgTmplPath = /u01/app/oracle/product/otd/lib/templates/config
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: scriptsTmplPath = /u01/app/oracle/product/otd/lib/templates/scripts
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: configName = admin-server
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: unixUser = null
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: isZip = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: createService = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance
FINEST: In AdminServerInstance constructor :: after calling super
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance
FINEST: 		 logger is null = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance createInstance
FINEST: Starting to create server instance...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance createDirectories
FINEST: Starting to create instance directory structure...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance setupSecurityDB
FINEST: AdminServerInstance.setupSecurityDB
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance setupSecurityDB
FINEST: dbDir = /u01/app/oracle/admin/otd/otdadmin/admin-server/config
Jan 14, 2014 11:06:54 AM com.sun.web.admin.configurator.AdminServerInstance createAdminCerts
FINEST: Starting to setup the administration self-signed certificates
Jan 14, 2014 11:06:55 AM com.sun.web.admin.configurator.AdminServerInstance createAdminCerts
FINEST: java.lang.SecurityException: Unable to initialize security library
com.sun.web.admin.security.NSSDBException: java.lang.SecurityException: Unable to initialize security library
	at com.sun.web.admin.security.SecurityUtil.initDB(SecurityUtil.java:69)
	at com.sun.web.admin.configurator.AdminServerInstance.createAdminCerts(AdminServerInstance.java:161)
	at com.sun.web.admin.configurator.AdminServerInstance.setupSecurityDB(AdminServerInstance.java:101)
	at com.sun.web.admin.configurator.ServerInstance.createInstance(ServerInstance.java:604)
	at com.sun.web.admin.configurator.ConfigureServer.configureServer(ConfigureServer.java:111)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.configure(ConfigureServerCommand.java:93)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.configureServer(ConfigureServerCommand.java:48)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.runCommand(ConfigureServerCommand.java:29)
	at com.sun.enterprise.cli.framework.CLIMain.invokeCommand(CLIMain.java:171)
	at com.sun.web.admin.cli.shelladapter.WSadminShell.invokeFramework(WSadminShell.java:162)
	at com.sun.web.admin.cli.shelladapter.WSadminShell.main(WSadminShell.java:79)
Caused by: java.lang.SecurityException: Unable to initialize security library
	at org.mozilla.jss.CryptoManager.initializeAllNative(Native Method)
	at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:919)
	at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:885)
	at com.sun.web.admin.security.SecurityUtil.initDB(SecurityUtil.java:62)
	... 10 more

OTD-62015 An error occurred while creating server certificates: java.lang.SecurityException: Unable to initialize security library

Now this seemed interesting to me, since I never had this error before. So, fond of tracing as I am I started an strace

strace -f -o /tmp/tadm.trc $ORACLE_HOME/bin/tadm configure-server --host=my_host --java-home=$ORACLE_HOME/jdk --port=8989 --user=admin --instance-home=/u01/app/oracle/admin/otd/otdadmin --server-user=oracle --port 8989 --verbose

This gave me a rather extensive trace file (close to 12k lines) which I won’t bother you with. One of the relevant lines that draw my attention was:

fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824, len=1}) = -1 ENOLCK (No locks available)

So, it is a NFS locking issue! Checking /etc/mtab showed me that the instance home was on a NFS mount: /u01/app/oracle/admin/otd . I changed the mountoptions to include noac,nolock and this instantly solved the error.

Hope this helps.

<UPDATE>
Well, that noac option caused some severe performance issues. Seems that this database best practice doesn’t work so much on Exalogic.

The nolock option should be handled with care. If you are absolutely sure that files can only be opened from one location this could solve the issues, but I was told by experts to avoid this as much as possible. Removing the nolock option did bring me back to a crashing tadm though. Back to the drawing board….

Written by Jacco H. Landlust

January 14, 2014 at 12:32 pm

yum exclude list for Exalogic vServers

leave a comment »

Recently I have been doing some work on Exalogic. While building a template for vServers on Exalogic I ran into an issue. After executing yum update following by a reboot, I wasn’t able to connect to the vServers anymore. This is caused by an issue with the network stack which, in the end, is caused by an documentation error.

It seems that the yum exclude list for vServers is not correctly documented , also Oracle Support Document 1594674.1 (Exalogic Virtual Environment – Guest vServer Upgrade to Oracle Linux v5.10 ) seems to be off.  The exclusion list that didn’t break the operating system after a yum update is:

exclude=kernel* compat-dapl* dapl* ib-bonding* ibacm* ibutils* ibsim* infiniband-diags* kmod-ovmapi-uek* libibcm* libibmad* libibumad* libibverbs* libmlx4* libovmapi* librdmacm* libsdp* mpi-selector* mpitests_openmpi_gcc* mstflint* mvapich* ofa* ofed* openmpi_gcc* opensm* ovm-template-config* ovmd* perftest* qperf* rds-tools* sdpnetstat* srptools* exalogic* infinibus* xenstoreprovider* initscripts* nfs-utils*

Written by Jacco H. Landlust

January 3, 2014 at 3:17 pm

new SOA HA paper

leave a comment »

Today I was pointed at a brand new SOA HA paper on OTN (thanks Simon. Although I didn’t give any direct input for the paper, it discusses the architecture I designed for my largest customer. I am very happy that Oracle recognizes that customers rely on active/active configurations.

Written by Jacco H. Landlust

August 26, 2013 at 10:09 pm

Lengthy errormessage while creating OID

with one comment

For most of the Fusion Middleware domains or systems components I setup for clients I use scripts. Today, while creating a OID, I decided to type in the commands manually. This resulted in the following:

$ /u01/app/oracle/admin/instances/XXXXX/oid_instance_XXXXXXXX/bin/opmnctl  createcomponent -componentType OID -componentName area51_oid -Db_info rdbms-scan.area51.local:1521:srv4oid -Host oid.area51.local -Port 3060 -Sport 3131

Command requires login to weblogic admin server (oid.area51.local):
  Username: weblogic
  Password:

Creating empty component directories...Done
Provisioning OID files for area51_oid
  OID onCreate....

Enter ODS password:
  Validating OID input parameters
Enter ODSSM password:
oracle.as.config.ProvisionException: Error deleting credential odssm from CSF
        at oracle.iam.management.oid.install.wls.OIDComponentHelper.delCredFromCSF(OIDComponentHelper.java:2373)
        at oracle.iam.management.oid.install.wls.OIDComponentHelper.removeCreds(OIDComponentHelper.java:2345)
        at oracle.iam.management.oid.install.wls.OIDComponent.onRemove(OIDComponent.java:429)
        at oracle.as.config.impl.OracleASComponentBaseImpl.remove(OracleASComponentBaseImpl.java:287)
        at oracle.as.config.impl.OracleASComponentBaseImpl.remove(OracleASComponentBaseImpl.java:174)
        at oracle.as.config.impl.OracleASComponentBaseImpl.remove(OracleASComponentBaseImpl.java:155)
        at oracle.iam.management.oid.install.wls.OIDComponent.onCreate(OIDComponent.java:227)
        at oracle.as.config.impl.OracleASComponentBaseImpl.createComponent(OracleASComponentBaseImpl.java:597)
        at oracle.as.config.impl.OracleASComponentBaseImpl.create(OracleASComponentBaseImpl.java:106)
        at oracle.as.config.provisioner.commands.CreateComponentCommand.execute(CreateComponentCommand.java:40)
        at oracle.as.config.provisioner.InstallerCmdLine.run(InstallerCmdLine.java:146)
        at oracle.as.config.provisioner.InstallerCmdLine.main(InstallerCmdLine.java:46)
Caused by: java.security.PrivilegedActionException: oracle.as.config.ProvisionException:
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.iam.management.oid.install.wls.OIDComponentHelper.delCredFromCSF(OIDComponentHelper.java:2354)
        ... 11 more
Caused by: oracle.as.config.ProvisionException:
        at oracle.iam.management.oid.install.wls.OIDComponentHelper$3.run(OIDComponentHelper.java:2367)
        ... 13 more
Caused by: oracle.security.jps.config.JpsConfigurationException: /u01/app/oracle/admin/instances/eoid2/oid_instance_oesv9510/config/JPS/jps-config-jse.xml (No such file or directory)
        at oracle.security.jps.internal.config.xml.XmlConfigurationFactory.initDefaultConfiguration(XmlConfigurationFactory.java:439)
        at oracle.security.jps.internal.config.xml.XmlConfigurationFactory.getDefaultConfiguration(XmlConfigurationFactory.java:338)
        at oracle.security.jps.internal.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:160)
        at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.(JpsContextFactoryImpl.java:112)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:355)
        at java.lang.Class.newInstance(Class.java:308)
        at oracle.security.jps.util.JpsUtil.newInstance(JpsUtil.java:190)
        at oracle.security.jps.JpsContextFactory$1.run(JpsContextFactory.java:74)
        at oracle.security.jps.JpsContextFactory$1.run(JpsContextFactory.java:72)
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.security.jps.JpsContextFactory.getContextFactory(JpsContextFactory.java:71)
        at oracle.iam.management.oid.install.wls.OIDComponentHelper$3.run(OIDComponentHelper.java:2357)
        ... 13 more
Caused by: java.io.FileNotFoundException: /u01/app/oracle/admin/instances/eoid2/oid_instance_oesv9510/config/JPS/jps-config-jse.xml (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.(FileInputStream.java:120)
        at oracle.security.jps.internal.common.util.XmlSchemaValidationUtil.doValidation(XmlSchemaValidationUtil.java:96)
        at oracle.security.jps.internal.config.xml.XmlConfigurationFactory.initDefaultConfiguration(XmlConfigurationFactory.java:418)
        ... 28 more
    Skipping oesv9510_oid unregistration. It is not currently registered with the adminserver.
    Deleting oesv9510_oid directories
    Invoking opmn reload...Done
Command failed: Exception in onCreate()
Details are logged in /u01/app/oracle/admin/instances/eoid2/oid_instance_oesv9510/diagnostics/logs/OPMN/opmn/provision.log

opmnctl createcomponent: failed.

Now this is a rather lengthy error message and it really surprised me. Since I just associated the security store for the domain to a database, and this error pointing towards JPS-config I figured something must be wrong with the reassociateSecurityStore wlst comamnd. So I checked logfiles, My Oracle Support and Google before I checked the provision.log.

The provision log showed me these messages

SEVERE: Command failed:
oracle.as.config.ProvisionException: Exception in onCreate()
        at oracle.iam.management.oid.install.wls.OIDComponent.onCreate(OIDComponent.java:235)
        at oracle.as.config.impl.OracleASComponentBaseImpl.createComponent(OracleASComponentBaseImpl.java:597)
        at oracle.as.config.impl.OracleASComponentBaseImpl.create(OracleASComponentBaseImpl.java:106)
        at oracle.as.config.provisioner.commands.CreateComponentCommand.execute(CreateComponentCommand.java:40)
        at oracle.as.config.provisioner.InstallerCmdLine.run(InstallerCmdLine.java:146)
        at oracle.as.config.provisioner.InstallerCmdLine.main(InstallerCmdLine.java:46)
Caused by: oracle.as.config.ProvisionException: -Namespace parameter missing
        at oracle.iam.management.oid.install.wls.OIDComponentHelper.validateParams(OIDComponentHelper.java:314)
        at oracle.iam.management.oid.install.wls.OIDComponent.onCreate(OIDComponent.java:158)
        ... 5 more

Aha. So that was a enormous error message that tried to tell me “Hey, you missed the -Namespace parameter”.

P.S. 11.1.1.7 also introduced a new process called oiddispd. documentation” gives this description:
“Beginning with Oracle Internet Directory 11g Release 1 (11.1.1.7.0), the OIDLDAPD process is separated as the OIDDISPD (dispatcher) process and the OIDLDAPD (server) process. On UNIX and Linux systems, however, the ps -ef command will continue to show both of these processes as OIDLDAPD at runtime.”

If you happen to separate software from configuration like I do (binaries owned by oracle, oid processes run as some other user) you need to chown the $ORACLE_HOME/bin/oiddispd process and chmod it to 4740

Hope this helps.

Written by Jacco H. Landlust

July 19, 2013 at 4:03 pm

Posted in Uncategorized

Active Data Guard & Fusion Middleware Repositories.

with one comment

Last year while working on a POC Rob den Braber noticed the following in Disaster Recovery for Oracle Elastic Cloud with Oracle ExaData Database Machine on page 13:

Currently, Oracle Fusion Middleware does not support configuring Oracle Active Data Guard for the database repositories that are a part of the Fusion Middleware topology. However, Active Data Guard can be configured if your custom applications are designed to leverage the technology.
Today this came up in a discussion with Simon Haslam , and he didn’t hear from this support issue before. So it seems that it is not that well know that Active Data Guard and Oracle Fusion Middleware is not a supported combination.
This makes this blog post a reminder from what is already in documentation (unless someone can comment and tell me that currently in the quote is not so currently anymore).
Hope this helps.
UPDATE:
While reading this brand new SOA HA paper I found this quote today:

The Oracle Active Data Guard Option available with Oracle Database 11g Enterprise Edition enables you to open a physical standby database for read-only access for reporting, for simple or complex queries, or sorting while Redo Apply continues to apply changes from the production database. Oracle Fusion Middleware SOA does not support Oracle Active Data Guard because the SOA components execute and update information regarding SOA composite instances in the database as soon as they are started.

Written by Jacco H. Landlust

April 26, 2013 at 4:43 pm

JDBC statement cache setting

leave a comment »

Recently I was asked about the statement cache setting in WebLogic by a colleague. Reason he asked about it, was that documentation wasn’t making any sense to him in combination with advise given to him from an external expert. Here’s the doc he was referring to.

The tooltip documentation in WebLogic says:

WebLogic Server can reuse statements in the cache without reloading the statements, which can increase server performance. Each connection in the connection pool has its own cache of statements.

Now this suggests that WebLogic is maintaining some kind of cache, but really it isn’t (in combination with an Oracle database). All it is doing is opening a cursor on the Oracle database and reusing this cursor.

To demonstrate what is happening I created a small example. The example I use is an sqlauthenticator for WebLogic, allowing users in some database table to authenticate in WebLogic. In this presentation you can find the DDL and DML for the tables and a description how to setup this sql authenticator.

So, my initial database has a statement cache of 10 (default). When I restart the database and WebLogic and I login to the WebLogic console, I can find the following open cursors:

<br />select hash_value, cursor_type, sql_text<br />  from v$open_cursor<br /> where user_name = 'DEMO'<br />/<br />HASH_VALUE CURSOR_TYPE SQL_TEXT<br />---------- --------- ------------------------------------------------<br />32127143   OPEN      SELECT 1 FROM DUAL<br />238104037  OPEN      SELECT G_NAME FROM GROUPMEMBERS WHERE G_MEMBER = :1<br />3221480394 OPEN      SELECT U_PASSWORD FROM USERS WHERE U_NAME = :1<br /><br />3 rows selected.<br />

The minute I reconfigure the statement cache to 0 (=disabled), restart database and WebLogic and login to the console, I find the following open cursors:

<br />HASH_VALUE CURSOR_TYPE SQL_TEXT<br />---------- --------- ------------------------------------------------<br />238104037  OPEN      SELECT G_NAME FROM GROUPMEMBERS WHERE G_MEMBER = :1<br />1 row selected.<br />

This simple test teaches me that a cursor is kept open on the users table and on dual. The query that is running on dual is actually the test query for the datasource.

It would suggest that the statement-cache does keep an administration on which query has run over which connection. This test is too small to bring proof off that. Also I wonder what happens in combination with the pin to thread setting of the jdbc driver. Food for a new blogpost :)

So, in short: the statement cache of your datasource has a direct impact on the number of open cursors. This can (is) improving performance, you don’t have to create a new cursor when you reuse a statement. Setting the statement cache to 0 (disable the cache) is in my opinion not a best practice, by default every session to your 11.2 database can have 50 cursors so you got plenty to spare. You should tune open_cursors and session_cached_cursors on the database according to your applications need.

Hope this helps.

Written by Jacco H. Landlust

April 19, 2013 at 10:55 pm

Posted in RDBMS, Weblogic

Oracle Database Applicance With WebLogic Server (ODA X3-2)

with 5 comments

On april 3th the new ODA X3-2 was released. Sadly I was sick from april 1th on so I had to miss the launch, and I was so well prepared…  others had the scoop. Anyway, as an administrator that not only manages databases this release is pretty exciting since it brings not only virtualization but also WebLogic to ODA. This would make ODA a pretty good appliance for some of my customers, so I did a little investigation in the product.

This blogpost is the first result of that investigation. My main focus was the WebLogic part of the box. The questions that arose with me were either answered by documentation or by product management. Obviously that doesn’t guarantee that I understood everything correct :) I left out references to documentation on  purpose, it would be smart for everyone interested in the product to hit the documentation thoroughly.

The most import slide in the slide deck I received about the ODA launch is this:

oda-slide

It does some pretty smart claims that can be verified easily. The three simplified statements call for some clarification. Here’s what my questions were, plus the answers I found:

Simplified provisioning / install

Q: Can we test any of this without ODA?

A: No, although I was able to get a virtual ODA in a virtual box environment. This is by no means supported and requires altering of the images that Oracle sends you. 

Q: So how do you configure this beast?

A: You install an image on the system with Oracle VM that you can download freely from My Oracle Support. This image contains oakcli which is the cli used to manage the ODA.

Q: Ah, Oracle VM. Where is the Oracle VM Manager?

A: there is none. oakcli deploys all your VM’s.

Q: ODA is 2 physical machines running OVM, where is the shared storage?

A: The only shared storage available is database shared storage, i.e. DBFS. ARGH… DBFS is already on my todo list! 

Q: So no HA features from OVM?

A: No.

Q: What about the VM’s that oakcli deploys, can I build my own templates?

A: No you cannot. Well, technically you can, but it’s not supported.

Q: what a minute, no custom templates? What about adding layered products to the VM?

A: No can’t do. Currently only WebLogic is supported.

Q: Well, if I can’t define my own templates, what about my WebLogic domain structure?

A: To my understanding that’s fixed too: one Administration Server on it’s own VM, two managed servers in one cluster (on two VM’s) and two Oracle Traffic Director (OTD) VM’s.

Q: What is the difference between that ODA-BASE VM and the other dom-u’s?

A: The ODA-BASE VM is the only one that can actually connect to the local disks directly. 

Q: So that means you should run databases preferably in the ODA-BASE VM ?

A: Yes.

investment model (a.k.a. licenses)

Q: How does this “pay-as-you-grow” thing work partition wise?

A: It is VM hard partitioning. Not Oracle trusted partition as on ExaLogic. And partitioning only works in multiples of two (2).

Q: So I pay per core, is hyper threading turned on?

A: Yes, but I didn’t find out yet what that means for your licenses….

Q: So I can scale up and down?

A: No. Oracle expects you to grow, not to scale down. You can scale down software, not licenses.

Q: What about this separate administration server?

A: License wise that should be treated as a managed server (= pay for it)

Q: And those OTD’s? Do I have to pay for them too?

A: No. OTD is included with WebLogic Enterprise Edition and WebLogic Suite.

maintenance

Q: The JDK is in the middleware home, how does that work with upgrades?

A: Oracle will provide patches as needed.

Q: So how does a domain  upgrade work?

A: Currently not supported. So no maintenance version wise.

Q: An EM agent exists on every VM? Which version is that?

A: Currently there is not EM agent installed. Oracle plans to have the agent installed and support in next patch releases. This will be a 12c EM agent.

Well. That covers all my findings. Hope it helps you in your investigation of ODA.

Written by Jacco H. Landlust

April 5, 2013 at 9:34 pm

Posted in RDBMS, Weblogic

Follow

Get every new post delivered to your Inbox.

Join 297 other followers