Online Master Key with Oracle Key Vault on a Consolidated Platform

Seems I am writing a series. In part 2 on Oracle Key Vault (OKV): having 2 databases using the same OKV.

In part 1 I mentioned an open item: what happens if you run this okvclient on a consolidated environment? Here are my notes.

Please mind: this is written with my little knowledge of Oracle Key Vault. It is likely that I will find out more in the (near) future and have to update this series as a consequence.

If you want to create the master key for TDE in the same virtual wallet for both databases, you can simply create a symbolic link that links the configurations together

ln -s $ORACLE_BASE/okv/saucer $ORACLE_BASE/okv/alien

Yes my databases have names that match the db_domain (area51).

The downside to this, is that you only have one endpoint and therefore both databases can read each others keys. I can imagine this being a problem if you ever decide to move your database to another server. Also there is a security risk, if one database is compromised, then the second database is automatically compromised also. So this was a no-go.

Snapshots of VMs rock, rollback and proceed with a second okvclient installation.

I created a second endpoint in OKV for a database and enrolled the endpoint. Furthermore I scp-ed the okvclient.jar file that was downloaded with enrolment and copied this to my database server.

When you run the okvclient.jar file as described in documentation and point to the same installation directory as used for the first okvclient.

java -jar okvclient_alien.jar -d /u01/app/oracle/product/12.2.0/okv -v
Oracle Key Vault endpoint software installed successfully.

For the reader that is familiar with the client an immediate problem occurs: no endpoint password is requested! Further investigation showed that only the installation logfile was updated and configuration was not changed. This means that you do not have any configuration for the new endpoint, basically you are the scenario where you share keys.

Since I don’t know how to create configuration manually, I rolled back to the snapshot again. (did I already mention that VirtualBox snapshots rock?)

So, I re-enrolled the endpoint and ran the installer again, only now pointing to a new directory

java -jar okvclient_alien.jar -d /u01/app/oracle/product/12.2.0/okv_alien -v
Detected JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/jre
Enter new Key Vault endpoint password (<enter> for auto-login):
Confirm new Key Vault endpoint password:
Oracle Key Vault endpoint software installed successfully.

This is not the situation I want, I prefer to have one set of software and multiple configurations. A very short investigation of the configuration setup suggested that there are exactly 2 differences between the two okv client installations:

1. okvclient.ora, where most prominent is the difference CONF_ID and SSL_WALLET_LOC
2. ewallet.p12, the password is the registration password and showed different keys

So, copying these files to the local configuration directory should get my desired result: 1 software tree with multiple configurations. First the setup of saucer:

rm /u01/app/oracle/okv/saucer/okvclient.ora
cp /u01/app/oracle/product/12.2.0/okv/ssl/ewallet.p12 /u01/app/oracle/okv/saucer/
cp /u01/app/oracle/product/12.2.0/okv/conf/okvclient.ora /u01/app/oracle/okv/saucer/

Turns out that if you move the files okvutil does not function anymore. Also you have to update okvclient.ora and point the SSL_WALLET_LOC to the new location (/u01/app/oracle/okv/saucer). Then check the configuration:

SQL> conn/as sysdba
Connected.
SQL> show parameter db_name

NAME				     TYPE	 VALUE
------------------------------------ ----------- ------------------------------
db_name 			     string	 saucer

SQL> select wrl_type, status from v$ENCRYPTION_WALLET;

WRL_TYPE	     STATUS
-------------------- ------------------------------
FILE		     NOT_AVAILABLE
HSM		     CLOSED

2 rows selected.

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY Welcome01;

keystore altered.

SQL> select wrl_type, status from v$ENCRYPTION_WALLET;

WRL_TYPE	     STATUS
-------------------- ------------------------------
FILE		     NOT_AVAILABLE
HSM		     OPEN

2 rows selected.

Super duper. Proceed with the second okvclient:

mkdir /u01/app/oracle/okv/alien
mv /u01/app/oracle/product/12.2.0/okv_alien/ssl/ewallet.p12 /u01/app/oracle/okv/alien/
mv /u01/app/oracle/product/12.2.0/okv_alien/conf/okvclient.ora /u01/app/oracle/okv/alien/

Now the /u01/app/oracle/product/12.2.0/okv_alien install is obsolete.

Proceed with setting the encryption key

SQL> conn/as sysdba
Connected.
SQL> show parameter db_name

NAME				     TYPE	 VALUE
------------------------------------ ----------- ------------------------------
db_name 			     string	 alien

SQL> select wrl_type, status from v$ENCRYPTION_WALLET;

WRL_TYPE	     STATUS
-------------------- ------------------------------
HSM		     CLOSED

1 row selected.

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY Welcome01;

keystore altered.

SQL>  select wrl_type, status from v$ENCRYPTION_WALLET;

WRL_TYPE	     STATUS
-------------------- ------------------------------
HSM		     OPEN_NO_MASTER_KEY

1 row selected.

SQL> ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY "Welcome01";

keystore altered.


SQL> select wrl_type, status from v$ENCRYPTION_WALLET;

WRL_TYPE	     STATUS
-------------------- ------------------------------
HSM		     OPEN

1 row selected.

Presto!. Hope this helps.

Oracle Key Vault: Migrating an Existing TDE Wallet to Oracle Key Vault

Currently I am evaluating Oracle Key Vault (OKV) by setting it up in a VirtualBox environment on my laptop. I have run into some small issues that might be specific to me (in which case this post is just a personal reminder), or it can be more generic.

My testing environment consists out of a single instance 12c database running on ASM. Before I investigated OKV I already tested with transparent database encryption and the wallet was located in ASM. Therefore the scenario described in the OKV documentation for migrating an existing TDE wallet to Oracle Key Vault applies to me.

Registration of the end-point (database) in OKV went perfectly, I was able to download a jar file and install the OKV software. The jar file writes configuration and also the OKV client software to disk. It is on my open items to investigate what happens if you register a second database on the same server, the way the software and configuration is installed makes me wonder if this will fly in a consolidated environment.

First issue I hit is the action to be performed at bullet 4. Documentation suggests to update the encryption_wallet_location in sqlnet.ora to

ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=HSM)(METHOD_DATA=(DIRECTORY=wallet_location))) 

It turns out you should leave this at the current wallet_location, in my case +DATA. This is required for the migration at step 8 to run successfully.

When you query V$ENCRYPTION_WALLET as suggested in step 6, you actually get two rows returned whereas you only had one row before you configured HSM as source method. I think the documentation could use an example there.

Since I am running on 12c, I can directly pass to step 8 and run the command

ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY "<endpoint password>" MIGRATE USING "<wallet password>" WITH BACKUP;

This took me a little longer to work on too. Turns out that you enter an when you register the endpoint. But only if you did not select auto-login. And that is exactly what I did… Only after re-enroling the endpoint I realized that I could have passed null as described in the 11R2 instruction some 2 lines above the 12c instructions. So after re-enroling and setting a password, I was able to migrate the encryption key into the OKV.

Now all that is left is opening up the keystore using the command

ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "Key_Vault_endpoint_password";

The command execute successfully given the feedback

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY Welcome01;

keystore altered.

However when I checked v$encryption_wallet it showed that the wallet was still closed:

SQL> select wrl_type, status from v$ENCRYPTION_WALLET;

WRL_TYPE	     STATUS
-------------------- ------------------------------
ASM		     CLOSED
HSM		     CLOSED

2 rows selected.

Now this was somewhat annoying. On to the alert.log This showed the following lines

kzthsmcc1: HSM heartbeat check failed to cache
object handle. Error code: 1014
HSM connection lost, closing wallet

Time to hit documentation. And it showed a clue: “Ensure that the ORACLE_BASE environment variable is set before you start the oracle process manually. This is very important.” And important it is indeed. Because without the ORACLE_BASE environment variable OKV cannot find the configuration. And that will break your connection to HSM. So I added ORACLE_BASE to the database configuration in crs:

srvctl setenv database -db saucer -env ORACLE_BASE=/u01/app/oracle

This requires a restart of the database (via srvctl!!!!) and:

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY Welcome01;

keystore altered.

SQL> select wrl_type, status from v$ENCRYPTION_WALLET;

WRL_TYPE	     STATUS
-------------------- ------------------------------
ASM		     CLOSED
HSM		     OPEN

2 rows selected.

So now there is only one problem left: the wallet in ASM still exists and currently has the current keys. OKV documentation does not describe what to do next. My suggestion would be to remove the wallet from ASM and update sqlnet:

ENCRYPTION_WALLET_LOCATION=
 (SOURCE=
   (METHOD=HSM))

Because that would leave the encryption_wallet view in the following state

SQL> select wrl_type, status from v$ENCRYPTION_WALLET;

WRL_TYPE	     STATUS
-------------------- ------------------------------
FILE		     NOT_AVAILABLE
HSM		     OPEN

2 rows selected.

Hope this helps.

On Exalogic, OTD and Multicast

Oracle Traffic Director is Oracle’s software loadbalancing product that you can use on Exalogic. When you deploy OTD on Exalogic, you can choose to configurge high availability. How this works is fully described within manuals and typically works all nice when you try this on your local testsystems (e.g. in VirtualBox). Additional quircks that you have to be aware of are described also, e.g. on Donals Forbes his blog here and here. I encourage you to read all of that.

However when deploying such a configuration I kept on running into issues with my active/passive failover groups. To describe the issue in somewhat more detail, let me first show you how a typical architecture looks. A typical setup with OTD and an application looks like the image depicted below:

There is a public network, in this case it is collored green. The public network runs on a bonded network interface, identified by 1. This is the network that your clients use to access the environment. Secondly there is an internal network that is non-routable and only available within the Exalogic. This network is collored red and is running via bonded interface identified as 2. The OTD sits in the middle and basically proxies traffic comming in on 1 and forward the traffic non-transparent for the client via interface 2 to the backend weblogic servers.

When you setup a active/passive failover group, the VIP you want to run is mounted in interface 1 (public network. Again see Donals Forbes blog for implementation again. If you create such a configuration via tadm (or in the GUI) what happens under the covers, is that keepalived is configured to use VRRP. You can find this configuration in the keepalived.conf configuration file that is stored with the instance.

This configuration looks something like this:

vrrp_instance otd-vrrp-router-1 {
        priority 250
        interface bond1
        virtual_ipaddress {
                XXX.XXX.XXX.XXX/XX
        }
        virtual_router_id 33
}

On the second OTD node you would see the same configuration, however the priority will be different. Based on priority the VIP is mounted on either one or the other OTD node.

As you can see in this configuration file, only only interface 1 is into play currently. This means that all traffic regarding OTD is send over interface 1. This is public network. The problem with this is two-fold:

1. Multicast over public network doesn’t always work
2. Sending cluster traffic over public network is a bad idea from security perspective, especially since OTD’s VRRP configuration does not require authentication

When I look at the architecture picture, I prefer to send cluster traffic over the private network (via interface 2) instead of via public. In my last endeavor the external switches didn’t allow any multicast traffic, so actually the OTD nodes weren’t able to find each other and both mounted the VIP. I found that multicast traffic was dropped by performing a tcpdump on the network interface (no multicast packets from other hosts arrived). Since tcpdump puts the network interface in a promiscuous mode, I get called by the security team after every time I perform a tcpdump. Therefore I typcally stay away from tcpdump and simply read the keepalived output in /var/log/messages when both OTD nodes are up. If you can see that one node is running as backup and one as master you are okay. Also you can see this by checking the network interfaces: if the VIP is mounted on both nodes you are in trouble.

The latter was the case for me: trouble. The VIP was mounted on both OTD nodes. This somehow did not lead to IP conflicts, however when the second OTD node was stopped the ARP table was not updated and hence traffic was not forwarded to the remaining OTD.

After a long search on Google, My Oracle Support and all kinds of other sources I almost started crying: no documentation how to configure this was to be found. Therefore I started fiddling with the configuration, just to see if I could fix this. Here’s what I found:

The directive interface in the keepalived.conf is the interface that you use for clustering communication. However you can run a VIP on every interface by adding a dev directive to the virtual_ipaddress configuration. So here’s my corrected configuration:

vrrp_instance otd-vrrp-router-1 {
#   Specify the default network interface, used for cluster traffic
    interface bond2
#   The virtual router ID must be unique to each VRRP instance that you define
    virtual_router_id 33
    priority 250
    virtual_ipaddress {
       # add dev to route traffic via a non-default interface
       XXXX.XXXX.XXXX.XXXX/XX dev bond1
    }
}

So what this does, is send all keepalived traffic (meaning: cluster traffic) via bond2, however the VIP is mounted on bond1. If you also want to introduce authentication, the directive advert_int 1 is your new best friend. Example snippet to add to keepalived.conf within the otd_vrrp-router configuration:

    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1066
    }

Hope this helps.

Configure Virtualized Exalytics Host for Enterprise Manager Cloud Control Discovery (and how to save you from a 4.3 GB download)

Today Rob Zoeteweij asked me to help discover an Exalytics in OVMM. When discovering the Exalytics in Enterprise Manager 12c an error returned on a missing /var/exalytics/info/em-context.info file.
Digging into the issue, Rob found that It seems that Configure Exalytics Host for Enterprise Manager Cloud Control Discovery gives some headache. The documentation mentions a script called exalytics_configure_for_emcc.sh , however this script cannot be found on the virtualized Exalytics (which is OVS). Given that the script was not to be found, we couldn’t be sure that other settings were missing too.

To find the script, you need to download “Oracle Exalytics X2-4/ X3-4/X4-4 Base Image 1.0.0.7 for Linux x86-64” from edelivery. Exalytics software can be found by searching on the “Oracle Business Intelligence” product and “Linux x86-64” platform. Select the link for “Oracle Exalytics Software Media Pack for Linux x86-64”

The 4.3GB download wil result in a file called V56223-01.iso . If you mount this iso on a tempory directory, e.g.

mount -o loop /var/tmp/V56223-01.iso /mnt

you can next up find an RPM that contains the exalytics_configure_for_emcc.sh script. To get the script from the rpm without installing it, you can extract the RPM to a temporary directory:

cd /var/tmp
rpm2cpio /mnt/Server/exalytics-scripts-1.0.0.7-18.noarch.rpm | cpio -idmv

The contents of this script is

[root@exalytics ]# cat /var/tmp/opt/exalytics/bin/exalytics_configure_for_emcc.sh
#!/bin/sh
#
# Copyright (c) 2013, Oracle and/or its affiliates. All rights reserved.
#

mkdir -p /var/exalytics/info/
IPMI_OUT_FILE=/var/exalytics/info/impi.out
ipmitool sunoem cli 'show /SP system_identifier'    2>&1 | tee -a $IPMI_OUT_FILE
cat $IPMI_OUT_FILE | grep "system_identifier =" > /var/exalytics/info/em-context.info
echo "Contents of /var/exalytics/info/em-context.info file:"
cat /var/exalytics/info/em-context.info
echo "Note: If the system_identifier is displayed as blank then it needs to be set through the ILOM web interface and this script needs to be rerun."
echo "Configured System Successfully"

So, as you can read the script only extracts the system identifier from ILOM and places it in a file. This means that the contents of this file should be something like:

[root@exalytics ]# cat /var/exalytics/info/em-context.info
        system_identifier = Oracle Exalytics XXXXXXXXX

Where XXXXXXXXX should be your specific system identifier.

This file is required to be available on the server where you have the Enterprise Manager agent installed that will monitor your virtualized Exalytics. Typically I would locate that on the same server as the Oracle VM Manager for Exalytics will run. The file is only required during configuration of the Exalytics in Enterprise Manager, hence it can be reused (with a different system identifier) for multiple Exalytics systems.

Secondly you have to install ipmitool on the server that has runs the monitoring agent. ipmitool can be found on the installation DVD of your Oracle Linux distribution. It is also distributed via the Exalytics .iso that you downloaded to find the Exalytics_configure_for_emcc.sh script.

After you have placed this file, you can follow the guided discovery process in Enterprise Manager for Exalytics.

Hope this helps.

Resource usage on Exalogic

For the coming  conference season I decided not to present. I have had plenty of interesting experiences, multiple could be interesting enough to present about, but working with new products requires so much energy that I rather skip for now. Right after this decision I noticed that I just can’t help myself. I like to share what I do, not only to share but also to learn (and perhaps even being told that I am wrong).

The last two years loads of my work has been around engineered systems. Last year (2014) I was involved with multiple customer as platform architect deploying all kinds of engineered systems, including a total of 11 racks Exalogic (either half or quarter) . Most of these racks were fitted with OVM (a.k.a. Exalogic Virtual), five of these were Hybrid (half “bare metal” and half OVM). There are plenty of things to say about the Exalogic product, what does and what does not work, what is missing etc. I would like to stay away from all that, not in the least because Exalogic 12c has been announced. Therefore I just share some of the tooling, scripts and notes that I wrote to support my every day work.

One of the topics that my customers keep on asking questions about is actual resource usage. Obviously tooling like Oracle Enterprise Manager can help with that, however not every customer is running OEM (yes these customers exist and have numerous reasons). So that leaves you, as consultant/administrator/local-techie with a challenge, how to find how many VMs are actually running on your rack. Especially the control stack of Exalogic does not have any features that makes output excel-friendly (apparently a must-have feature for any resource report).

OTD-62015 An error occurred while creating server certificates

<UPDATE !!!>
One of my colleagues asked for help creating an OTD configuration on an engineered system. For some reason the creation of the administration server failed. Here’s the command he issued:

-bash-3.2$ export ORACLE_HOME=/u01/app/oracle/product/otd
-bash-3.2$ export PATH=$ORACLE_HOME/bin:$PATH
-bash-3.2$ $ORACLE_HOME/bin/tadm configure-server --host=my_host --java-home=$ORACLE_HOME/jdk --port=8989 --user=admin --instance-home=/u01/app/oracle/admin/otd/otdadmin --server-user=oracle --port 8989 --verbose
This command will create the administration server. The password that is provided will be required to access the administration server.
Enter admin-user-password>
Enter admin-user-password again>
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ConfigureServer validateRuntimeUser
FINEST: Checking availability of valid runtime user...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance init
FINEST: Initing AdminServerInstance
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: Initing ServerInstance...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareDirsAndFiles
FINEST: AdminServerInstance.prepareDirsAndFiles()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareInstanceNameAndDir
FINEST: AdminServerInstance.prepareInstanceNameAndDir()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance prepareTokens
FINEST: AdminServerInstance.prepareTokens()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance prepareTokens
FINEST: ServerInstance.prepareTokens()
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: isWindows = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: oracleHome = /u01/app/oracle/product/otd
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: instanceHome = /u01/app/oracle/admin/otd/otdadmin
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: cfgTmplPath = /u01/app/oracle/product/otd/lib/templates/config
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: scriptsTmplPath = /u01/app/oracle/product/otd/lib/templates/scripts
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: configName = admin-server
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: unixUser = null
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: isZip = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance init
FINEST: createService = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance
FINEST: In AdminServerInstance constructor :: after calling super
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance
FINEST: 		 logger is null = false
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance createInstance
FINEST: Starting to create server instance...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.ServerInstance createDirectories
FINEST: Starting to create instance directory structure...
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance setupSecurityDB
FINEST: AdminServerInstance.setupSecurityDB
Jan 14, 2014 11:06:52 AM com.sun.web.admin.configurator.AdminServerInstance setupSecurityDB
FINEST: dbDir = /u01/app/oracle/admin/otd/otdadmin/admin-server/config
Jan 14, 2014 11:06:54 AM com.sun.web.admin.configurator.AdminServerInstance createAdminCerts
FINEST: Starting to setup the administration self-signed certificates
Jan 14, 2014 11:06:55 AM com.sun.web.admin.configurator.AdminServerInstance createAdminCerts
FINEST: java.lang.SecurityException: Unable to initialize security library
com.sun.web.admin.security.NSSDBException: java.lang.SecurityException: Unable to initialize security library
	at com.sun.web.admin.security.SecurityUtil.initDB(SecurityUtil.java:69)
	at com.sun.web.admin.configurator.AdminServerInstance.createAdminCerts(AdminServerInstance.java:161)
	at com.sun.web.admin.configurator.AdminServerInstance.setupSecurityDB(AdminServerInstance.java:101)
	at com.sun.web.admin.configurator.ServerInstance.createInstance(ServerInstance.java:604)
	at com.sun.web.admin.configurator.ConfigureServer.configureServer(ConfigureServer.java:111)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.configure(ConfigureServerCommand.java:93)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.configureServer(ConfigureServerCommand.java:48)
	at com.sun.web.admin.cli.commands.ConfigureServerCommand.runCommand(ConfigureServerCommand.java:29)
	at com.sun.enterprise.cli.framework.CLIMain.invokeCommand(CLIMain.java:171)
	at com.sun.web.admin.cli.shelladapter.WSadminShell.invokeFramework(WSadminShell.java:162)
	at com.sun.web.admin.cli.shelladapter.WSadminShell.main(WSadminShell.java:79)
Caused by: java.lang.SecurityException: Unable to initialize security library
	at org.mozilla.jss.CryptoManager.initializeAllNative(Native Method)
	at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:919)
	at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:885)
	at com.sun.web.admin.security.SecurityUtil.initDB(SecurityUtil.java:62)
	... 10 more

OTD-62015 An error occurred while creating server certificates: java.lang.SecurityException: Unable to initialize security library

Now this seemed interesting to me, since I never had this error before. So, fond of tracing as I am I started an strace

strace -f -o /tmp/tadm.trc $ORACLE_HOME/bin/tadm configure-server --host=my_host --java-home=$ORACLE_HOME/jdk --port=8989 --user=admin --instance-home=/u01/app/oracle/admin/otd/otdadmin --server-user=oracle --port 8989 --verbose

This gave me a rather extensive trace file (close to 12k lines) which I won’t bother you with. One of the relevant lines that draw my attention was:

fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824, len=1}) = -1 ENOLCK (No locks available)

So, it is a NFS locking issue! Checking /etc/mtab showed me that the instance home was on a NFS mount: /u01/app/oracle/admin/otd . I changed the mountoptions to include noac,nolock and this instantly solved the error.

Hope this helps.

<UPDATE>
Well, that noac option caused some severe performance issues. Seems that this database best practice doesn’t work so much on Exalogic.

The nolock option should be handled with care. If you are absolutely sure that files can only be opened from one location this could solve the issues, but I was told by experts to avoid this as much as possible. Removing the nolock option did bring me back to a crashing tadm though. Back to the drawing board….

yum exclude list for Exalogic vServers

Recently I have been doing some work on Exalogic. While building a template for vServers on Exalogic I ran into an issue. After executing yum update following by a reboot, I wasn’t able to connect to the vServers anymore. This is caused by an issue with the network stack which, in the end, is caused by an documentation error.

It seems that the yum exclude list for vServers is not correctly documented , also Oracle Support Document 1594674.1 (Exalogic Virtual Environment – Guest vServer Upgrade to Oracle Linux v5.10 ) seems to be off.  The exclusion list that didn’t break the operating system after a yum update is:

exclude=kernel* compat-dapl* dapl* ib-bonding* ibacm* ibutils* ibsim* infiniband-diags* kmod-ovmapi-uek* libibcm* libibmad* libibumad* libibverbs* libmlx4* libovmapi* librdmacm* libsdp* mpi-selector* mpitests_openmpi_gcc* mstflint* mvapich* ofa* ofed* openmpi_gcc* opensm* ovm-template-config* ovmd* perftest* qperf* rds-tools* sdpnetstat* srptools* exalogic* infinibus* xenstoreprovider* initscripts* nfs-utils*

new SOA HA paper

Today I was pointed at a brand new SOA HA paper on OTN (thanks Simon. Although I didn’t give any direct input for the paper, it discusses the architecture I designed for my largest customer. I am very happy that Oracle recognizes that customers rely on active/active configurations.

Lengthy errormessage while creating OID

For most of the Fusion Middleware domains or systems components I setup for clients I use scripts. Today, while creating a OID, I decided to type in the commands manually. This resulted in the following:

$ /u01/app/oracle/admin/instances/XXXXX/oid_instance_XXXXXXXX/bin/opmnctl  createcomponent -componentType OID -componentName area51_oid -Db_info rdbms-scan.area51.local:1521:srv4oid -Host oid.area51.local -Port 3060 -Sport 3131

Command requires login to weblogic admin server (oid.area51.local):
  Username: weblogic
  Password:

Creating empty component directories...Done
Provisioning OID files for area51_oid
  OID onCreate....

Enter ODS password:
  Validating OID input parameters
Enter ODSSM password:
oracle.as.config.ProvisionException: Error deleting credential odssm from CSF
        at oracle.iam.management.oid.install.wls.OIDComponentHelper.delCredFromCSF(OIDComponentHelper.java:2373)
        at oracle.iam.management.oid.install.wls.OIDComponentHelper.removeCreds(OIDComponentHelper.java:2345)
        at oracle.iam.management.oid.install.wls.OIDComponent.onRemove(OIDComponent.java:429)
        at oracle.as.config.impl.OracleASComponentBaseImpl.remove(OracleASComponentBaseImpl.java:287)
        at oracle.as.config.impl.OracleASComponentBaseImpl.remove(OracleASComponentBaseImpl.java:174)
        at oracle.as.config.impl.OracleASComponentBaseImpl.remove(OracleASComponentBaseImpl.java:155)
        at oracle.iam.management.oid.install.wls.OIDComponent.onCreate(OIDComponent.java:227)
        at oracle.as.config.impl.OracleASComponentBaseImpl.createComponent(OracleASComponentBaseImpl.java:597)
        at oracle.as.config.impl.OracleASComponentBaseImpl.create(OracleASComponentBaseImpl.java:106)
        at oracle.as.config.provisioner.commands.CreateComponentCommand.execute(CreateComponentCommand.java:40)
        at oracle.as.config.provisioner.InstallerCmdLine.run(InstallerCmdLine.java:146)
        at oracle.as.config.provisioner.InstallerCmdLine.main(InstallerCmdLine.java:46)
Caused by: java.security.PrivilegedActionException: oracle.as.config.ProvisionException:
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.iam.management.oid.install.wls.OIDComponentHelper.delCredFromCSF(OIDComponentHelper.java:2354)
        ... 11 more
Caused by: oracle.as.config.ProvisionException:
        at oracle.iam.management.oid.install.wls.OIDComponentHelper$3.run(OIDComponentHelper.java:2367)
        ... 13 more
Caused by: oracle.security.jps.config.JpsConfigurationException: /u01/app/oracle/admin/instances/eoid2/oid_instance_oesv9510/config/JPS/jps-config-jse.xml (No such file or directory)
        at oracle.security.jps.internal.config.xml.XmlConfigurationFactory.initDefaultConfiguration(XmlConfigurationFactory.java:439)
        at oracle.security.jps.internal.config.xml.XmlConfigurationFactory.getDefaultConfiguration(XmlConfigurationFactory.java:338)
        at oracle.security.jps.internal.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:160)
        at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.(JpsContextFactoryImpl.java:112)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:355)
        at java.lang.Class.newInstance(Class.java:308)
        at oracle.security.jps.util.JpsUtil.newInstance(JpsUtil.java:190)
        at oracle.security.jps.JpsContextFactory$1.run(JpsContextFactory.java:74)
        at oracle.security.jps.JpsContextFactory$1.run(JpsContextFactory.java:72)
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.security.jps.JpsContextFactory.getContextFactory(JpsContextFactory.java:71)
        at oracle.iam.management.oid.install.wls.OIDComponentHelper$3.run(OIDComponentHelper.java:2357)
        ... 13 more
Caused by: java.io.FileNotFoundException: /u01/app/oracle/admin/instances/eoid2/oid_instance_oesv9510/config/JPS/jps-config-jse.xml (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.(FileInputStream.java:120)
        at oracle.security.jps.internal.common.util.XmlSchemaValidationUtil.doValidation(XmlSchemaValidationUtil.java:96)
        at oracle.security.jps.internal.config.xml.XmlConfigurationFactory.initDefaultConfiguration(XmlConfigurationFactory.java:418)
        ... 28 more
    Skipping oesv9510_oid unregistration. It is not currently registered with the adminserver.
    Deleting oesv9510_oid directories
    Invoking opmn reload...Done
Command failed: Exception in onCreate()
Details are logged in /u01/app/oracle/admin/instances/eoid2/oid_instance_oesv9510/diagnostics/logs/OPMN/opmn/provision.log

opmnctl createcomponent: failed.

Now this is a rather lengthy error message and it really surprised me. Since I just associated the security store for the domain to a database, and this error pointing towards JPS-config I figured something must be wrong with the reassociateSecurityStore wlst comamnd. So I checked logfiles, My Oracle Support and Google before I checked the provision.log.

The provision log showed me these messages

SEVERE: Command failed:
oracle.as.config.ProvisionException: Exception in onCreate()
        at oracle.iam.management.oid.install.wls.OIDComponent.onCreate(OIDComponent.java:235)
        at oracle.as.config.impl.OracleASComponentBaseImpl.createComponent(OracleASComponentBaseImpl.java:597)
        at oracle.as.config.impl.OracleASComponentBaseImpl.create(OracleASComponentBaseImpl.java:106)
        at oracle.as.config.provisioner.commands.CreateComponentCommand.execute(CreateComponentCommand.java:40)
        at oracle.as.config.provisioner.InstallerCmdLine.run(InstallerCmdLine.java:146)
        at oracle.as.config.provisioner.InstallerCmdLine.main(InstallerCmdLine.java:46)
Caused by: oracle.as.config.ProvisionException: -Namespace parameter missing
        at oracle.iam.management.oid.install.wls.OIDComponentHelper.validateParams(OIDComponentHelper.java:314)
        at oracle.iam.management.oid.install.wls.OIDComponent.onCreate(OIDComponent.java:158)
        ... 5 more

Aha. So that was a enormous error message that tried to tell me “Hey, you missed the -Namespace parameter”.

P.S. 11.1.1.7 also introduced a new process called oiddispd. documentation” gives this description:
“Beginning with Oracle Internet Directory 11g Release 1 (11.1.1.7.0), the OIDLDAPD process is separated as the OIDDISPD (dispatcher) process and the OIDLDAPD (server) process. On UNIX and Linux systems, however, the ps -ef command will continue to show both of these processes as OIDLDAPD at runtime.”

If you happen to separate software from configuration like I do (binaries owned by oracle, oid processes run as some other user) you need to chown the $ORACLE_HOME/bin/oiddispd process and chmod it to 4740

Hope this helps.

Active Data Guard & Fusion Middleware Repositories.

Last year while working on a POC Rob den Braber noticed the following in Disaster Recovery for Oracle Elastic Cloud with Oracle ExaData Database Machine on page 13:

Currently, Oracle Fusion Middleware does not support configuring Oracle Active Data Guard for the database repositories that are a part of the Fusion Middleware topology. However, Active Data Guard can be configured if your custom applications are designed to leverage the technology.

Today this came up in a discussion with Simon Haslam , and he didn’t hear from this support issue before. So it seems that it is not that well know that Active Data Guard and Oracle Fusion Middleware is not a supported combination.

This makes this blog post a reminder from what is already in documentation (unless someone can comment and tell me that currently in the quote is not so currently anymore).

Hope this helps.

UPDATE:

While reading this brand new SOA HA paper I found this quote today:

The Oracle Active Data Guard Option available with Oracle Database 11g Enterprise Edition enables you to open a physical standby database for read-only access for reporting, for simple or complex queries, or sorting while Redo Apply continues to apply changes from the production database. Oracle Fusion Middleware SOA does not support Oracle Active Data Guard because the SOA components execute and update information regarding SOA composite instances in the database as soon as they are started.