Resource usage on Exalogic

For the coming  conference season I decided not to present. I have had plenty of interesting experiences, multiple could be interesting enough to present about, but working with new products requires so much energy that I rather skip for now. Right after this decision I noticed that I just can’t help myself. I like to share what I do, not only to share but also to learn (and perhaps even being told that I am wrong).

The last two years loads of my work has been around engineered systems. Last year (2014) I was involved with multiple customer as platform architect deploying all kinds of engineered systems, including a total of 11 racks Exalogic (either half or quarter) . Most of these racks were fitted with OVM (a.k.a. Exalogic Virtual), five of these were Hybrid (half “bare metal” and half OVM). There are plenty of things to say about the Exalogic product, what does and what does not work, what is missing etc. I would like to stay away from all that, not in the least because Exalogic 12c has been announced. Therefore I just share some of the tooling, scripts and notes that I wrote to support my every day work.

One of the topics that my customers keep on asking questions about is actual resource usage. Obviously tooling like Oracle Enterprise Manager can help with that, however not every customer is running OEM (yes these customers exist and have numerous reasons). So that leaves you, as consultant/administrator/local-techie with a challenge, how to find how many VMs are actually running on your rack. Especially the control stack of Exalogic does not have any features that makes output excel-friendly (apparently a must-have feature for any resource report).

So, as always: command-line tooling to the rescue 🙂 To create a resource report, all you need are the following tools. These tools are installed by default on your Exalogic:

• First of all there is dcli. This distributed command line interface (duh) utility helps you running multiple commands of a cluster of machines
•  xm; the Oracle VM Server management command-line management tool xm, creates, destroys, manages and migrates guests. Now typically documentation and My Oracle Support tells you to stay away from this tool, but it can be very helpful when looking for information or debugging issues. In this case I am using the list option, which displays information on all the running domains

I guess you already know where I am going: send a command with dcli across all compute nodes calling xm list to retrieve information on hosts. Sounds like a simple idea, doesn’t it? Typically I have files with lists of compute nodes (or iloms, or IB gateways) available on every Exalogic I am involved in, however creating such a list is somewhat tedious and it also doesn’t match my idea of having flexible scripts. The information required, all compute nodes, is already available if the Exalogic ECU files are still available on the rack. I recommend keeping the ECU files on your first compute node, it turns out that all kinds of tooling depends on it. Hence it is safe to let my “tool” depend on it too 🙂

The information about compute nodes can be found in the ops_center.properties file of ECU that is located in  /var/tmp/exalogic/ecu . The second piece of information you are looking for, is the actual name of the VM (not that idiotic UUID that you cannot remember), the amount of vCPU and RAM that has been used. This information is stored in EMOC, but also in the vm.cfg files that are stored in the VM repository.

Putting all this together, you end up with a script:

#!/bin/bash
#

if [ $(grep VM /etc/oracle-release | wc -l) == 0 ]; then
        echo "This script must be run from a compute node"
        exit 1
elif [ ! -f /var/tmp/exalogic/ecu/ops_center.properties ]; then
        echo "Cannot find ops_center.properties, ECU files are missing"
        exit 1
fi

RamSubTotal=0
CpuSubTotal=0

echo "UUID                             - Name "

for CN in $(grep ecu_server_oshostname /var/tmp/exalogic/ecu/ops_center.properties | cut -d \= -f 2); do
        for i in $(/opt/exalogic.tools/tools/dcli -c $CN xm list |egrep -v "Name|Domain"|awk ' { print $1"_"$2 } '); do
                UUID=$(echo $i | awk -F_ ' { print $2 }')
                VM=$(grep ExalogicRepo /etc/mtab | cut -d \  -f 2)/VirtualMachines/$UUID/vm.cfg
                Name=$(grep OVM_simple_name $VM |awk -F= ' { print $NF }')
                Memory=$(grep memory $VM |awk -F= ' { print $NF } ')
                vCPUs=$(grep ^vcpus $VM |awk -F= ' { print $NF } ')
                paddedName=$(echo $Name | sed -e :a -e 's/^.\{1,30\}$/& /;ta')
                paddedMemory=$(echo $Memory | sed -e :a -e 's/^.\{1,6\}$/& /;ta')
                paddedCpus=$(echo $vCPUs | sed -e :a -e 's/^.\{1,3\}$/& /;ta')
                echo "$UUID - $paddedName - RAM: ${paddedMemory} - vCPUs: ${paddedCpus}"

                RamSubTotal=$(( $RamSubTotal+ $Memory))
                CpuSubTotal=$(( $CpuSubTotal+ $vCPUs))
        done
done

paddedRamTotal=$(echo $RamSubTotal | sed -e :a -e 's/^.\{1,6\}$/& /;ta')
paddedCpuTotal=$(echo $CpuSubTotal | sed -e :a -e 's/^.\{1,3\}$/& /;ta')

echo "                                                                          ------- -        ----"
echo "                                                             Total - RAM: $paddedRamTotal - vCPUs: $paddedCpuTotal"

Output would look like this (obviously I messed a little with the names of the VMs, but you’ll get the point):

UUID                             - Name
0004fb0000060000211a6457d0b15805 - 'pbla1ovm52'                    - RAM: 12288   - vCPUs: 2
0004fb00000600004a930a14367a2e11 - 'ExalogicControlOpsCenterPC1'   - RAM: 4096    - vCPUs: 2
0004fb00000600008d1740a2367eb14a - 'ExalogicControl'               - RAM: 32768   - vCPUs: 12
0004fb0000060000823f6f75f8b77ff2 - 'pbla1ovm52'                    - RAM: 24576   - vCPUs: 4
0004fb0000060000bf6b7229153cbf55 - 'pbla1ovm53'                    - RAM: 16384   - vCPUs: 4
0004fb0000060000d78210f47656feae - 'ExalogicControlOpsCenterPC2'   - RAM: 4096    - vCPUs: 2
0004fb0000060000f19eb069b9e6574a - 'pbla1ovm58'                    - RAM: 16384   - vCPUs: 4
0004fb000006000049eb61d14523f160 - 'pbla1kvm51'                    - RAM: 8192    - vCPUs: 2
0004fb00000600005e0323bd9667fb05 - 'pbla1ovm56'                    - RAM: 16384   - vCPUs: 4
0004fb000006000093c95c7bf46dbd48 - 'pbla1ovm51'                    - RAM: 16384   - vCPUs: 4
0004fb0000060000f078f5bf642120af - 'pbla1ovm51'                    - RAM: 16384   - vCPUs: 4
0004fb00000600000e36c64d1f18f9a0 - 'pbla1ovm54'                    - RAM: 16384   - vCPUs: 2
0004fb00000600006c95d03f754a1d42 - 'pbla1ovm51'                    - RAM: 73728   - vCPUs: 4
0004fb00000600007b5d46a3b5a9c35b - 'pbla1u52'                      - RAM: 8192    - vCPUs: 2
0004fb0000060000337109fadc444e96 - 'exalytics-ovmm'                - RAM: 8192    - vCPUs: 2
0004fb00000600004f9be464b2eb4579 - 'pbla17vm54'                    - RAM: 16384   - vCPUs: 4
0004fb0000060000697302cfa3fba726 - 'pbla1ovm51'                    - RAM: 24576   - vCPUs: 4
0004fb00000600008691936f196d4b84 - 'pbla1ovm51'                    - RAM: 12288   - vCPUs: 2
0004fb0000060000a2483741921d2956 - 'pbla1u53'                      - RAM: 8192    - vCPUs: 2
0004fb00000600005bfff563bcd19d6c - 'pbla1ovm57'                    - RAM: 16384   - vCPUs: 4
0004fb000006000064e3f24d60cc3b2a - 'pbla1ovm52'                    - RAM: 73728   - vCPUs: 4
0004fb0000060000615f170967f16ee7 - 'pbla1ovm53'                    - RAM: 16384   - vCPUs: 2
0004fb00000600008051c866128466f5 - 'pbla1ovm52'                    - RAM: 16384   - vCPUs: 4
0004fb0000060000984f9519b61d19c2 - 'pfgs1bla12'                    - RAM: 16384   - vCPUs: 4
0004fb0000060000afc660ba67e9be8d - 'pfbla1vm52'                    - RAM: 8192    - vCPUs: 2
0004fb0000060000db9c37c5c4d5706c - 'pbla151'                       - RAM: 8192    - vCPUs: 2
0004fb00000600004824e220a9726fb4 - 'pfgs9bla152'                   - RAM: 24576   - vCPUs: 4
0004fb0000060000779916d56d4565df - 'pfbla11235'                    - RAM: 16384   - vCPUs: 4
0004fb0000060000d60eb931d38fa0ed - 'pbla11vm51'                    - RAM: 24576   - vCPUs: 4
                                                                          ------- -        ----
                                                             Total - RAM: 557056  - vCPUs: 100

Hope this helps. And as always folks, don’t trust me but test!