Oracle MVA

Tales from a Jack of all trades

What happens when you change WebLogic configuration

with 4 comments

This post is a little bit a brain dump. It is based on some experiences while changing configuration for WebLogic. Most of the post is a copy of an email I send to a friend that was struggling with unexpected configuration changes (changes that got reversed all of a sudden, etc.). All statements are based on investigation, I did not decompile and read WLS source code. If I am wrong with some statement, please let me know by commenting.

First of let me state this: You are running in production mode, right? Please always do. Fix and set production_mode=true (instead of production_mode= blank), that changes some memory settings for your JVM and changes some settings. Most relevant in my opinion are if you are running SUN -client is for development mode and -server is for production mode. Also auto deployment is disabled.

If you are running production mode you have to lock the configuration. A lockfile is created in the DOMAIN_HOME (edit.lock). The file contains the (encrypted) username of the user holding the lock, the time when the lock was acquired etc. Whenever you make some change to the configuration in subdirectory “pending” in your DOMAIN_HOME you can find the new configuration files. All configuration files that are changed by the edit are temporarily stored there. The current configuration is only overwritten when you click on activate.

When you activate the configuration, the console application sends a request to all the running servers in the domain to overwrite the configuration in DOMAIN_HOME/config (config.xml, plus possibly some extra configuration files e.g. an xml describing a jdbc datasource). Overwriting of the configuration files on the managed servers happens sequentially, the order used for this config push is the order of managed servers in your config.xml . I think this is why the adminserver is always first (that is always the first managed server in config.xml)

Whenever your datasource is not created successfully on any running node of the domain, the complete configuration change is rolled back. One of the reasons that an update in the domain can fail is file locking. This can happen if more than one java server share a domain home, e.g. the adminserver and managed server 1, or because you setup the domain home on shared storage (and therefore all managed servers share the same directory with config files) . If for some reason the config change did succeed on the administration server, but not on a running managed server (possibly because you followed some enterprise deployment guide that guided you into setting up a separate administration server), the console will not give an error. You can find errors in the log files though, usually standard out.

If your managed server is running in managed server independent mode (MSI) it will not receive any configuration changes. For the configuration update process the managed server in MSI mode is considered down. Typically a managed server starts running in MSI mode when your administration server was down during startup of the managed server.

When a managed server (or multiple) are not running when the configuration is changed, the configuration is pushed upon startup time of the managed server. Typically you start the managed server from the console. The console application sends a request to the nodemanager running on the machine where you want to start the managed server. If you start a managed server from the console, in DOMAIN_HOME/servers/SERVER_NAME/data/nodemanager a file called is created (if not already existing. Otherwise will be overwritten, unless you made manual changes in that file before). One of the entries in that property file is the reference to the administration server.

Whenever you start a managed server from the console, the admin url setting is checked and/or set. If you start a managed server from node manager directly (e.g. with wlst) the value is considered as “truth”. Obviously you have configured startScriptEnabled=true in This startScriptEnabled, in combination with the name of the managed server, causes nodemanager to call the shellscript with parameters machine_name and adminserver_url. (again: the url is grabbed from the file I just mentioned).

As soon as a managed server boots, it calls the adminserver url and checks for configuration changes. If configuration changes exist, the managed server gets the new config.xml (and other configuration changes) and uses those to startup the managed server. If you start multiple managed servers at once, and they share the same disk for configuration (shared storage) this can also cause unsuccessful changes (again: file locking). When using shared storage you should always start 1 managed server first, than you can start the rest in a group (all together). This is because the first will change the configuration, the servers that will be started later will find out that they already have access to the “latest” configuration file.

This also explains why you shouldn’t hack in configuration files on your managed servers: these get overwritten when you reboot. But, and this is the tricky part, some files on the managed servers get updated when you start the managed server the first time. A typical example is system-jazn-data.xml and cwallet.sso. You can prevent copying those buggers around by sticking them in LDAP (OID) or database (reassociateSecureityStore). Which of these options (OID or RDBMS) is valid depends on the middleware product and version you are using.

So far for my brain dump. Hope this helps. Should I put this in a presentation for some conference?


Written by Jacco H. Landlust

March 18, 2013 at 10:27 pm

Posted in Weblogic

4 Responses

Subscribe to comments with RSS.

  1. Yes, that sounds to me like the basis of an interesting presentation (including locking, version management and auditing of domain changes). And no, I don’t understand why setting the OPSS security store in the database is not an option (and probably default) in the Configuration Wizard for most layered products (how many people only have one managed server?).

    Simon Haslam

    March 18, 2013 at 10:40 pm

    • Since you have to license your administration server you might find that most shops tend to setup their administration server on the same box (sharing the same volume) as their managed server. So possibly not that many people have to get into the reassociating business?

      BTW: I really feel that Oracle should stop charging the full monty for the administration server if it is running on different cores than the managed servers that are doing the production load.

      Jacco H. Landlust

      March 18, 2013 at 10:44 pm

      • Maybe but that’s not HA if you only have one server (unless you use a virtualisation layer to handle hardware failure but then if the hypervisor is not OVM you get into the licensing quagmire)… I may be biased but only tend to see HA customers for Fusion Middleware products.

        100% agree about not charging for Admin Server (assuming you’ve got licensed managed servers and you’re not sneakily running apps on the Admin Server itself of course!) – it’s an administration tool, like the OEM OMS for example.

        Simon Haslam

        March 19, 2013 at 12:22 am

  2. […] What happens when you change WebLogic configuration […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: