What happens when you change WebLogic configuration
This post is a little bit a brain dump. It is based on some experiences while changing configuration for WebLogic. Most of the post is a copy of an email I send to a friend that was struggling with unexpected configuration changes (changes that got reversed all of a sudden, etc.). All statements are based on investigation, I did not decompile and read WLS source code. If I am wrong with some statement, please let me know by commenting.
First of let me state this: You are running in production mode, right? Please always do. Fix setDomainEnv.sh and set production_mode=true (instead of production_mode= blank), that changes some memory settings for your JVM and changes some settings. Most relevant in my opinion are if you are running SUN -client is for development mode and -server is for production mode. Also auto deployment is disabled.
If you are running production mode you have to lock the configuration. A lockfile is created in the DOMAIN_HOME (edit.lock). The file contains the (encrypted) username of the user holding the lock, the time when the lock was acquired etc. Whenever you make some change to the configuration in subdirectory “pending” in your DOMAIN_HOME you can find the new configuration files. All configuration files that are changed by the edit are temporarily stored there. The current configuration is only overwritten when you click on activate.
When you activate the configuration, the console application sends a request to all the running servers in the domain to overwrite the configuration in DOMAIN_HOME/config (config.xml, plus possibly some extra configuration files e.g. an xml describing a jdbc datasource). Overwriting of the configuration files on the managed servers happens sequentially, the order used for this config push is the order of managed servers in your config.xml . I think this is why the adminserver is always first (that is always the first managed server in config.xml)
Whenever your datasource is not created successfully on any running node of the domain, the complete configuration change is rolled back. One of the reasons that an update in the domain can fail is file locking. This can happen if more than one java server share a domain home, e.g. the adminserver and managed server 1, or because you setup the domain home on shared storage (and therefore all managed servers share the same directory with config files) . If for some reason the config change did succeed on the administration server, but not on a running managed server (possibly because you followed some enterprise deployment guide that guided you into setting up a separate administration server), the console will not give an error. You can find errors in the log files though, usually standard out.
If your managed server is running in managed server independent mode (MSI) it will not receive any configuration changes. For the configuration update process the managed server in MSI mode is considered down. Typically a managed server starts running in MSI mode when your administration server was down during startup of the managed server.
When a managed server (or multiple) are not running when the configuration is changed, the configuration is pushed upon startup time of the managed server. Typically you start the managed server from the console. The console application sends a request to the nodemanager running on the machine where you want to start the managed server. If you start a managed server from the console, in DOMAIN_HOME/servers/SERVER_NAME/data/nodemanager a file called startup.properties is created (if not already existing. Otherwise will be overwritten, unless you made manual changes in that file before). One of the entries in that property file is the reference to the administration server.
Whenever you start a managed server from the console, the admin url setting is checked and/or set. If you start a managed server from node manager directly (e.g. with wlst) the value is considered as “truth”. Obviously you have configured startScriptEnabled=true in nodemanager.properties. This startScriptEnabled, in combination with the name of the managed server, causes nodemanager to call the shellscript startWebLogic.sh with parameters machine_name and adminserver_url. (again: the url is grabbed from the startup.properties file I just mentioned).
As soon as a managed server boots, it calls the adminserver url and checks for configuration changes. If configuration changes exist, the managed server gets the new config.xml (and other configuration changes) and uses those to startup the managed server. If you start multiple managed servers at once, and they share the same disk for configuration (shared storage) this can also cause unsuccessful changes (again: file locking). When using shared storage you should always start 1 managed server first, than you can start the rest in a group (all together). This is because the first will change the configuration, the servers that will be started later will find out that they already have access to the “latest” configuration file.
This also explains why you shouldn’t hack in configuration files on your managed servers: these get overwritten when you reboot. But, and this is the tricky part, some files on the managed servers get updated when you start the managed server the first time. A typical example is system-jazn-data.xml and cwallet.sso. You can prevent copying those buggers around by sticking them in LDAP (OID) or database (reassociateSecureityStore). Which of these options (OID or RDBMS) is valid depends on the middleware product and version you are using.
So far for my brain dump. Hope this helps. Should I put this in a presentation for some conference?