WebLogic 12c & SOA 12c

The road ahead for WebLogic 12c

Before we can describe all the new features of WebLogic 12.1.3 & 12.1.4 and compare this to the 12.1.2 version we should first take a look at the 10.3.6 version. WebLogic 10.3.6 is still the latest 11g version but Oracle will support 10.3.6 till 2018 and extended support till 2021. So Oracle’s Fusion Apps and we have enough time to migrate to WebLogic 12.1.X. Oracle also promised that the upgrade should be easy.  That being said we can take look at the WebLogic 12.1.X features.

Last summer Oracle already released WebLogic 12.1.2 which has since WebLogic 12.1.1 been certified for Java EE 6 and it looks like the Java EE 7 certification is still far away, so Oracle updated the 12.1.2 version with some badly needed frameworks like WebSockets. To make the developer experience more complete Oracle added more support for Maven and it comes with a utility to synchronize a Maven repository with all the needed WebLogic libraries.

12.1.2 is also the first release, which comes with Fusion Middleware infrastructure components. For now FMW 12.1.2 contains ADF & OWSM and comes with Enterprise Manager & MDS.
WebLogic 12.1.2 replaced the BEA installer and the BSU patching utilities with the Oracle Universal Installer and the OPatch utilities for applying patches.  To make it even more easier it just comes in one taste ( no Java included or a specific Operating System installer ).  Just one big jar file for WebLogic with Coherence or one for WebLogic, Coherence and FMW.

WebLogic 12.1.2 introduced Dynamic Clusters, Elastic JMS and integrated Coherence configuration and management. Dynamic Clusters is a great feature to extend the WebLogic Cluster with Managed Servers based on a Server Template. For this we only need to change a parameter on the cluster and the new Managed Servers are distributed over the NodeManagers. To use this feature your blades need to have enough free resources to handle the extra Managed Servers.
Elastic JMS can be combined with Dynamic Clusters, this way we only need to create 1 JMS server, target this to the cluster and every Cluster node will have its own JMS server.

When we look at the announced features of WebLogic 12.1.3 and 12.1.4 we can see that Oracle continues on this road.

WebLogic 12.1.3 will be the first version for many FMW 12c products like Oracle SOA Suite 12c and probably come in one big jar. 12.1.3 & 12.1.4 will add extra features and improvements to Elastic JMS & Dynamic Clusters. Elastic JMS in 12.1.3 will support Server Migration so you can’t lose any JMS messages.
In 12.1.4, Dynamic Clusters will have support for auto-scaling based on thresholds based on user-defined metrics. WebLogic 12.1.4 will also have an API to control the Dynamic Clusters, this way we can easily program when to stop, start or remove nodes from a dynamic cluster.

WebLogic 12.1.3 will come with the JDBC 12c driver, this driver will gives us better integration and reliability between WebLogic 12c and Oracle Database 12c (Application Continuity).
When we take a look at the administration side of WebLogic we can see that the Enterprise Manager will be more important, in WebLogic 12.1.3 we can also do Application and JMS administration from the EM application. Also 12.1.3 will contain a Restful management APIs for additional monitoring, operations, Datasource and deployment support.  Plus a global OWSM policy to protect all your Web and Rest services with one server policy.

WebLogic 12.1.4 will have a new feature called Multi-Tenant Applications, this way you can define an WebLogic template for an application, so one or more customers of this application which will have its own Cluster, Managed Servers, Application and (not shared) resources plus it will support Oracle Database 12c pluggable databases.



The last part of this article will talk about the development side of WebLogic 12.1.3 and 12.1.4.  First WebLogic 12.1.3 has to be the Modern Development Platform and will focus on HTML5 development.

12.1.3 has some great new features like
• Updated version of JAX-RS and WebSockets.
• Server-Sent Events, this allows you to continuously sending updates to the browsers.
• JSON-P or JSON with Padding will solve your cross domain security problems.
• Maven improvements.
• Avatar allows you to develop and deploy server side Javascript Services without knowing Java.

WebLogic 12.1.4 will be certified for Java EE 7 and this means the WebLogic platform will get some framework updates and some new frameworks like
• Java API for JSON 1.0
• Java API for WebSocket 1.0
• Batch Applications 1.0
• Concurrency Utilities 1.0


The last question is off course when can we expect WebLogic 12.1.3 & 12.1.4.  The 12.1.3 version will probably be released in March or April and 12.1.4 will be much later.

Weblogic Dynamic Cluster configuration

This is a new feature available in Weblogic 12c (12.1.2).
Dynamic cluster contains one or more dynamic servers. It allows you to add a new server dynamically at peak loads.  This would eliminate the need of adding the managed server manually to a cluster.  The businesses can define number of servers that needs to be added dynamically anticipating the peak loads.
The concept of dynamic cluster is based on a single shared server template.
For further reading on server template please refer the below post.
Prerequisites:

1. Configured machine .

You can create the setup in two ways.

1. From Administration console
2. Using WLST

I. Using Administration console:
1. Login into the administration console  – >  Cluster  – > Click on new — > Select Dynamic Cluster from the drop down.

Weblogic Dynamic Cluster

2. Select the messaging mode and define number of servers you would need at the peak load.
You can either create a new server template or use an already existing one.

Dynamic Cluster-NoOfServers

3. Associate dynamic servers with machines.
Note: Associating dynamic servers with machines is essential if you intend to use the Node Manager and the Administration Console (or WLST) to start servers.

Weblogic Dynamic Cluster-Machine

4. Specify Listen port bindings:
You would need to define port bindings for only one dynamic server, and each subsequent server will be given an incremental port.

Weblogic Dynamic Cluster

weblogic Dynamic Cluster

Weblogic Dynamic Cluster Details

This would create a server template based on the dynamic cluster configuration, which can be reused.

Dynamic Cluster Server Template

Introducing Dynamic Clusters in WebLogic Server 12.1.2

Introducing Dynamic Clusters in WebLogic Server 12.1.2 

In WebLogic 12.1.2, we added Dynamic Clusters, which is a set of new cluster configuration options that make it really easy to initially roll out a new cluster, and then make it really easy to scale out that cluster.
With Dynamic Clusters, you define how many servers you want in the cluster and you define a server template to use as the base configuration for servers in the cluster. Machine mappings, listen addresses, and port mappings can be mapped auto-magically. You can also use the WebLogic Diagnostics Framework to drive automation around expanding a cluster based on any metric you choose.

These features rely on some new and extended MBeans and configuration objects:

 

As shown in this diagram, the Cluster MBean has a new child: DynamicServers. That MBean handles the configuration aspects of the cluster that enable the cluster to be expanded dynamically, including a link to the Server Template, machine mapping settings, Managed Server Name prefix, the number of Dynamic Servers to create, and so forth.
The number of Dynamic Servers that you define in the Dynamic Servers configuration spurs the system to create a ServerLifeCycleRuntime MBean, which is a lightweight bean that reserves a Managed Server name and provides methods for starting and stopping the servers with the Node Manager.
The Server Template is a new configuration object that can store the same configuration attributes that a Server configuration contains. A Server Template is used as the basis for other Managed Servers, both configured servers and Dynamic Servers. And when you make a change to a Server Template, all servers that are based on that template inherit the changes dynamically. That makes changing configuration much easier when you have a bunch of servers to update.
 At runtime, a cluster is still a cluster: the runtime behavior for clusters has not changed. It is only the configuration and scaling that we have made easier. And just to be clear: configured clusters are still fully available and supported, so your existing WLST scripts that you use to configure a cluster are still valid.

http://docs.oracle.com/middleware/1212/wls/CLUST/dynamic_clusters.html.

WLST script for Dynamic Cluster creation in weblogic

# This example demonstrates the WLST commands needed to create a dynamic cluster# 
(dynamic-cluster). The dynamic cluster utilizes a server template 
# dynamic-cluster-server-template. To keep this example simple, error handling# was omitted.#
connect()
edit()
startEdit()
## Create the server template for the dynamic servers and set the attributes for# the dynamic servers. Setting the cluster is not required.#
dynamicServerTemplate=cmo.createServerTemplate("dynamic-cluster-server-template")
dynamicServerTemplate.setAcceptBacklog(2000)
dynamicServerTemplate.setAutoRestart(true)
dynamicServerTemplate.setRestartMax(10)
dynamicServerTemplate.setStartupTimeout(600)
## Create the dynamic cluster and set the dynamicservers.#
dynCluster=cmo.createCluster("dynamic-cluster")
dynServers=dynCluster.getDynamicServers()
dynServers.setMaximumDynamicServerCount(10)
dynServers.setServerTemplate(dynamicServerTemplate)
## Dynamic server names will be dynamic-server-1, dynamic-server-2, ...,# dynamic-server-10.#
dynServers.setServerNamePrefix("dynamic-server-")
## Listen ports and machines assignments will be calculated. Using a round-robin# 
algorithm, servers will be assigned to machines with names that start with#dyn-machine.#
dynServers.setCalculatedMachineNames(true)dynServers.setMachineNameMatchExpression("dyn-machine*")
## activate the changes#
activate()

WebLogic Server 12c - PerDomain Node Manager Configuration Model

Let's start with giving a brief definition of Node Manager. Server instances in a WebLogic Server production environment are often distributed across multiple domains, machines, and geographic locations. Node Manager is a WebLogic Server utility that enables you to start, shut down, and restart server instances from a remote location. Node Manager is optional but it is recommended if your WebLogic Server environment hosts applications with high availability requirements. 

WebLogic Server 12c has over 200 new features. One of these features is a new Node Manager configuration model. In WebLogic 12c, the Java version of Node Manager has a simplified, out-of-the-box configuration. Node Manager is moved from PerHost configuration to PerDomain configuration model. The Install Windows Service screen, which provided the option to install Node Manager as a service on Windows systems, is no longer included.

In WebLogic 12c for each domain you create, an instance of domain-specific Node Manager is created/configured by default. With this default configuration, using the security credentials supplied for the Administration Server, nm_password.properties is created in DOMAIN_HOME\config\nodemanager. We can use Configuration Wizard or WLST tools to create Domain/Node Manager configuration. We can select a PerDomain or CustomLocationJava-based Node Manager configuration model. The PerDomain configuration is exactly as described for the default configuration, except that it allows you to provide unique Node Manager credentials. If you want a seperate location for NodeManagerHome, select CustomLocation and specify an empty directory or select to create one. The resulting Node Manager also runs as a per domain process.

After the default installation, the nodemanager.properties and nodemanager.domains files are created for you under DOMAIN_HOME\nodemanager and domain-specific scripts to start, stop, install and uninstall Node Manager as a Windows service, are created under DOMAIN_HOME\bin. With the default Node Manager configuration, you cannot edit the NodeManagerHome location, in this case it will be DOMAIN_HOME\nodemanager.

Let's see the process step by step. I will use the configuration wizard tool to create a sample domain. After starting the wizard and proceeding for the first few steps you will be prompted to choose the components to modify/create. Here you can select Node Manager option to configure some of it's properties like Node Manager type, Node Manager username and password, and Node Manager location.

On the next screen we can modify the following properties;

Node Manager Type:

Per Domain: If you select this option, the Node Manager home is predefined within the domain as <domain_name>/nodemanager and you cannot edit the Node Manager home. 

Custom Location: Select this option if you want the Node Manager configuration files to be created in a specific location for this domain. Specify the directory in the Node Manager Home field, or click Browse to navigate to the location. The specified directory must be empty. The nodemanager.properties and nodemanager.domains files will be created in this directory. 

Manual Node Manager Setup: If you select this option, creation of the Node Manager configuration for the domain is skipped, and you must manually create and update the Node Manager configuration for the domain.

Node Manager Credentials: Here you enter the username and password for this Node Manager instance.

After the wizard exits, we will have the following folder structure and the Node Manager related configuration files. 

Please remember you can still use Per Host Node Manager (scripts are located in WL_HOME\server\bin). However there is extra step you must take to use this model. Below is the summary of this steps;

1. Create a nodemanager.domains file that specifies the domains that you want this Node Manager instance to control, under      ORACLE_HOME\oracle_common\common\nodemanager, the per host NodeManagerHome location.

2. Generate a certificate and private key to communicate with Node manager over SSL.

Analysing GC log using GC Viewer

This article is about analysing GC log using GC Viewer.
We can download the software from this link
http://www.tagtraum.com/gcviewer-download.html
In order to capture the GC information we need to start Weblogic Server by adding the following JAVA Option
set MEM_ARGS=-Xms1024m -Xmx1024m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:gc.log
If you see Out Of Memory errors in you server logs, you should be ready with GC Viewer to anylyze the gc log file.
Out Of Memory occurs due to the exhaustion of the heap. This can be due to an overload condition or programming negligence.If its due to an overload condition, we need to increase the heap size depending on the load. If its due to programming errors, we need to take a deeper look. Take heapdumps and analyze using Memory Analyzer tools.
This has been described in the following article
TROUBLESHOOTING OUT OF MEMORY ERRORS IN WEBLOGIC USING ECLIPSE MEMORY ANANLYSER
There are two types of garbage collection, major and minor. In minor garbage collection only young generation is cleared where as major garbage collection cleans the tenured and the young generation.
e.g.
[GC [PSYoungGen: 40951K->3360K(42432K)] 564334K->526886K(566720K), 0.0720524 secs]
2274304.031: [GC [PSYoungGen: 41632K->3584K(42496K)] 565158K->527274K(566784K), 0.0665242 secs]
2274663.654: [GC [PSYoungGen: 41920K->3296K(42560K)] 565610K->527134K(566848K), 0.0636812 secs]
2275023.726: [GC [PSYoungGen: 41696K->3746K(42560K)] 565534K->527728K(566848K), 0.0642125 secs]
2275428.250: [GC [PSYoungGen: 42210K->3424K(42752K)] 566192K->527574K(567040K), 0.0637819 secs]
2275428.314: [Full GC [PSYoungGen: 3424K->2828K(42752K)] [PSOldGen: 524150K->524287K(524288K)] 527574K->527116K(567040K) [PSPermGen: 88758K->88758K(90112K)], 11.1667328 secs]
2275743.907: [Full GC [PSYoungGen: 38528K->5936K(42752K)] [PSOldGen: 524287K->524287K(524288K)] 562815K->530224K(567040K) [PSPermGen: 88891K->88891K(90112K)], 11.0188252 secs]
2276067.615: [Full GC [PSYoungGen: 38528K->5349K(42752K)] [PSOldGen: 524287K->524287K(524288K)] 562815K->529637K(567040K) [PSPermGen: 88956K->88956K(90112K)], 11.0896643 secs]
2276404.096: [Full GC [PSYoungGen: 38420K->0K(42752K)] [PSOldGen: 524287K->517865K(524288K)] 562707K->517865K(567040K) [PSPermGen: 88987K->88611K(90112K)], 14.6099121 secs]
2276764.208: [GC [PSYoungGen: 38350K->1890K(42752K)] 556215K->519755K(567040K), 0.0890990 secs]
2277126.468: [GC [PSYoungGen: 40482K->1932K(40640K)] 558347K->519797K(564928K), 0.0915124 secs]
Line starting with GC is minor GC and the line starting with Full GC is major GC.
If we take a closer look at a single Full GC
2275428.314: [Full GC [PSYoungGen: 3424K->2828K(42752K)] [PSOldGen: 524150K->524287K(524288K)] 527574K->527116K(567040K) [PSPermGen: 88758K->88758K(90112K)], 11.1667328 secs]
Heap in use in young generation befor GC is 3424K and after is 2828K, whereas the total heap is 42752K.
Similarly for tenured generation heap usage before GC is 88758K and after GC it is 88758K, and the time taken is 11.1667328 secs. The timestamp helps us keep an eye on the frequency of Garbage collection. From the frequency we can deduce whether its a leak or not.
If we see FUll GC happening continuously, its a leak and you can expect an Out Of Memory.
4364435.019: [Full GC [PSYoungGen: 43776K->42364K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->566652K(572544K) [PSPermGen: 88381K->88381K(90112K)], 12.2471219 secs]
4371857.025: [Full GC [PSYoungGen: 43776K->43776K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->568064K(572544K) [PSPermGen: 88381K->88381K(90112K)], 12.5911727 secs]
4371869.617: [Full GC [PSYoungGen: 43776K->42364K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->566652K(572544K) [PSPermGen: 88381K->88381K(90112K)], 12.9294566 secs]
4379280.165: [Full GC [PSYoungGen: 43776K->43776K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->568064K(572544K) [PSPermGen: 88384K->88384K(90112K)], 13.1517479 secs]
4379293.318: [Full GC [PSYoungGen: 43776K->42364K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->566652K(572544K) [PSPermGen: 88384K->88384K(90112K)], 13.0984445 secs]
4386653.185: [Full GC [PSYoungGen: 43776K->43776K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->568064K(572544K) [PSPermGen: 88384K->88384K(90112K)], 13.1889873 secs]
4386666.375: [Full GC [PSYoungGen: 43776K->42364K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->566652K(572544K) [PSPermGen: 88384K->88384K(90112K)], 13.3019379 secs]
If the heap usage keeps on increasing with time even after FUll GC,it is a memory leak.
In GCViewer it should look something like this

The other thing that we look for is Throughput.(Click the summary tab)
It should ideally be above 97%.

WebLogic Server: "TOO MANY OPEN FILES" Exception

WebLogic Server: "TOO MANY OPEN FILES" Exception

In Solaris and Linux, the file descriptors parameter specifies the number of open files permitted per process. If this value is too low or your application requires too many files to be opened or there is a huge number of concurrent users then WebLogic may throw the "TOO MANY OPEN FILES" exception. Note that on UNIX systems, each socket connection to WebLogic Server consumes a file descriptor.

…
Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(ProcessImpl.java:65)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
    ... 40 more

If you face this error, then you must increase the limit beyond the current value for the WebLogic. We must also ensure that the operating system is able to handle the increase in the number of open files.

On OS level we can use the following to check or increase the file descriptor limit;

$ ps –ef |grep java                     (Get  the WebLogic process id)
$ pfiles processId | grep rlimit        (Display the current file descriptor limit)
$ ulimit -H -n                             (Display hard limit, cannot be exceeded)
$ ulimit -S -n                             (Display soft limit may be increased up to hard limit value)
$ ulimit -S -n 4096                    (Increase the soft limit to 4096)

By default, WebLogic Server configures 1024 file descriptors. WebLogic limits the number of open file descriptors by checking the OS limit in a shell script. You can configure WebLogic specific number fromWEBLOGIC_HOME/common/bin/commEnv.sh script (note that this script is used on each domain created, if you want the domain specific limits you can useDOMAIN_HOME/bin/setDomainEnv.sh). 

Note that when you tune the number of file descriptors for WebLogic Server, your changes should be in balance with any changes made to the complete message timeout parameter. A higher complete message timeout setting results in a socket not closing until the message timeout occurs, which therefore results in a longer hold on the file descriptor. So if the complete message timeout setting is high, the file descriptor limit should also be set high