Threads, ThreadDump in WebLogic - TroubleShooting

Every WebLogic server is one process on the Operating System. Each Java process is going to have multiple Java Threads or Native Threads. The Native threads usage will be optimizing input and output operations between CPU and the system resources.

In many WebLogic/FMW admin interviews people asks the following

Difference between Server Hang, server Crash

In case of Server Hang the incoming requests keep on coming. Where as Server cannot accept the requests from the clients.

What is the meaning of WebLogic server hang?

Potential Causes of Server Hang

• RMI, RJVM responses – all threads tied up waiting for RJVM, RMI responses
  Application Deadlock 
• Threads  are all used up, none available for new work.Garbage Collection taking too much time
• Long Running JDBC calls or JDBC deadlocks lead to a hang
• JVM hang  during code optimization, may look like server hang
• JSP compilation causes server hang under heavy load
• JVM bugs 

Why does the WebLogic Server crash occurs?

When a WebLogic Server crash (or JVM crash, machine crash) an application gets a binary core file produced. The server crashes due to some invalid native code (machine specific code)
Troubleshooting Steps:

• The WebLogic Server performance pack 
• Any native libraries accessed with JNI calls 
• The JVM itself JVM will produce a small log file hs_err_pid.log 
• If there is a core file produced, run dbx or gdb debuggers on the core file, which will let us know what was the last thread processing

How to take the ThreadDump in WebLogic environment?

Using command line thread dump with kill command with -3 option will take the snapshot of what threads doing in a WebLogic server.

Install and Configure a mail service on Ubuntu

Install ssmtp

Step 1: We must use super previllages then Install ssmtp:

sudo apt-get install ssmtp

Step 2: Edit the ssmtp config file : vi /etc/ssmtp/ssmtp.conf Enter this inside the file:

root=username@gmail.com
mailhub=smtp.gmail.com:465
rewriteDomain=gmail.com
AuthUser=username
AuthPass=password
FromLineOverride=YES
UseTLS=YES

Step 3: Enter the email address of the person who will receive your email: Now enter this:

ssmtp recepient_name@gmail.com

To: recipient_name@gmail.com
From: username@gmail.com
Subject: Sent from a terminal!

Your content goes here. Lorem ipsum dolor sit amet, consectetur adipisicing.
(Notice the blank space between the subject and the body.)

To send the email: Ctrl + D

Send mail from ssmtp in Ubuntu box

SSMTP Error:
 Here I got the Authorization failed error

To resolve it I've used the following fix:
Log into your google email account and then go to this link: https://www.google.com/settings/security/lesssecureapps and set "Access for less secure apps" to ON. Test to see if your issue is resolved. Got It!! it works for me...

Apache HTTP Server proxy configuration with WebLogic

There are many HTTP servers which serves web requests. Apache HTTP server is one of the most widely used webserver in the IT. Different vendors providing HTTP servers but all of them originated from Apache web server only. Due to open source many contributors expanded it with more capabilities.

Apache HTTP server Proxy with WebLogic 12c

Architecture overview

Apache Proxy to WebLogic cluster

Download Apache HTTP Server

Select stable version of Web server which will give minimum guaranty to work as expected.

Install Apache HTTP Server

tar -xvf httpd-2.2.27.tar.gz
cd httpd-2.2.27/
./configure --prefix=/home/oracle/apache2.27

Apache HTTP Server Configuration Successful

Compile and Install the Apache binaries

Actually Apache HTTP server uses native C, C++ libraries to compile and the compilation utility is 'make' command. You need to run the 'make' command in the same same path where configure was ran.

make

Linking the LD_LIBRARY to HTTP Server ready to run this we can do with 'make install ' command.

make install

Apache Configuration

The HTTP server configuration is in the conf folder. It contains various sum directories. Once the installation completed you can define environment variable in the profile as :

export APACHE_HOME=/home/oracle/apache2.27
vi $APACHE_HOME/conf/httpd.conf

Apache Control

Start the Apache Web server

cd $APACHE_HOME/bin
 ./apachectl start 
 

Stop the Apache Web server

cd $APACHE_HOME/bin
 ./apachectl start 
 

Check the Apache HTTP Server working

Check Apache HTTP Server configuration Listen Port issue

Apache HTTP Server start failed when the port 80 used from non root user. 80 is reserved port in Linux OS for the web server that run as administrator.

make_sock error Fix: To over come you can use different port e.g: 8080. Modify this Listen port value in the httpd.conf file and start the web server. 

It Works!

Listen 8080

Confirmation of HTTP server accessability Open a web browser and test the following URL:

http://192.168.33.100:8080

Validation of Web Server access

Integrate Weblogic with Apache

Configure Apache HTTP Server Proxy plugin for WebLogic Server


Post 10g weblogic stopped providing the plugins along with the installation, you have to download them externally. You can download the web server plugins from the below location


http://www.oracle.com/technetwork/middleware/webtier/downloads/index-jsp-156711.html

or

Oracle - downloads - middleware - webtier - Accept the license agreement - plugin you want - Download

Once you download the plugins you will find the following files after extraction as shown below.

I am selecting Linux 64 as I am configuring it on Linux 64 bit machine, extract the Linux ZIP and go to the lib folder, you will find a plugin called mod_wl.so under lib as shown below.

copy the mod_wl.so plugin from the above location to $APACHE_HOME/modules

NOTE – The mod_wl.so file differs for different apache versions. Make sure that the name of the file you copied to modules is same in httpd.conf.(sometimes you might prefix it with
Now edit the httpd.conf file present under /home/vagrant/apache2.2/conf/httpd.conf

Add the below lines to the httpd.conf file and save it integrate your weblogic cluster with apache.

LoadModule weblogic_module /home/vagrant/apache2.2/modules/mod_wl.so


SetHandler weblogic-handler



WebLogicCluster localhost:7002,localhost:7003
Debug ON
WLLogFile /tmp/wlproxy.log
WLTempDir /tmp

You can replace the cluster with individual host, by replacing weblogicCluster with weblogic Host as shown below.

  WebLogicHost 192.168.33.100
  WebLogicPort 7001
  Debug ALL
  DebugConfigInfo ON
  WLLogFile /tmp/wl-proxy.log

Restart the apache to get the changed reflected.

go to /home/vagrant/apache2.2/bin and run the below command.

./apachectl restart

Deploy any web application that is .war/ear file to the weblogic cluster

Application deployment on WebLogic  Server

Test it by accessing thru Individual manged servers as shown below.

Application accessing with WebLogic server port

Now try accessing it thru Apache using 8080 port as shown below.

Application accessing with Apache Server port

  WebLogicHost 192.168.33.100
  WebLogicPort 7001
  Debug ALL
  DebugConfigInfo ON
  WLLogFile /tmp/wl-proxy.log

Restart the apache to get the changed reflected.

go to /home/vagrant/apache2.2/bin and run the below command.

./
./apachectl restart

Deploy any web application that is .war/ear file to the weblogic cluster

Application deployment on WebLogic  Server

Test it by accessing thru Individual manged servers as shown below.

Application accessing with WebLogic server port

Now try accessing it thru Apache using 8080 port as shown below.

Application accessing with Apache Server port

JMX Monitoring for WebLogic domain

Server State using JMX Program

Java Management Extension (JMX) programs are going to use three levels.

1. Probe level
2. Agent level
3. Remote Connection level

 In the probe level we have Configuration MBean, Runtime MBean.

Every Managed server, Admin Server have the Agent level - MBean Server which connects with probe level

JMX Architectuer WebLogic Domain MBeans

Remote Connection level is the top level where RMI, HTTP, SOAP, SNMP protocols will help to provide monitoring capabilities.

Sample PrintServerState.java

The following program will illustrates about how to use the JMX for huge WebLogic Environments where number or managed servers in 100s or 1000s.

import java.io.IOException;
import java.net.MalformedURLException;
import java.util.Hashtable;
import javax.management.MBeanServerConnection;
import javax.management.MalformedObjectNameException;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.naming.Context;

public class PrintServerState {

   private static MBeanServerConnection connection;
   private static JMXConnector connector;
   private static final ObjectName service;

   // Initializing the object name for DomainRuntimeServiceMBean
   // so it can be used throughout the class.
   static {
      try {
         service = new ObjectName(
            "com.bea:Name=DomainRuntimeService,Type=weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean");
      }catch (MalformedObjectNameException e) {
         throw new AssertionError(e.getMessage());
      }
   }

   /*
   * Initialize connection to the Domain Runtime MBean Server
   */
   public static void initConnection(String hostname, String portString, 
      String username, String password) throws IOException,
      MalformedURLException { 
      String protocol = "t3";
      Integer portInteger = Integer.valueOf(portString);
      int port = portInteger.intValue();
      String jndiroot = "/jndi/";
      String mserver = "weblogic.management.mbeanservers.domainruntime";
      JMXServiceURL serviceURL = new JMXServiceURL(protocol, hostname,
         port, jndiroot + mserver);
      Hashtable h = new Hashtable();
      h.put(Context.SECURITY_PRINCIPAL, username);
      h.put(Context.SECURITY_CREDENTIALS, password);
      h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES,
         "weblogic.management.remote");
      connector = JMXConnectorFactory.connect(serviceURL, h);
      connection = connector.getMBeanServerConnection();
   }

   /* 
   * Print an array of ServerRuntimeMBeans.
   * This MBean is the root of the runtime MBean hierarchy, and
   * each server in the domain hosts its own instance.
   */
   public static ObjectName[] getServerRuntimes() throws Exception {
      return (ObjectName[]) connection.getAttribute(service,
         "ServerRuntimes");
   }

   /* 
   * Iterate through ServerRuntimeMBeans and get the name and state
   */
   public void printNameAndState() throws Exception {
      ObjectName[] serverRT = getServerRuntimes();
      System.out.println("got server runtimes");
      int length = (int) serverRT.length;
      for (int i = 0; i < length; i++) {
         String name = (String) connection.getAttribute(serverRT[i],
            "Name");
         String state = (String) connection.getAttribute(serverRT[i],
            "State");
         System.out.println("Server name: " + name + ".   Server state: "
            + state);
      }
   }

   public static void main(String[] args) throws Exception {
      String hostname = args[0];
      String portString = args[1];
      String username = args[2];
      String password = args[3];

      PrintServerState s = new PrintServerState();
      initConnection(hostname, portString, username, password);
      s.printNameAndState();
      connector.close();
   }
}

Compile the JMX program:

javac PrintServerState.java

Execution of the JMX program is as follows:

java PrintServerState 192.168.33.100 7001 weblogic welcome1

JMX WebLogic Program execution output

Message Driven Bean (EJB 3.0) deployment

JMS Server, Module creation you can refer the previous posts.

This program requires the JMS Module configurations

Sample JMS Module configuration for different types of JMS destinations

We can use any kind of JMS destinations in the MDB program. Here in the sample program we are going to use the Queue, where we will refer to its JNDI name is jms.test.TestQ1.

Create a Message Driven Bean (MDB) 3.0

This sample is for Admins and WebLogic based developers. Now open a editor and write the following Java code:

package ejb30;
import javax.ejb.MessageDriven;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.TextMessage;
import javax.ejb.ActivationConfigProperty;

@MessageDriven(
activationConfig = {
@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue") ,
@ActivationConfigProperty(propertyName="connectionFactoryJndiName",propertyValue="jms.test.QCF"),
@ActivationConfigProperty(propertyName="destinationJndiName", propertyValue="jms.test.TestQ")
}
,mappedName="jms.test.TestQ"
)

public class MyMDB implements MessageListener
{
public void onMessage(Message message)
{
TextMessage textMessage = (TextMessage) message;
try {
System.out.println("nnt(mdb) MyMDB Received n"+ textMessage.getText());
}
catch (JMSException e)
{
e.printStackTrace();
}
}
}

Note: The ConnectionFactory, Queue JNDI names must match to your configured on the WebLogic admin console.

Compile the MDB Code

This is regular process of Java program compilation, -d option used to create the package directory, dot indicate current location path and a Java file.

javac -d . MyMDB.java

Generate MDB Jar

Now lets generate the artifact which we can deploy on WebLogic platform

jar   -cvf     mdb30.jar   ejb30     MyMDB.java

Deploying  MDB application on WebLogic

Goto your WebLogic admin console select deployments select install and select the path where the mdb30.jar file created. then do the deploy the artifact.

MDB Deployment on WebLogic Cluster

Running QueueSend

All set to go!! Execute the QueueSend program to send messages to the MDB, which is listening on the WebLogic cluster/server.

java jms.test.QueueSend t3://192.168.33.100:7011

QueueSend to MDB

MDB Listening

 tail -100f ms1.out

Finally you can see the received messages in the Managed server output log. That confirms JMS capabilities.