JBoss Application Monitoring and JVM Tuning

Jconsole

Ability to monitor Live application is a costly process. JBoss offers some out of the box tools to monitor the same. One of the tools we are very familiar with is jconsole, this comes bundled with Java JBOSS_HOME/bin and can be used to access the JVM running the server instance. Use Remote Process Option to connect and the connection string looks like: service:jmx:remoting-jmx://my_app_url:9999 The default port is 9999 unless u have port offet in place in which case you would have to add the offset.

Once you are able to log in, you can monitor the JVM activities and GC:

If your application is dependent on container to manage persistence and session beans, you may consider having a good memory to start with, our application with request coming in at millions was initially set to 3GB (-Xmx2G)
This was crashing the application as it was not able to clear old gen space. Moreover the new G1 Garbage Collector doesnt do its job as well as the older GC.
A stable app VM arguments looks like:

-XX:+UseCompressedOops -Dprogram.name=standalone.bat -Xms1G -Xmx4G -XX:MaxPermSize=512M -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jbos88,server=y,suspend=n -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=heap_dump/myapp -Dorg.jboss.boot.log.file=D:\JBOSS6.2\myapp log\server.log

• +UseConcMarkSweepGC does a much better job in clearing old gen space periodically.
• +HeapDumpOnOutOfMemoryError along with HeapDumpPath dumps heap for analysis in case of JVM crashes with out of memory exception.

Heap Dump Analysis

If your application is frequntly running out of memory or you suspect memory leaks, you could either wait for the application to crash and generate heap dumps for you(provided u have given the VM arguments saying so), You can use the jconsole to generate heap dumps at any point if you need them for analysis.

The above dump will be generated into the Jboss working directory thats the bin folder.
Once you have the dump you could use anlaysis tool to look further.
One such tool available is from eclipse community:
Eclipse Memory Analyzer (MAT) This tool will help us identify any leak suspects:

HTML5 and contentEditable

---

HTML5 came up with contentEditable feature, its cool but difficult to work around, well is it? contentEditable is a HTML5 property and can be applied to any HTML element, thus rendering all its text editable. Lets see how to use it and bind events to it: Below is how to apply:

<li contenteditable="true" id="HTMLDivElement_lastName" href="#">Last Name</li>

Above code makes the text content under the li editable, now if you want the <li> itself to be editable(delete/edit it then the contentEditable must be applied to the <ul> that holds the li or the <div> that encapsulates the whole.) Thats pretty much on how to use.

---

Adding events is what is the trick for contenteditable. When i had to do this, there was not much help and few of the following observations were intresting: Why ContentEditable is Terrible However, to capture edits, jquery was a rescue, binding a combination of hover and blur did the trick. Below is javascript code:

$(this).blur(function() {
  if(before !== $(this).text()){
     $.post('./rest/htmlEditor/updateText',  $(this).attr("id") );
 }
     }); 

(this).hover(function () {
    before = $(this).text();
 , function () {
    //do nothing
 }
);

---

Remote Debug with Eclipse

How many times has it happened with us developers that it all runs fine in local and the moment the code gets into QA or PROD it behaves like a step-bro. The same has been happening with me and debugging is hell in such scenarios, unless i realised how simple its to debug remote applications using eclipse debugger.

1. Normally we have our local apps deployed in http://localhost:8080 and debug port at 8787,

2. Suppose the QA or PROD app is deployed at http://my-qa-site.com, you would have to restart the web application with additional JVM parameters to
-Xrunjdwp:transport=dt_socket,address=8888,server=y,suspend=n
and restart the application

3. Once the server is up in debug profile assuming u have the same version of code in your local eclipse, open debug configuration

4. Under host:my-qa-site and Port:8888

5. Thats it you are good to go!! But mind it everyone else accessing this site will now be stopped at your breakpoints

Good luck!!

Jboss as a Windows Service

Note the param SVCNAME this corrsponds to the server directory at JBOSS_HOME path. Also note that we use SVCNAME to stop the server, if your svc name is different from the actual url, you may need an additional parameter here. Below is the bat file, edit it and install it.

@echo off
REM JBoss, the OpenSource webOS
REM
REM Distributable under LGPL license.
REM See terms of license at gnu.org.
REM
REM -------------------------------------------------------------------------
REM JBoss Service Script for Windows
REM -------------------------------------------------------------------------


@if not "%ECHO%" == "" echo %ECHO%
@if "%OS%" == "Windows_NT" setlocal
set DIRNAME=%CD%

REM
REM VERSION, VERSION_MAJOR and VERSION_MINOR are populated
REM during the build with ant filter.
REM

REM ---------------------Aldera-----------------------------
REM --------------------------------------------------------
REM SVCNAME should be set with the value of the 
REM The Service bat should be named -SERVICE.bat
REM The Client start up bat should be named .bat
REM --------------------------------------------------------
REM --------------------------------------------------------

set SVCNAME=DEPLOYMENT_DIRECTORY
set SVCDISP=%SVCNAME%
set SVCDESC=JBoss Application Server 6.2 for BASE_HEAD
set NOPAUSE=Y
set JBOSS_HOME=C:\JBOSS6.2
set JAVA_HOME=C:\jdk1.7.0_51
REM Suppress killing service on logoff event
set "JAVA_OPTS=-Xms2G -Xmx2G -XX:MaxPermSize=512M -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jbos88,server=y,suspend=n"
set CLIENT_FOLDER=%SVCNAME%
set SERVICE_LOG=%JBOSS_HOME%\%CLIENT_FOLDER%\log\service.log
set CLIENT_DIR_BAT=%SVCNAME%.bat
set JBOSS_SERVICE_BAT=%SVCNAME%-SERVICE.bat

REM Figure out the running mode

if /I "%1" == "install"   goto cmdInstall
if /I "%1" == "uninstall" goto cmdUninstall
if /I "%1" == "start"     goto cmdStart
if /I "%1" == "stop"      goto cmdStop
if /I "%1" == "restart"   goto cmdRestart
if /I "%1" == "signal"    goto cmdSignal
echo Usage: service install^|uninstall^|start^|stop^|restart^|signal
goto cmdEnd

REM jbosssvc retun values
REM ERR_RET_USAGE           1
REM ERR_RET_VERSION         2
REM ERR_RET_INSTALL         3
REM ERR_RET_REMOVE          4
REM ERR_RET_PARAMS          5
REM ERR_RET_MODE            6

:errExplain
if errorlevel 1 echo Invalid command line parameters
if errorlevel 2 echo Failed installing %SVCDISP%
if errorlevel 4 echo Failed removing %SVCDISP%
if errorlevel 6 echo Unknown service mode for %SVCDISP%
goto cmdEnd

:cmdInstall
jbosssvc.exe -imwdc %SVCNAME% "%DIRNAME%" "%SVCDISP%" "%SVCDESC%" %JBOSS_SERVICE_BAT%
if not errorlevel 0 goto errExplain
echo Service %SVCDISP% installed
goto cmdEnd

:cmdUninstall
jbosssvc.exe -u %SVCNAME%
if not errorlevel 0 goto errExplain
echo Service %SVCDISP% removed
goto cmdEnd

:cmdStart
REM Executed on service start
jbosssvc.exe -p 1 "Starting %SVCDISP%" > %SERVICE_LOG%
call %CLIENT_DIR_BAT% 
jbosssvc.exe -p 1 "Shutdown %SVCDISP% service" >> %SERVICE_LOG%
goto cmdEnd

:cmdStop
REM Executed on service stop
jbosssvc.exe -p 1 "Shutting down %SVCDISP%" > %SERVICE_LOG%
call jboss-cli.bat --connect --controller=%SVCNAME%:9999 command=:shutdown > %SERVICE_LOG% 2>&1
jbosssvc.exe -p 1 "Shutdown %SVCDISP% service" >> %SERVICE_LOG%
goto cmdEnd

:cmdRestart
REM Executed manually from command line
REM Note: We can only stop and start
jbosssvc.exe -p 1 "Shutting down %SVCDISP%" >> %SERVICE_LOG%
call jboss-cli.bat --connect --controller=%SVCNAME%:9999 command=:shutdown

:waitRun
jbosssvc.exe -s 1
jbosssvc.exe -p 1 "Restarting %SVCDISP%" >> %SERVICE_LOG%
call %CLIENT_DIR_BAT%
jbosssvc.exe -p 1 "Shutdown %SVCDISP% service" >> %SERVICE_LOG%
 goto cmdEnd

:cmdSignal
REM Send signal to the service.
REM Requires jbosssch.dll to be loaded in JVM
@if not ""%2"" == """" goto execSignal
echo Missing signal parameter.
echo Usage: service signal [0...9]
goto cmdEnd
:execSignal
jbosssvc.exe -k%2 %SVCNAME%
goto cmdEnd

:cmdEnd

Jboss Multiple Standalone Servers

Suppose in JBoss Web server you want to deploy multiple instances of a web server in standalone mode outside of the standalone directory...

• Now that u have multiple servers 1, 2 3 and 4(essentially copies of standalone directory that are simply renamed) , you can place your war/ear file in their respective deployment folder
• Now create a bat files to each of the servers so as to start them individually (server1.bat) in the bin directory with text in it:
  standalone.bat -Djboss.server.base.dir=../Server1 --debug 8888
  Above line means you are calling standalone.bat on server1 with debug port 8888
• Make sure you have the port offset configured in the respective standalone.xml so that you can access each of the sites simultaneously
  Offset 5 means u access the local host using http://localhost:8085
• Now you can run custom bat files simultaneously and access the url at different ports as offset.
  For external sites with public facing URL u can directly specify the URL at the <interface name="public">
  <inet-address value="${jboss.bind.address:UR_URL}"/>
  </interface>
  Make sure Socket Binding for http is 80 not 8080 in cases of URL binding (Local host binding need to have 8080)

Sample Code Java Mail POP3

//Import These Classes

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Properties;

import javax.mail.Folder;
import javax.mail.Message;
import javax.mail.MessagingException;
import javax.mail.Multipart;
import javax.mail.Session;
import javax.mail.Store;
import javax.mail.internet.MimeBodyPart;


String HOST = "pop.mail.yahoo.com";
  String USERNAME = "";
  String PASSWORD = "";

  // Mail Test

  Folder folder = null;
  Store store = null;
  try {
  Properties props = new Properties();
  props.put("mail.store.protocol", "pop3"); 
// Google uses POP3S not POP3
  Session session = Session.getDefaultInstance(props);
  // session.setDebug(true);
  store = session.getStore();
  store.connect(HOST, USERNAME, PASSWORD);
  folder = store.getDefaultFolder().getFolder("SENT");
  folder.open(Folder.READ_ONLY);
  Message[] messages = folder.getMessages();
  //System.out.println("No of Messages : " + folder.getMessageCount());
  //System.out.println("No of Unread Messages : "
  //  + folder.getUnreadMessageCount());
  for (int i = 0; i < messages.length; ++i) {
   //System.out.println("MESSAGE #" + (i + 1) + ":");
   Message msg = messages[i];
   String from = "unknown";
   if (msg.getReplyTo().length >= 1) {
    from = msg.getReplyTo()[0].toString();
   } else if (msg.getFrom().length >= 1) {
    from = msg.getFrom()[0].toString();
   }
   //String subject = msg.getSubject();
   //System.out.println("Saving ... " + subject + " " + from);
   // you may want to replace the spaces with "_"
   // the files will be saved into the TEMP directory
                        //Use msg.getContent() to fetch and manage the contents
System.out.println("From # "+msg.getFrom()[0].toString());
   }
  } catch (MessagingException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  } finally {
   if (folder != null) {
    try {
     folder.close(true);
    } catch (MessagingException e) {
     // TODO Auto-generated catch block
     e.printStackTrace();
    }
   }
   if (store != null) {
    try {
     store.close();
    } catch (MessagingException e) {
     // TODO Auto-generated catch block
     e.printStackTrace();
    }
   }
  }