Tag Archive | jmx

cjmx: A command-line version of JConsole

JConsole is a nice tool when it comes to monitoring a running Java application. But when it is not possible to connect to a JVM with JConsole directly (due to network restrictions for example) and SSH tunneling is not possible, then it would be great to have a command line version of JConsole.

jcmx is such a command line version of JConsole. After having downloaded the single jar file cjmx_2.10-2.1.0-app.jar you can start it by including the tools.jar into the classpath:

java -cp $JAVA_HOME/lib/tools.jar:cjmx_2.10-2.1.0-app.jar cjmx.Main

This will open a “JMX shell” with the following basic commands:

  • help: This shows a basic help screen that explains the available commands.
  • jps/list: Like the jps tool from the JDK this command prints out all java processes with their process id.
  • connect: You can use this command to connect to a running JVM process.
  • format: Let’s you specify whether you want your output in a simple text format or as a JSON string.
  • exit: Quits the application.

To learn more about cjmx let us start a session and connect to the JVM that is running cjmx itself:

> jps
13198 cjmx.Main
> connect 13198
Connected to local virtual machine 13198
Connection id: rmi://0:0:0:0:0:0:0:1  2
Default domain: DefaultDomain
5 domains registered consisting of 19 total MBeans
>
describe     disconnect   exit         format       help         invoke       mbeans       names        names        sample       select       status

After the last appearance of > you see a great feature of cjmx: auto-completion. Every time you do not know which commands are available, you can just type [TAB] and cjmx will list them. This even works for MBean names as we will see.

Now that we are connected to our JVM we can let cjmx describe an available MBean. With auto-completion we can just start typing describe '[TAB] to retrieve a list of all available packages:

> describe '
:                     JMImplementation:     com.sun.management:   java.lang:            java.nio:             java.util.logging:

This way we can dig through the MBean names until we have found what we are looking for. In this example we are interested in the MBean ‘java.lang:type=OperatingSystem’:

> describe 'java.lang:type=OperatingSystem'
Object name: java.lang:type=OperatingSystem
-------------------------------------------
Description: Information on the management interface of the MBean

Attributes:
  MaxFileDescriptorCount: long
  OpenFileDescriptorCount: long
  FreePhysicalMemorySize: long
  CommittedVirtualMemorySize: long
  FreeSwapSpaceSize: long
  ProcessCpuLoad: double
  ProcessCpuTime: long
  SystemCpuLoad: double
  TotalPhysicalMemorySize: long
  TotalSwapSpaceSize: long
  AvailableProcessors: int
  Arch: String
  SystemLoadAverage: double
  Name: String
  Version: String
  ObjectName: ObjectName

As we can see, the MBean ‘java.lang:type=OperatingSystem’ provides information about the number of open files and the current CPU load, etc.. So let’s query the number of open files by invoking the command mbeans with the name of the MBean as well as the sub-command select and the MBean’s attribute:

> mbeans 'java.lang:type=OperatingSystem' select OpenFileDescriptorCount
java.lang:type=OperatingSystem
------------------------------
  OpenFileDescriptorCount: 35

We can even query all available attributes by using the star instead of the concrete name of an attribute. Please note that using the cursor up key recalls the last issued command, hence we do not have to type it again. Instead we just replace the attribute’s name with the star:

> mbeans 'java.lang:type=OperatingSystem' select *
java.lang:type=OperatingSystem
------------------------------
  MaxFileDescriptorCount: 10240
  OpenFileDescriptorCount: 36
...

By using the sub-command invoke we can even invoke MBean methods like in the following example:

> mbeans 'java.lang:type=Memory' invoke gc()
java.lang:type=Memory: null

Now that we know how to query attributes and invoke methods, we can start to script this functionality in order to monitor the application. To support this kind of scripting, cjmx provides the feature that one can pass all “commands” also as an argument to the application itself, hence you can invoke cjmx in the following way (where <PID> has to be replaced by a concrete process id of a running JVM):

java -cp $JAVA_HOME/lib/tools.jar:cjmx_2.10-2.1.0-app.jar cjmx.Main &amp;amp;lt;PID&amp;amp;gt; &amp;amp;quot;mbeans 'java.lang:type=OperatingSystem' select OpenFileDescriptorCount&amp;amp;quot;
java.lang:type=OperatingSystem
------------------------------
  OpenFileDescriptorCount: 630

With this knowledge we can write a simple bash script that queries the JVM each second for the number of open files:

#!/bin/bash
while [ true ] ; do
        echo `date` | tr -d '\n'
        java -cp /usr/java/default/lib/tools.jar:cjmx_2.10-2.1.0-app.jar cjmx.Main $1 &amp;amp;quot;mbeans 'java.lang:type=OperatingSystem' select OpenFileDescriptorCount&amp;amp;quot;|grep OpenFileDescriptorCount|cut -f 2 -d :
		sleep 1
done

This produces each second a new line with a timestamp and the the current number of open files. When redirected into a file, we have a simple log file and can evaluate it later on.

Conclusion: cjmx is a great alternative to JConsole when the latter cannot be used due to network restrictions on a server machine. The ability to even issue commands by passing them on the command line makes it suitable for small monitoring scripts.

Analysing the performance degradation/improvements of a Java EE application with interceptors

When you are developing a Java EE application with certain performance requirements, you have to verify that these requirements are fulfilled before each release. An Hudson job that nightly executes a bunch of test measurements on some specific hardware platform is what you may think about.

You can check the achieved timings and compare them with the given requirements. If the measured values deviate from the requirements too much, you can either break the build or at least send an email to the team.

But how do you measure the execution time of your code? The very first thought might be to add thousands of insertions of time measuring code into your code base. But this is not only a lot of work, but also has an impact on the performance of your code, as now the time measurements are also executed in production. To get rid of the many insertions you might want to leverage an aspect oriented framework (AOP) that introduces the code for time measurements at compile time. Using this way you have at least two versions of your application: the one with and the one without the additional overhead. To measure performance at some production site still requires a redeployment of your code. And you have to decide which methods you want to observe already at compile time.

Java EE comes therefore with an easy to use alternative: Interceptors. This is were the inversion of control pattern plays out its advantages. As the Application Server invokes your bean methods/web service calls, it is easy for it to intercept these invocations and provide you a way of adding code before and after each invocation.

Using interceptors is then fairly easy. You can either add an annotation to your target method or class that references your interceptor implementation or you can add the interceptor using the deployment descriptor:

@Interceptors(PerformanceInterceptor.class)
public class CustomerService {
...
}

The same information supplied in the deployment descriptor looks like this:

<interceptor-binding>
    <target-name>myapp.CustomerService</target-name>
    <interceptor-class>myapp.PerformanceInterceptor.class</interceptor-class>
</interceptor-binding>

The interceptor itself can be a simple POJO class with a method that is annotated with @AroundInvoke and one argument:

@AroundInvoke
public Object measureExecutionTime(InvocationContext ctx) throws Exception {
	long start = System.currentTimeMillis();
	try {
		return ctx.proceed();
	} finally {
		long time = System.currentTimeMillis() - start;
		Method method = ctx.getMethod();
		RingStorage ringStorage = RingStorageFactory.getRingStorage(method);
		ringStorage.addMeasurement(time);
	}
}

Before the try block and in the finally block we add our code for the time measurement. As can be seen from the code above, we also need some in-memory location where we can store the last measurement values in order to compute for example a mean value and the deviation from the mean value. In this example we have a simple ring storage implementation that overrides old values after some time.

But how to expose these values to the world outside? As many other values of the Application Server are exposed over the JMX interface, we can implement a simple MXBean interface like shown in the following code snippet:

public interface PerformanceResourceMXBean {
    long getMeanValue();
}

public class RingStorage implements PerformanceResourceMXBean {
	private String id;

	public RingStorage(String id) {
		this.id = id;
		registerMBean();
		...
	}
	
	private void registerMBean() {
        try {
            ObjectName objectName = new ObjectName("performance" + id + ":type=" + this.getClass().getName());
            MBeanServer platformMBeanServer = ManagementFactory.getPlatformMBeanServer();
            try {
                platformMBeanServer.unregisterMBean(objectName);
            } catch (Exception e) {
            }
            platformMBeanServer.registerMBean(this, objectName);
        } catch (Exception e) {
            throw new IllegalStateException("Problem during registration:" + e);
        }
    }
	
	@Override
    public long getMeanValue() {
        ...
    }
	...
}

Now we can start the jconsole and query the exposed MXBean for the mean value:

jconsole

Writing a small JMX client application that writes the sampled values for example into a CSV file, enables you to later on process these values and compare them with later measurements. This gives you an overview of the evolution of your application’s performance.

Conclusion: Adding dynamically through the deployment descriptor performance measurement capabilities to an existing Java EE application is with the use of interceptors easy. If you expose the measured values over JMX, you can apply further processing of the values afterwards.