Updating code at runtime (spring-loaded demystified)

Updating code at runtime (spring-loaded demystified)

When the development cycle from compile over deployment up to testing takes too long, one wishes to be able to replace the running code just in time without the need for restarting an application server and waiting until deployment has been finished. Commercial solutions like JRebel or open source frameworks like Grails help in such kind of situations.

Replacing code at runtime is not supported out of the box by the JVM in a kind like you can dynamically load classes with for example Class.forName(). Basically you have the following options:

  • HotSwap: A technolog introduced with Java 1.4 that allows you to redefine classes within a debugger session. This approach is very limited as it only allows you to change the body of a method but not the addition of new methods or classes.
  • OSGi: This technology allows you to define bundles. At runtime a bundle can be replaced by a newer version of this bundle.
  • Throwaway classloaders: By wrapping a separate Classloader over all classes of your module, you can throw away the Classloader and replace it, once a new version of your module is availalbe.
  • Instrumenting classes with a Java Agent: A Java Agent can instrument classes before they are defined. This way it can inject code into loaded classes that connects this class with one version of the class file. Once a new version is available, the new code gets executed.

The technology behing Grails is called spring-loaded and uses the “Java Agent” approach to instrument classes that are loaded from the file system and not from a jar file. But how does this work under the hood?

To understand spring-loaded, we setup a small sample project that allows us to examine the technology in more detail. This project only consists of two classes: the Main class calls the print() method of the ToBeChanged class and sleeps for a while:

public static void main(String[] args) throws InterruptedException {
  while (true) {
    ToBeChanged toBeChanged = new ToBeChanged();
    toBeChanged.print();
    Thread.sleep(500);
  }
}

The print() method just prints out a version, such that we can see that it has changed. Additionally we also print out the stack trace in order to see how this changes over time:

public void print() {
  System.out.println("V1");
  StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();
  for (StackTraceElement element : stackTrace) {
    System.out.println("\t" + element.getClassName() + "." 
      + element.getMethodName() + ":" + element.getLineNumber());
  }
}

When starting the application we have to provide the jar file that contains the Java Agent using the option javaagent. As spring-loaded modifies the bytecode in a way that the verifier does not like, we have to disable verification of the bytecode by passing the option noverify to the JVM. Finally we pass the folder that contains our class files with cp and tell the JVM the class that contains the main() method:

java -javaagent:springloaded-1.2.4.BUILD-SNAPSHOT.jar 
  -noverify 
  -cp target/classes 
  com.martinsdeveloperworld.springloaded.Main

After having updated the version in class ToBeChanged from V1 to V2 and rebuilding the project with mvn package, we see the following output:

...
V1
        java.lang.Thread.getStackTrace:-1
        com.martinsdeveloperworld.springloaded.ToBeChanged.print:7
        com.martinsdeveloperworld.springloaded.Main.main:8
V2
        java.lang.Thread.getStackTrace:-1
        com.martinsdeveloperworld.springloaded.ToBeChanged$$EPBF0gVl.print:7
        com.martinsdeveloperworld.springloaded.ToBeChanged$$DPBF0gVl.print:-1
        com.martinsdeveloperworld.springloaded.ToBeChanged.print:-1
        com.martinsdeveloperworld.springloaded.Main.main:8
...

The stacktrace of version V1 looks like we have expected. From Main.main() the method ToBeChanged.print() gets called. This differs for version V2. Here the method ToBeChanged.print now calls the method ToBeChanged$$DPBF0gVl.print(). Please also note that the line number for the call ToBeChanged.print() has changed from 8 to -1, indicating that the line is not known.

The new line number -1 is a strong indication that the Java Agent has instrumented the method ToBeChanged.print() in a way that allows it to call the new method instead of executing the old code. To prove this assumption, I have added a few logging statements to the code of spring-loaded and a feature that dumps each instrumtend file to the local hard drive. This way we can inspect how the method ToBeChanged.print() looks like after instrumentation:

  0 getstatic #16 <com/martinsdeveloperworld/springloaded/ToBeChanged.r$type>
  3 ldc #72 <0>
  5 invokevirtual #85 <org/springsource/loaded/ReloadableType.changed>
  8 dup
  9 ifeq 42 (+33)
 12 iconst_1
 13 if_icmpeq 26 (+13)
 16 new #87 <java/lang/NoSuchMethodError>
 19 dup
 20 ldc #89 <com.martinsdeveloperworld.springloaded.ToBeChanged.print()V>
 22 invokespecial #92 <java/lang/NoSuchMethodError.<init>>
 25 athrow
 26 getstatic #16 <com/martinsdeveloperworld/springloaded/ToBeChanged.r$type>
 29 invokevirtual #56 <org/springsource/loaded/ReloadableType.fetchLatest>
 32 checkcast #58 <com/martinsdeveloperworld/springloaded/ToBeChanged__I>
 35 aload_0
 36 invokeinterface #94 <com/martinsdeveloperworld/springloaded/ToBeChanged__I.print> count 2
 41 return
 42 pop
 43 getstatic #100 <java/lang/System.out>
 46 ldc #102 <V1>
 48 invokevirtual #107 <java/io/PrintStream.println>
 51 invokestatic #113 <java/lang/Thread.currentThread>
 54 invokevirtual #117 <java/lang/Thread.getStackTrace>
 57 astore_1
...
152 return

The getstatic opcode retrieves the value for the new field r$type and pushes it on the stack (opcode ldc). Then the method ReloadableType.changed() gets called for the object reference that was pushed on the stack before. As the name indicates, the method ReloadableType.changed() checks whether a new version of this type exists. It returns 0 if the method did not change and 1 if it has changed. The following opcode ifeq jumps to line 42 if the returned value was zero, i.e. the method has not changed. From line 42 on we see the original implementation which I have shortened here a little bit.

If the value is 1, the if_icmpeq instruction jumps to line 26, where the static field r$type is read once again. This reference is used to invoke the method ReloadableType.fetchLatest() on it. The following checkcast instruction verifies that the returned reference is of type ToBeChanged__I. Here we stumble for the first time over this artifical interface that spring-loaded generates for each type. It reflects the methods the original class had when it was instrumented. Two lines later this interface is used to invoke the method print() on the reference that was returned by ReloadableType.fetchLatest().

This reference is not the reference to the new version of the class but to a so called dispatcher. The dispatcher implements the interface ToBeChanged__I and implements the method print() with the following instructions:

0 aload_1
1 invokestatic #21 <com/martinsdeveloperworld/springloaded/ToBeChanged$$EPBF0gVl.print>
4 return

The dynamically generated class ToBeChanged$$EPBF0gVl is the so called executor and embodies the new version of the type. For each new version a new dispatcher and executor is created, only the interface remains the same. Once a new version is available, the interface method is invoked on the new dispatcher and this one forwards in the simplest case to the new version of the code embodied in the executor. The reason why the interface method is not called directly on the exeuctor is the fact that spring-loaded can also handle cases in which methods are added in a new version of the class. As this methods do not exist in the old version, a generic method __execute() is added to the interface and the dispatcher. This dynamic method can then dispatch calls to new methods as shown in the following instruction set taken from the generated dispatcher:

 0 aload_3
 1 ldc #25 <newMethod()V>
 3 invokevirtual #31 <java/lang/String.equals>
 6 ifeq 18 (+12)
 9 aload_2
10 checkcast #33 <com/martinsdeveloperworld/springloaded/ToBeChanged>
13 invokestatic #36 <com/martinsdeveloperworld/springloaded/ToBeChanged$$EPBFaboY.newMethod>
16 aconst_null
17 areturn
18 aload_3
...
68 areturn

In this case I have added a new method called newMethod() to the class ToBeChanged. The beginning of the __execute() method compares whether the descriptor invoked matches the new method. If this is the case, it forwards the invocation to the new executor. In order to let this work, all invocations of the new method have to be rewritten to the __execute() method. This is also done via instrumentation of the original classes and does also work for reflection.

Conclusion: spring-loaded demonstrates that it is possible to “replace” a class with a newer version at runtime. To achieve this, a series of Java technologies like the Java Agent and bytecode instrumentation are utilized. By taking a closer look at the implementation, one can learn a lot of things about the JVM and Java in general.

Creating a DSL for AWT’s Robot

The Java SDK ships with the class java.awt.Robot that allows the automation of keyboard and mouse input as well as the creation of screen captures. When you want to write a small test application that simulates user input or you just want to automate the input of some recurring text, this features comes in handy. But you do not want to write every time a complete Java application.

On the other hand ANTLR is a parser generator that enables us to create “Domain Specific Languages” (DSL). With the help of ANTLR we can develop a simple DSL that provides one command for each of the methods of java.awt.Robot. From then on we can easily write a script for various kinds of simple automation tasks.

The first step is to invent the syntax of our new “DSL”:

  • Different “statements” should be separated by a semicolon.
  • Each statement should consist of one “command” and a few parameters for this command.
  • Comments should either span multiple lines (using the C-like comments /* … */ or only until the end of the line.

A simple file could look like this:

/*
* A simple example demonstrating the basic features.
*/
delay 300; // sleep for 300ms
mouseMove 20,30;
createScreenCapture 100,100,200,200 file=/home/siom/capture.png;
mouseClick button1;
keyboardInput "Test";
delay 400;

With these requirements we can start to write down the grammar:

grammar Robot;

instructions:
    (instruction ';')+
    EOF;

instruction:
    instructionDelay |
    instructionMouseMove |
    instructionCreateScreenCapture |
    instructionMouseClick |
    instructionKeyboardInput;

We name the grammar “Robot” and define the first rule instructions such that we have one or more instructions followed by a semicolon as instruction separator until the end of the file is reached (EOF). The instructions that we want to support are listed as part of the rule instruction. The pipe between the different rules denotes a logical OR, i.e. only one of these rules has to match.

The most simple rule is the instructionDelay one:

instructionDelay:
    'delay' paramMs=INTEGER;
...
INTEGER:
    [0-9]+;

The rule starts with the command ‘delay’ followed by the only parameter that specifies the number of milliseconds to sleep as an integer. The token INTEGER is shown below the rule. It just defines that we expect at least one number between zero and nine. In order to ease the processing of the parameter later on, we assign the parameter to a separate tree node named paramMs.

The rule to take a screen capture looks like the following one:

instructionCreateScreenCapture:
    'createScreenCapture' x=INTEGER ',' y=INTEGER ',' w=INTEGER ',' h=INTEGER  'file=' file=FILENAME;
...
FILENAME:
    FileNameChar+;
fragment FileNameChar:
    [a-zA-Z0-9/\\:_-$~.];

Followed by the keyword createScreenCapture the user has to provide the two coordinates on the screen of the upper left point of the rectangle that should be captured. The two following coordinates denote the width and the height of the rectangle. Finally the user has to provide a filename for the captured image.

The filename consists of one or more characters from the fragment FileNameChar. This fragment defines all characters that should be allowed for a filename.

Using maven we can now store this grammar as file Robot.g4 in the folder src/main/antlr4 and utilize the corresponding maven plugin to generate the Java lexer and parser:

<build>
	<plugins>
		<plugin>
			<groupId>org.antlr</groupId>
			<artifactId>antlr4-maven-plugin</artifactId>
			<version>${antlr.version}</version>
			<executions>
				<execution>
					<goals>
						<goal>antlr4</goal>
					</goals>
				</execution>
			</executions>
		</plugin>
		...
	</plugins>
</build>

<dependencies>
	<dependency>
		<groupId>org.antlr</groupId>
		<artifactId>antlr4-runtime</artifactId>
		<version>${antlr.version}</version>
	</dependency>
	...
</dependencies>

The dependency on antlr4-runtime is necessary to use the generated classes in our own code.

The method execute() takes a Path to an input file as parameter and parses and executes it:

public void execute(Path inputPath) throws IOException, AWTException {
	RobotLexer lexer = new RobotLexer(new ANTLRInputStream(new FileInputStream(inputPath.toFile())));
	RobotParser parser = new RobotParser(new CommonTokenStream(lexer));
	final Robot robot = new Robot();
	parser.addParseListener(new RobotBaseListener() {
		@Override
		public void exitInstructionDelay(@NotNull RobotParser.InstructionDelayContext ctx) {
			int delayParam = Integer.parseInt(ctx.paramMs.getText());
			LOGGER.info("delay(" + delayParam + ")");
			robot.delay(delayParam);
		}
		...
	});
	parser.instructions();
}

The content of the file is forwarded via the ANTLRInputStream to the RobotLexer that has been generated by ANTLR. After the lexer has parsed the file and generated a stream of tokens, this stream can be passed to the actual RobotParser.

In order to react to the incoming instructions, a ParseListener is added. Fortunately ANTLR has already created a base listener that implements all callback methods with an empty implementation. Hence we only have to override the methods we want to process. As ANTLR creates for each parser rule one callback method, we can override for example the method exitInstructionDelay(). The parameter passed in by the generated code is of type RobotParser.InstructionDelayContex. This context object has a field paramMs as we have assigned the parameter in the grammar before to a separate node. Its getText() method returns the value for this parameter as String. We only have to convert it to an integer value and then pass it to the delay() method of the Robot instance.

The implementation for the rule instructionCreateScreenCapture is shown in the following block:

@Override
public void exitInstructionCreateScreenCapture(@NotNull 
	RobotParser.InstructionCreateScreenCaptureContext ctx) {
	int x = Integer.parseInt(ctx.x.getText());
	int y = Integer.parseInt(ctx.y.getText());
	int w = Integer.parseInt(ctx.w.getText());
	int h = Integer.parseInt(ctx.h.getText());
	LOGGER.info("Rectangle rectangle = new Rectangle(" + x + "," + y + 
		"," + w + "," + h + ")");
	Rectangle rectangle = new Rectangle(x, y, w, h);
	LOGGER.info("createScreenCapture(rectangle);");
	BufferedImage bufferedImage = robot.createScreenCapture(rectangle);
	File output = new File(ctx.file.getText());
	LOGGER.info("Save file to " + output.getAbsolutePath());
	try {
		ImageIO.write(bufferedImage, "png", output);
	} catch (IOException e) {
		throw new RuntimeException("Failed to write image file: " + e.getMessage(), e);
	}
}

The principle is the same as shown for the last instruction. The context object passed in has one field for each parameter and these string values have to be converted into integer values. With this information we can construct a Rectangle object, call the createScreenCapture() method of the Robot and store its BufferedImage.

Conclusion: Creating a specialized DSL for AWT’s Robot was easier than expected. The provided maven plugin creates all necessary classes out of the grammar file and therewith integrates smoothly into the build process. The resulting DSL can be used to automate simple mouse and keyboard tasks including the creation of screenshots.

PS: The source code is available at github.

JPA 2.1: Unsynchronized persistence context

The JPA version 2.1 brings a new way how to handle the synchronization between the persistence context and the current JTA transaction as well as the resource manager. The term resource manager comes from the Java Transaction API and denotes a component that manipulates one resource (for example a concrete database that is manipulated by using its JDBC driver). Per default a container-managed persistence context is of type SynchronizationType.SYNCHRONIZED, i.e. this persistence context automatically joins the current JTA transaction and updates to the persistence context are propagated to the underlying resource manager.

By creating a persistence context that is of the new type SynchronizationType.UNSYNCHRONIZED, the automatic join of the transaction as well as the propgation of updates to the resource manager is disabled. In order to join the current JTA transaction the code has to call the method joinTransaction() of the EntityManager. This way the EntityManager’s persistence context gets enlisted in the transaction and is registered for subsequent notifications. Once the transaction is commited or rolled back, the persistence context leaves the transaction and is not attached to any further transaction until the method joinTransaction() is called once again for a new JTA transaction.

Before JPA 2.1 one could implement a conversation that spans multiple method calls with a @Stateful session bean as described by Adam Bien here:

@Stateful
@TransactionAttribute(TransactionAttributeType.NEVER)
public class Controller {
	@PersistenceContext(type = PersistenceContextType.EXTENDED)
	EntityManager entityManager;

	public Person persist() {
		Person p = new Person();
		p.setFirstName(&quot;Martin&quot;);
		p.setLastName(&quot;Developer&quot;);
		return entityManager.merge(p);
	}

	public List&lt;Person&gt; list() {
		return entityManager.createQuery(&quot;from Person&quot;, Person.class).getResultList();
	}

	@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
	public void commit() {
		
	}

	@Remove
	public void remove() {

	}
}

The persistence context is of type EXTENDED and therefore lives longer than the JTA transactions it is attached to. As the persistence context is per default also of type SYNCHRONIZED it will automatically join any transaction that is running when any of the session bean’s methods are called. In order to prevent that to happen for most of the bean’s methods, the annotation @TransactionAttribute(TransactionAttributeType.NEVER) tells the container to not open any transaction for this bean. Therefore the methods persist() and list() run without a transaction. This behavior is different for the method commit(). Here the annotation @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) tells the container to create a new transaction before the method is called and therefore the bean’s EntityManager will join it automatically.

With the new type SynchronizationType.UNSYNCHRONIZED the code above can be rewritten as depicted in the following listing:

@Stateful
public class Controller {
	@PersistenceContext(type = PersistenceContextType.EXTENDED, 
		synchronization = SynchronizationType.UNSYNCHRONIZED)
	EntityManager entityManager;

	public Person persist() {
		Person p = new Person();
		p.setFirstName(&quot;Martin&quot;);
		p.setLastName(&quot;Developer&quot;);
		return entityManager.merge(p);
	}

	public List&lt;Person&gt; list() {
		return entityManager.createQuery(&quot;from Person&quot;, Person.class).getResultList();
	}

	public void commit() {
		entityManager.joinTransaction();
	}

	@Remove
	public void remove() {

	}
}

Now that the EntityManager won’t automatically join the current transaction, we can omit the @TransactionAttribute annotations. Any running transaction won’t have an impact on the EntityManager until we explicitly join it. This is now done in the method commit() and could even be done on the base on some dynamic logic.

In order to test the implementation above, we utilize a simple REST resource:

@Path(&quot;rest&quot;)
@Produces(&quot;text/json&quot;)
@SessionScoped
public class RestResource implements Serializable {
	@Inject
	private Controller controller;

	@GET
	@Path(&quot;persist&quot;)
	public Person persist(@Context HttpServletRequest request) {
		return controller.persist();
	}

	@GET
	@Path(&quot;list&quot;)
	public List&lt;Person&gt; list() {
		return controller.list();
	}

	@GET
	@Path(&quot;commit&quot;)
	public void commit() {
		controller.commit();
	}

	@PreDestroy
	public void preDestroy() {
                controller.remove();
	}
}

This resource provides methods to persist a person, list all persisted person and to commit the current changes. As we are going to use a stateful session bean, we annotate the resource with @SessionScoped and let the container inject the Controller bean.

By calling the following URL after the application has been deployed to some Java EE container, a new person gets added to the unsynchronized persistence context, but is not stored in the database.

http://localhost:8080/jpa2.1-unsychronized-pc/rest/persist

Even a call of the list() method won’t return the newly added person. Only by finally synchronizing the changes in the persistence context to the underlying resource with a call of commit(), the insert statement is send to the underlying database.

Conclusion: The new UNSYNCHRONIZED mode of the persistence context lets us implement conversations over more than one method invocation of a stateful session bean with the flexibility to join a JTA transaction dynamically based on our application logic without the need of any annotation magic.

PS: The source code is available at github.

JPA 2.1 criteria delete/update and temporary tables in Hibernate

Since JPA version 2.0 the EntityManager offers the method getCriteriaBuilder() to dynamically build select queries without the need of string concatenation using the Java Persistence Query Languge (JPQL). With version 2.1 this CriteriaBuilder offers the two new methods createCriteriaDelete() and createCriteriaUpdate() that let us formulate delete and update queries using the criteria API.

For illustration purposes lets use a simple inheritance use case with the two entities Person and Geek:

@Entity
@Table(name = "T_PERSON")
@Inheritance(strategy = InheritanceType.JOINED)
public class Person {
	@Id
	@GeneratedValue
	private Long id;
	@Column(name = "FIRST_NAME")
	private String firstName;
	@Column(name = "LAST_NAME")
	private String lastName;
	...
}

@Entity
@Table(name = "T_GEEK")
@Access(AccessType.PROPERTY)
public class Geek extends Person {
	private String favouriteProgrammingLanguage;
	...
}

To delete all geeks from our database that favour Java as their programming language, we can utilize the following code using EntityManager’s new createCriteriaDelete() method:

EntityTransaction transaction = null;
try {
	transaction = entityManager.getTransaction();
	transaction.begin();
	CriteriaBuilder builder = entityManager.getCriteriaBuilder();
	CriteriaDelete<Geek> delete = builder.createCriteriaDelete(Geek.class);
	Root<Geek> geekRoot = delete.from(Geek.class);
	delete.where(builder.equal(geekRoot.get("favouriteProgrammingLanguage"), "Java"));
	int numberOfRowsUpdated = entityManager.createQuery(delete).executeUpdate();
	LOGGER.info("Deleted " + numberOfRowsUpdated + " rows.");
	transaction.commit();
} catch (Exception e) {
	if (transaction != null && transaction.isActive()) {
		transaction.rollback();
	}
}

Like with pure SQL we can use the method from() to specify the table the delete query should be issued against and where() to declare our predicates. This way the criteria API allows the definition of bulk deletion operations in a dynamic way without using too much string concatenations.

But how does the SQL look like that is created? First of all the ORM provider has to pay attention that we are deleting from an inheritance hierarchy with the strategy JOINED, meaning that we have two tables T_PERSON and T_GEEK where the second tables stores a reference to the parent table. Hibernate in version 4.3.8.Final creates the following SQL statements:

insert 
into
	HT_T_GEEK
	select
		geek0_.id as id 
	from
		T_GEEK geek0_ 
	inner join
		T_PERSON geek0_1_ 
			on geek0_.id=geek0_1_.id 
	where
		geek0_.FAV_PROG_LANG=?;

delete 
from
	T_GEEK 
where
	(
		id
	) IN (
		select
			id 
		from
			HT_T_GEEK
	);

delete 
from
	T_PERSON 
where
	(
		id
	) IN (
		select
			id 
		from
			HT_T_GEEK
	)

delete 
from
	HT_T_GEEK;

As we can see, Hibernate fills a temporary table with the ids of the geeks/persons that match our search criteria. Then it deletes all rows from the geek table and then all rows from the person table. Finally the temporary table gets purged.

The sequence of delete statements is clear, as the table T_GEEK has a foreign key constraint on the id column of the T_PERSON table. Hence the rows in the child table have to be deleted before the rows in the parent table. The reason why Hibernate creates a temporary table is explained in this article. To summarize it, the underlying problem is that the query restricts the rows to be deleted on a column that only exists in the child table. But the rows in the child table have to be deleted before the corresponding rows in the parent table. Having deleted the rows in the child table, i.e. all geeks with FAV_PROG_LANG='Java', makes it impossible to delete afterwards all corresponding persons as the geek rows have already been deleted. The solution to this problem is the temporary table that first collects all row ids that should be deleted. Once all ids are known, this information can be used to delete the rows first from the geek table and then from the person table.

The generated SQL statements above are of course independent from the usage of the criteria API. Using the JPQL approach leads to the same generated SQL:

EntityTransaction transaction = null;
try {
	transaction = entityManager.getTransaction();
	transaction.begin();
	int update = entityManager.createQuery("delete from Geek g where g.favouriteProgrammingLanguage = :lang").setParameter("lang", "Java").executeUpdate();
	LOGGER.info("Deleted " + update + " rows.");
	transaction.commit();
} catch (Exception e) {
	if (transaction != null && transaction.isActive()) {
		transaction.rollback();
	}
}

When we change the inheritance strategy from JOINED to SINGLE_TABLE, the generated SQL statements also changes to a single one (here the discriminator column is DTYPE):

delete 
from
	T_PERSON 
where
	DTYPE='Geek' 
	and FAV_PROG_LANG=?

Conclusion: The new additions to the criteria API for deletion and update let you construct your SQL statements without the need of any string concatenation. But be aware that bulk deletions from an inheritance hierarchy can force the underlying ORM to use temporary tables in order to assemble the list of rows that have to be removed in advance.

Testing System.in and System.out with system-rules

Writing unit tests is an integral part of software development. One problem you have to solve when your class under test interacts with the operating system, is to simulate its behaviours. This can be done by using mocks instead of the real objects provided by the Java Runtime Environment (JRE). Libraries that support mocking for Java are for example mockito or jMock.

Mocking objects is a great thing when you have complete control over their instantiation. When dealing with standard input and standard output this is a little bit tricky, but not impossible as java.lang.System lets you replace the standard InputStream and OutputStream.

System.setIn(in);
System.setOut(out);

In order that you do not have to replace the streams before and after each test case manually, you can utilize org.junit.rules.ExternalResource. This class provides the two methods before() and after() that are called, like their names suggest, before and after each test case. This way you can easily setup and cleanup resources that all of your tests within one class need. Or, to come back to the original problem, replace the input and output stream for java.lang.System.

Exactly what I have described above, is implemented by the library system-rules. To see how it works, lets start with a simple example:

public class CliExample {
	private Scanner scanner = new Scanner(System.in, "UTF-8");

	public static void main(String[] args) {
		CliExample cliExample = new CliExample();
		cliExample.run();
	}

	private void run() {
		try {
			int a = readInNumber();
			int b = readInNumber();
			int sum = a + b;
			System.out.println(sum);
		} catch (InputMismatchException e) {
			System.err.println("The input is not a valid integer.");
		} catch (IOException e) {
			System.err.println("An input/output error occurred: " + e.getMessage());
		}
	}

	private int readInNumber() throws IOException {
		System.out.println("Please enter a number:");
		String nextInput = scanner.next();
		try {
			return Integer.valueOf(nextInput);
		} catch(Exception e) {
			throw new InputMismatchException();
		}
	}
}

The code above reads two intergers from standard input and prints out its sum. In case the user provides an invalid input, the program should output an appropriate message on the error stream.

In the first test case, we want to verify that the program correctly sums up two numbers and prints out the result:

public class CliExampleTest {
	@Rule
	public final StandardErrorStreamLog stdErrLog = new StandardErrorStreamLog();
	@Rule
	public final StandardOutputStreamLog stdOutLog = new StandardOutputStreamLog();
	@Rule
	public final TextFromStandardInputStream systemInMock = emptyStandardInputStream();

	@Test
	public void testSuccessfulExecution() {
		systemInMock.provideText("2\n3\n");
		CliExample.main(new String[]{});
		assertThat(stdOutLog.getLog(), is("Please enter a number:\r\nPlease enter a number:\r\n5\r\n"));
	}
	...
}

To simulate System.in we utilize system-rules’ TextFromStandardInputStream. The instance variable is initialized with an empty input stream by calling emptyStandardInputStream(). In the test case itself we provide the intput for the application by calling provideText() with a newline at the appropriate points. Then we call the main() method of our application. Finally we have to assert that the application has written the two input statements and the result to the standard input. The latter is done through an instance of StandardOutputStreamLog. By calling its method getLog() we retrieve everything that has been written to standard output during the current test case.

The StandardErrorStreamLog can be used alike for the verification of what has been written to standard error:

	@Test
	public void testInvalidInput() throws IOException {
		systemInMock.provideText("a\n");
		CliExample.main(new String[]{});
		assertThat(stdErrLog.getLog(), is("The input is not a valid integer.\r\n"));
	}

Beyond that system-rules also offers rules for the work with System.getProperty(), System.setProperty(), System.exit() and System.getSecurityManager().

Conclusion: With system-rules testing command line applications with unit tests becomes even more simpler than using junit’s Rules itself. All the boilerplate code to update the system environment before and after each test case comes within some easy to use rules.

PS: You can find the complete sources here.

cjmx: A command-line version of JConsole

JConsole is a nice tool when it comes to monitoring a running Java application. But when it is not possible to connect to a JVM with JConsole directly (due to network restrictions for example) and SSH tunneling is not possible, then it would be great to have a command line version of JConsole.

jcmx is such a command line version of JConsole. After having downloaded the single jar file cjmx_2.10-2.1.0-app.jar you can start it by including the tools.jar into the classpath:

java -cp $JAVA_HOME/lib/tools.jar:cjmx_2.10-2.1.0-app.jar cjmx.Main

This will open a “JMX shell” with the following basic commands:

  • help: This shows a basic help screen that explains the available commands.
  • jps/list: Like the jps tool from the JDK this command prints out all java processes with their process id.
  • connect: You can use this command to connect to a running JVM process.
  • format: Let’s you specify whether you want your output in a simple text format or as a JSON string.
  • exit: Quits the application.

To learn more about cjmx let us start a session and connect to the JVM that is running cjmx itself:

> jps
13198 cjmx.Main
> connect 13198
Connected to local virtual machine 13198
Connection id: rmi://0:0:0:0:0:0:0:1  2
Default domain: DefaultDomain
5 domains registered consisting of 19 total MBeans
>
describe     disconnect   exit         format       help         invoke       mbeans       names        names        sample       select       status

After the last appearance of > you see a great feature of cjmx: auto-completion. Every time you do not know which commands are available, you can just type [TAB] and cjmx will list them. This even works for MBean names as we will see.

Now that we are connected to our JVM we can let cjmx describe an available MBean. With auto-completion we can just start typing describe '[TAB] to retrieve a list of all available packages:

> describe '
:                     JMImplementation:     com.sun.management:   java.lang:            java.nio:             java.util.logging:

This way we can dig through the MBean names until we have found what we are looking for. In this example we are interested in the MBean ‘java.lang:type=OperatingSystem':

> describe 'java.lang:type=OperatingSystem'
Object name: java.lang:type=OperatingSystem
-------------------------------------------
Description: Information on the management interface of the MBean

Attributes:
  MaxFileDescriptorCount: long
  OpenFileDescriptorCount: long
  FreePhysicalMemorySize: long
  CommittedVirtualMemorySize: long
  FreeSwapSpaceSize: long
  ProcessCpuLoad: double
  ProcessCpuTime: long
  SystemCpuLoad: double
  TotalPhysicalMemorySize: long
  TotalSwapSpaceSize: long
  AvailableProcessors: int
  Arch: String
  SystemLoadAverage: double
  Name: String
  Version: String
  ObjectName: ObjectName

As we can see, the MBean ‘java.lang:type=OperatingSystem’ provides information about the number of open files and the current CPU load, etc.. So let’s query the number of open files by invoking the command mbeans with the name of the MBean as well as the sub-command select and the MBean’s attribute:

> mbeans 'java.lang:type=OperatingSystem' select OpenFileDescriptorCount
java.lang:type=OperatingSystem
------------------------------
  OpenFileDescriptorCount: 35

We can even query all available attributes by using the star instead of the concrete name of an attribute. Please note that using the cursor up key recalls the last issued command, hence we do not have to type it again. Instead we just replace the attribute’s name with the star:

> mbeans 'java.lang:type=OperatingSystem' select *
java.lang:type=OperatingSystem
------------------------------
  MaxFileDescriptorCount: 10240
  OpenFileDescriptorCount: 36
...

By using the sub-command invoke we can even invoke MBean methods like in the following example:

> mbeans 'java.lang:type=Memory' invoke gc()
java.lang:type=Memory: null

Now that we know how to query attributes and invoke methods, we can start to script this functionality in order to monitor the application. To support this kind of scripting, cjmx provides the feature that one can pass all “commands” also as an argument to the application itself, hence you can invoke cjmx in the following way (where <PID> has to be replaced by a concrete process id of a running JVM):

java -cp $JAVA_HOME/lib/tools.jar:cjmx_2.10-2.1.0-app.jar cjmx.Main &amp;amp;lt;PID&amp;amp;gt; &amp;amp;quot;mbeans 'java.lang:type=OperatingSystem' select OpenFileDescriptorCount&amp;amp;quot;
java.lang:type=OperatingSystem
------------------------------
  OpenFileDescriptorCount: 630

With this knowledge we can write a simple bash script that queries the JVM each second for the number of open files:

#!/bin/bash
while [ true ] ; do
        echo `date` | tr -d '\n'
        java -cp /usr/java/default/lib/tools.jar:cjmx_2.10-2.1.0-app.jar cjmx.Main $1 &amp;amp;quot;mbeans 'java.lang:type=OperatingSystem' select OpenFileDescriptorCount&amp;amp;quot;|grep OpenFileDescriptorCount|cut -f 2 -d :
		sleep 1
done

This produces each second a new line with a timestamp and the the current number of open files. When redirected into a file, we have a simple log file and can evaluate it later on.

Conclusion: cjmx is a great alternative to JConsole when the latter cannot be used due to network restrictions on a server machine. The ability to even issue commands by passing them on the command line makes it suitable for small monitoring scripts.

Writing an interpreter in Ceylon

Ceylon is a statically typed language for the Java Virtual Machine (JVM) that comes with a powerful set of features that the Java language does not have or is missing (depending on your point of view). However, about three years after the first release and about one year after the version 1.0.0 release it is time to take a closer look.

A good way to get in touch with a new language is to start with a simple project, just for fun. One thing that came into my mind was to implement an interpreter for the programming language Brainfuck. The language consists of only eight simple commands and one instruction pointer. More details about Brainfuck can be found here.

This first surprising point about Ceylon is that we do not have the typically “main” method we know from other C-like languages. We can just implement a “global” method (here we call it “run”):

"Run the module `brainfuck`."
shared void run() {
	String? programFile = process.namedArgumentValue("program");
	if (exists programFile) {
		value bytes = readFile(programFile);
		BrainFuckInterpreter interpreter = BrainFuckInterpreter(bytes);
		interpreter.run();
	} else {
		print("Please provide the argument 'program'.");
	}
}

The string before the method definition is just the short form for the doc(“…”) annotation, the counterpart of the javadoc feature. To format the comments we can use the means of Markdown formatting.

Another surprising fact is the access modifier “shared” that we see in this example. Instead of using public in order to make the method accessible by other code, we use the keyword shared. In Ceylon the visibility hierarchy starts with modules that consist of one or more packages. A package consists of one or more source files, which again can share their elements. Sharing an element from one level of the hierarchy means to make it visible to the next higher level. This way even packages can be invisible to other modules, as we see in the following snippet from the package.celyon file, which is placed within each package and which declares in our case the package brainfuck as shared:

shared package brainfuck;

By the way: The package declarations that all Java classes start with are not necessary in Ceylon. Instead the package name is derived from the names of the folders the source file resides in, hence the redundant information can be left out. But let’s go back to the run() method above. As we do not have a special method which gets the command line arguments from the runtime environment passed in, we ask the global process instance for the named argument “program”. The returned String is an “optional” String, i.e. the value may be or may not be set. In Java that would be modelled by a String reference that may be null. But in order to circumvent possible NullPointerExceptions, the compiler ensures that you can only access the value of an “optional” references when you have checked them using the exists keyword.

Now that we have retrieved the path to our brainfuck “code”, we can read the bytes from the file using the following method:

ArrayList<Byte> readFile(String programFile) {
	value bytes = ArrayList<Byte>();
	print("Reading file '" + programFile + "'.");
	Path path = parsePath(programFile);
	OpenFile openfile = newOpenFile(path.resource);
	ByteBuffer buffer = newByteBuffer(1024);
	variable Integer bytesRead = openfile.read(buffer);
	print("bytesRead=" + bytesRead.string);
	while (bytesRead != -1) {
		buffer.flip();
		for (Byte byte in buffer) {
			bytes.add(byte);
		}
		buffer.clear();
		bytesRead = openfile.read(buffer);
	}
	openfile.close();
	return bytes;
}

As a storage for the bytes read from the program file, we utilize an ArrayList from the Java SDK. In order to do that, we have to import them in our module definition. This one resides in our module.celyon file:

module brainfuck "1.0.0" {
	shared import ceylon.io "1.1.0";
	shared import java.base "8";
}

Next to the name of the module and its version we can import other modules. Line 3 actually imports the java “base” module, i.e. the Java packages java.lang, java.util, java.io, java.net, java.text, NIO and security. Now that we have imported the java.base module, we can import the ArrayList at the beginning of our source file:

import java.util {
	ArrayList
}

Interestingly Ceylon emphasizes immutability, hence you can assign a value to every reference per default only once. If you want to change its value, you have to mark the reference as mutable using the keyword variable. You can see this in the code of the readFile() method as the reference to path, openfile and buffer are not marked as variable. In these cases the value is only assigned once. In contrast to that the number of bytes read from the file is of course variable, hence we mark it so.

After having read the code of the brainfuck program into memory, we can start its execution. This is done by creating an instance of the Brainfuck interpreter and calling its run() method:

BrainFuckInterpreter interpreter = BrainFuckInterpreter(bytes);
interpreter.run();

The beginning of this class is shown in the following:

shared class BrainFuckInterpreter(ArrayList<Byte> buffer) {
	ArrayList<Integer> field = ArrayList<Integer>();
	variable Integer ptr = 0;
	
	shared void run() {
		variable Integer bufferPos = 0;
		while (bufferPos >= 0 && bufferPos < buffer.size() - 1) {
			value oldBufferPos = bufferPos;
			value byte = buffer.get(bufferPos);
			switch (byte.unsigned)
			case (43) {
				ensureFieldCapacity();
				value oldValue = field.get(ptr);
				field.set(ptr, oldValue + 1);
			}
			...
			else {
				print("Ignoring char " + byte.string);
			}
			bufferPos++;
		}
		...
	}
	...
}

If you are used to Java, you might be missing the constructor. Where do we assign the passed in argument buffer to the corresponding member variable? The Ceylon feature here is that we can define some of our member variables within the braces after the class name. In our case we define a member with the name buffer of type ArrayList. The constructor we would have to write in Java that assigns the given value to the member variable is obsolete in Ceylon. The compiler creates some kind of “default” constructor that does this job for us. Additional members of the class, that are not passed to the constructor, can be defined as we would do that in Java (see field and ptr in our example). And as long as we do not declare them to be shared, they are only visible to the class members. Defining the “signature” of the constructor already within the class name also means that we can have only one constructor per class. In case we also want to execute additional code during the “construction”, we can put that code into the class body.

Our run() method now iterates of the buffer with the instructions that we have read from the file. Depending on which value the current byte has, we execute the appropriate operation. This is implemented by a switch statement. In contrast to Java we can omit the curly braces around all cases, but we cannot omit the “default” case, which is modelled in Ceylon as “else” clause of the switch statement. In this “else” case we just print the input byte that we are ignoring. Interesting about this point for Java developers is the fact that we cannot just “add” the byte to the previous String, as we would do in Java. Instead we have to access the String representation of the byte by writing byte.string. An implicit invocation of a method like toString() in Java does not exist in Ceylon.

Conclusion: The interpreter does not utilize all language features, but at least it covers a few interesting points. As someone who has worked with Java for years, the Ceylon features seem to tackle many well known issues of the Java language (versioned modules, package visibility, immutability, optional pattern, etc.). Although features like the absence of multiple constructors is a thing you have to get used to, I must admit that I liked to work with Ceylon.

PS: The sources can be found here.

Follow

Get every new post delivered to your Inbox.