Tag Archive | jboss

Using @NamedEntityGraph to load JPA entities more selectively in N+1 scenarios

The N+1 problem is a common issue when working with ORM solutions. It happens when you set the fetchType for some @OneToMany relation to lazy, in order to load the child entities only when the Set/List is accessed. Let’s assume we have a Customer entity with two relations: a set of orders and a set of addresses for each customer.

@OneToMany(mappedBy = "customer", cascade = CascadeType.ALL, fetch = FetchType.LAZY)
private Set<OrderEntity> orders;

@OneToMany(mappedBy = "customer", cascade = CascadeType.ALL, fetch = FetchType.LAZY)
private Set<AddressEntity> addresses;

To load all customers, we can issue the following JPQL statement and afterwards load all orders for each customer:

List<CustomerEntity> resultList = entityManager.createQuery("SELECT c FROM CustomerEntity AS c", CustomerEntity.class).getResultList();
for(CustomerEntity customerEntity : resultList) {
    Set<OrderEntity> orders = customerEntity.getOrders();
    for(OrderEntity orderEntity : orders) {
	...
    }
}

Hibernate 4.3.5 (as shipped with JBoss AS Wildfly 8.1.0CR2) will generate the following series of SQL statements out of it for only two(!) customers in the database:

Hibernate: 
     select
         customeren0_.id as id1_1_,
         customeren0_.name as name2_1_,
         customeren0_.numberOfPurchases as numberOf3_1_ 
     from
         CustomerEntity customeren0_
Hibernate: 
     select
         orders0_.CUSTOMER_ID as CUSTOMER4_1_0_,
         orders0_.id as id1_2_0_,
         orders0_.id as id1_2_1_,
         orders0_.campaignId as campaign2_2_1_,
         orders0_.CUSTOMER_ID as CUSTOMER4_2_1_,
         orders0_.timestamp as timestam3_2_1_ 
     from
         OrderEntity orders0_ 
     where
         orders0_.CUSTOMER_ID=?
Hibernate: 
     select
         orders0_.CUSTOMER_ID as CUSTOMER4_1_0_,
         orders0_.id as id1_2_0_,
         orders0_.id as id1_2_1_,
         orders0_.campaignId as campaign2_2_1_,
         orders0_.CUSTOMER_ID as CUSTOMER4_2_1_,
         orders0_.timestamp as timestam3_2_1_ 
     from
         OrderEntity orders0_ 
     where
         orders0_.CUSTOMER_ID=?

As we can see, the first query selects all customers from the table CustomerEntity. The following two selects fetch then the orders for each customer we have loaded in the first query. When we have 100 customers instead of two, we will get 101 queries. One initial query to load all customers and then for each of the 100 customers an additional query for the orders. That is the reason why this problem is called N+1.

A common idiom to solve this problem is to force the ORM to generate an inner join query. In JPQL this can be done by using the JOIN FETCH clause like demonstrated in the following code snippet:

entityManager.createQuery("SELECT c FROM CustomerEntity AS c JOIN FETCH c.orders AS o", CustomerEntity.class).getResultList();

As expected the ORM now generates an inner join with the OrderEntity table and therewith only needs one SQL statement to load all data:

select
    customeren0_.id as id1_0_0_,
    orders1_.id as id1_1_1_,
    customeren0_.name as name2_0_0_,
    orders1_.campaignId as campaign2_1_1_,
    orders1_.CUSTOMER_ID as CUSTOMER4_1_1_,
    orders1_.timestamp as timestam3_1_1_,
    orders1_.CUSTOMER_ID as CUSTOMER4_0_0__,
    orders1_.id as id1_1_0__
from
    CustomerEntity customeren0_
inner join
    OrderEntity orders1_
        on customeren0_.id=orders1_.CUSTOMER_ID

In situations where you know that you will have to load all orders for each customer, the JOIN FETCH clause minimizes the number of SQL statements from N+1 to 1. This comes of course with the drawback that you now transfer for all orders of one customer the customer data again and again (due to the additional customer columns in the query).

The JPA specification introduces with version 2.1 so called NamedEntityGraphs. This annotation lets you describe the graph a JPQL query should load in more detail than a JOIN FETCH clause can do and therewith is another solution to the N+1 problem. The following example demonstrates a NamedEntityGraph for our customer entity that is supposed to load only the name of the customer and its orders. The orders are described in the subgraph ordersGraph in more detail. Here we see that we only want to load the fields id and campaignId of the order.

@NamedEntityGraph(
        name = "CustomersWithOrderId",
        attributeNodes = {
                @NamedAttributeNode(value = "name"),
                @NamedAttributeNode(value = "orders", subgraph = "ordersGraph")
        },
        subgraphs = {
                @NamedSubgraph(
                        name = "ordersGraph",
                        attributeNodes = {
                                @NamedAttributeNode(value = "id"),
                                @NamedAttributeNode(value = "campaignId")
                        }
                )
        }
)

The NamedEntityGraph is given as a hint to the JPQL query, after it has been loaded via EntityManager using its name:

EntityGraph entityGraph = entityManager.getEntityGraph("CustomersWithOrderId");
entityManager.createQuery("SELECT c FROM CustomerEntity AS c", CustomerEntity.class).setHint("javax.persistence.fetchgraph", entityGraph).getResultList();

Hibernate supports the @NamedEntityGraph annotation since version 4.3.0.CR1 and creates the following SQL statement for the JPQL query shown above:

Hibernate: 
    select
        customeren0_.id as id1_1_0_,
        orders1_.id as id1_2_1_,
        customeren0_.name as name2_1_0_,
        customeren0_.numberOfPurchases as numberOf3_1_0_,
        orders1_.campaignId as campaign2_2_1_,
        orders1_.CUSTOMER_ID as CUSTOMER4_2_1_,
        orders1_.timestamp as timestam3_2_1_,
        orders1_.CUSTOMER_ID as CUSTOMER4_1_0__,
        orders1_.id as id1_2_0__ 
    from
        CustomerEntity customeren0_ 
    left outer join
        OrderEntity orders1_ 
            on customeren0_.id=orders1_.CUSTOMER_ID

We see that Hibernate does not issue N+1 queries but that instead the @NamedEntityGraph annotation has forced Hibernate to load the orders per left outer join. This is of course a subtle difference to the FETCH JOIN clause, where Hibernate created an inner join. The left outer join would also load customers for which no order exists in contrast to the FETCH JOIN clause, where we would only load customers that have at least one order.

Interestingly is also that Hibernate loads more than the specified attributes for the tables CustomerEntity and OrderEntity. As this conflicts with the specification of @NamedEntityGraph (section 3.7.4) I have created an JIRA issue for that.

Conclusion: We have seen that with JPA 2.1 we have two solutions for the N+1 problem: We can either use the FETCH JOIN clause to eagerly fetch a @OneToMany relation, which results in an inner join, or we can use @NamedEntityGraph feature that lets us specify which @OneToMany relation to load via left outer join.

Advertisements

Tracing SQL statements in JBoss AS 7 using a custom logging handler

Using an ORM to abstract from your specific database and to let it create and issue all the SQL statements you would have to write by hand yourself seems handy. This is what made ORM solutions popular.

But it also comes with a downside: As the ORM does a lot of work for you, you lose to some degree control over the generated SQL and you have to rely on the ORM to create a high-performance statement for you. But it can happen that the SQL generated by the ORM is not what you might have written by hand and expected the ORM to do for you. In this case you have to get back control over the SQL and put your hands on the code again.

In huge applications this task is not as trivial, as there might be hundreds of statements issued to the database that stem from hundreds of lines of Java code that makes heavy usage of JPA features. Tracing the SQL statement that your database profiling tool has identified as problematic down to the actual code line becomes tedious.

We know that we can enable SQL statement logging for Hibernate with the following two lines in our persistence.xml:

<property name="hibernate.show_sql" value="true"/>
<property name="hibernate.format_sql" value="true"/>

But this will only output the already generated SQL; the actual Java code line is still not visible. For smaller applications it might be feasible to attach a debugger to the application server and debug through the code until you have found the line that logs the problematic SQL statement, but for bigger applications this is time consuming.

As Hibernate itself does not provide any means of intercepting the logging and enhance it with more information, we will have to do this on our own. The JBoss documentation indicates that it is possible to write your own custom logging handler. As this logging handler receives all the logging messages and therewith also the messages produces by Hibernate with enabled SQL logging, we can try to find the line we are looking for and then output a stack trace to our own log file.

Writing a custom logging handler turns out to be very simple. All you have to do is setup a small project with a class that extends the class Handler from the JDK package java.util.logging:

package mypackage;

import java.util.logging.Handler;
import java.util.logging.LogRecord;

public class MyJBossLogger extends Handler {

	@Override
	public void publish(LogRecord record) {
	
	}
	
	@Override
	public void flush() {
	}
	
	@Override
	public void close() throws SecurityException {

	}
}

The publish() method receives all logging output in form of an instance of LogRecord. Its method getMessage() lets us access the output directly. Hence we can match this message against some keywords we have loaded from some configuration file:

@Override
public void publish(LogRecord record) {
	String message = record.getMessage();
	buffer.add(message + "\n");
	if (keywords == null) {
		keywords = loadKeywords();
	}
	if (matches(message, keywords)) {
		String stacktrace = "\nStacktrace:\n";
		StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();
		for (StackTraceElement element : stackTrace) {
			stacktrace += element.toString() + "\n";
		}
		buffer.add(stacktrace);
		flush();
	}
}

Buffer is here some simple data structure (e.g. guava’s EvictingQueue) that buffers the last lines, as the method publish() is called for each line(!) of output. As a complete SQL statement spans more than one line, we have to remember a couple of them. Next to the buffered lines and the current line we also output a String representation of the current stack trace. This tells us later in the log file from where we are called and therewith which line of Java code in our project causes the current statement.

Once we have compiled the project we can copy the resulting jar file to the newly created folder structure under: $JBOSS_HOME/modules/system/layers/base/com/mydomain/mymodule/main (for JBoss AS 7.2). In order to tell JBoss AS about our new module, we have to create a XML file called module.xml with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.1" name="com.mydomain.mymodule">
	<resources>
		<resource-root path="MyJBossLogger-0.0.1-SNAPSHOT.jar"/>
	</resources>
</module>

The name of the module corresponds to the path within the JBoss modules folder. It will also be used in the configuration file to configure our custom logging handler:

...
<subsystem xmlns="urn:jboss:domain:logging:1.2">
	<custom-handler name="CUSTOM" module="com.mydomain.mymodule" class="com.mydomain.mymodule.MyJBossLogger">
		<level name="DEBUG"/>
	</custom-handler>
	...

When we implement the flush() method of our logging handler to write the output to some log file, we will see something like the following (of course in condensed form):

Hibernate:     select ... from customer ...
Stacktrace:
java.lang.Thread.getStackTrace(Thread.java:1568)
com.mydomain.mymodule.MyJBossLogger.publish(MyJBossLogger.java:20)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:292)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:300)
org.jboss.logmanager.Logger.logRaw(Logger.java:721)
org.jboss.logmanager.Logger.log(Logger.java:506)
...
com.mydomain.myapp.ArticleEntity.getCustomers(ArticleRepository.java:234)
...

Here we can see clearly which OneToMany relation causes the problematic select statement we were looking for.

Conclusion: Using a custom logging handler to inject the current stack trace into the logging of the SQL statements may help you when you want to find the
exact location in the source code where a concrete query is issued. It turned also out that writing your own custom logging handler for JBoss AS is also a straight forward task.

Securing a JSF application with Java EE security and JBoss AS 7.x

A common requirement for enterprise applications is to have all JSF pages protected behind a login page. Sometimes you even want to have protected areas inside the application that are only accessible by users that own a specific role. The Java EE standards come with all the means you need to implement a web application that is protected by some security constraints. In this blog post we want to develop a simple application that demonstrates the usage of these means and shows how you can build a complete JSF application for two different roles. As the solution might look straight forward at first glance, there are a few pitfalls one has to pay attention to.

The first point we have to care about, is the folder layout of our application. We have three different kinds of pages:

  • The login page and the error page for the login should be accessible by all users.
  • We have a home page that should only be accessible for authenticated users.
  • We have a protected page that should only be visible for users of the role protected-role.

These three types of pages are therefore put into three different folders: the root folder src/main/webapp, the folder src/main/webapp/pages and the protected page resides in src/main/webapp/pages/protected:

security-web-ide

The web.xml file is the place to define which roles we want to use and how to map the accessibility of some URL pattern to these roles:

<security-constraint>
    <web-resource-collection>
        <web-resource-name>pages</web-resource-name>
        <url-pattern>/pages/*</url-pattern>
        <http-method>PUT</http-method>
        <http-method>DELETE</http-method>
        <http-method>GET</http-method>
        <http-method>POST</http-method>
    </web-resource-collection>
    <auth-constraint>
        <role-name>security-role</role-name>
        <role-name>protected-role</role-name>
    </auth-constraint>
</security-constraint>
<security-constraint>
    <web-resource-collection>
        <web-resource-name>protected</web-resource-name>
        <url-pattern>/pages/protected/*</url-pattern>
        <http-method>PUT</http-method>
        <http-method>DELETE</http-method>
        <http-method>GET</http-method>
        <http-method>POST</http-method>
    </web-resource-collection>
    <auth-constraint>
        <role-name>protected-role</role-name>
    </auth-constraint>
</security-constraint>

<security-role>
    <role-name>security-role</role-name>
</security-role>
<security-role>
    <role-name>protected-role</role-name>
</security-role>

As you can see we define two roles: security-role and protected-role. URLs matching the pattern /pages/* are only accessible by users that own the roles security-role and protected-role, whereas the pages under /pages/protected/* are restricted to users with the role protected-role.

Another point you may stumble upon is the welcome page. At first guess you would want to the specify the login page as welcome page. But this does not work, as the login module of the servlet container automatically redirects all unauthorized accesses to the login page. Therefore we specify the home page of our application as welcome page. This is already a protected page, but the user will get redirected to the login page automatically when he calls its URL directly.

<welcome-file-list>
    <welcome-file>pages/home.xhtml</welcome-file>
</welcome-file-list>

Now we are nearly done with the web.xml page. All we to do is to define the authentication method as well as the login page and the error page, which is shown in case the user enters invalid credentials. One has to page attention that both pages don’t include protected URLs (e.g. CSS or JavaScript files), otherwise even the access to these two pages is forbidden and the user gets an Application Server specific error page to see.

<login-config>
    <auth-method>FORM</auth-method>
    <form-login-config>
        <form-login-page>/login.xhtml</form-login-page>
        <form-error-page>/error.xhtml</form-error-page>
    </form-login-config>
</login-config>

As we are going to deploy the application to the JBoss Application Server, we provide a file named jboss-web.xml that connects our application to a security-domain:

<?xml version="1.0" encoding="UTF-8"?>
<jboss-web>
    <security-domain>java:/jaas/other</security-domain>
</jboss-web>

The “other” security-domain is configured inside the standalone.xml. The default configuration requires that the user passes the “RealmUsersRoles” login module, which gets its user and role definitions from the two files application-users.properties and application-roles.properties inside the configuration folder. You can use the provided add-user script to add a new user to this realm:

What type of user do you wish to add?
 a) Management User (mgmt-users.properties)
 b) Application User (application-users.properties)
(a): b

Enter the details of the new user to add.
Realm (ApplicationRealm) :
Username : bart
Password :
Re-enter Password :
What roles do you want this user to belong to? (Please enter a comma separated list, or leave blank for none) : security-role,protected-role
About to add user 'bart' for realm 'ApplicationRealm'
Is this correct yes/no? yes

Here it is important to choose the correct realm (ApplicationRealm), as this realm is configured by default in the standalone.xml for the “other” login module. This is also the place where you provide the roles the user possesses as a comma separated list.

<form method="POST" action="j_security_check" id="">
	<h:panelGrid id="panel" columns="2" border="1" cellpadding="4" cellspacing="4">
		<h:outputLabel for="j_username" value="Username:" />
		<input type="text" name="j_username"/>
		<h:outputLabel for="j_password" value="Password:" />
		<input type="password" name="j_password"/>
		<h:panelGroup>
			<input type="submit" value="Login"/>
		</h:panelGroup>
	</h:panelGrid>
</form>

The next step is to implement a simple login form that submits its data to the login module. Note the ids of the input fields as well as the action of the form. This way the form is posted to the login module, which extracts the entered username and password from the request. JSF developers may wonder why we use a standard HTML form instead of the element. The reason for this is that the JSF form elements spans its own namespace and therefore the ids of the input fields are prefixed with the id of the form (and this form id cannot be empty).

If the user has passed the login form, we present him a home page. But the link to the protected page should only be accessible for users that own the role protected-role. This can be accomplished by the following rendered condition:

<h:link value="Protected page" outcome="protected/protected" rendered="#{facesContext.externalContext.isUserInRole('protected-role')}"/>

Last but not least we need the logout functionality. For this case we implement a simple backing bean like the following one that invalidates the user’s session and redirects him back to the login page:

@Named(value = "login")
public class Login {

    public String logout() {
        FacesContext.getCurrentInstance().getExternalContext().invalidateSession();
        return "/login";
    }
}

As usual the complete source code can be found on github.

Building and testing a websocket server with undertow

The upcoming version of JBoss Application Server will no longer use Tomcat as integrated webserver, but will replace it with undertow. The architecture of undertow is based on handlers that can be added dynamically via a Builder API to the server. This approach is similar to the way of constructing a webserver in Node.js. It allows developers to embed the undertow webserver easily into their applications. As the addition of features is done via the Builder API, one can only add the features that are really required in one’s application. Beyond that undertow supports WebSockets and the Servlet API in version 3.1. It can be run as blocking or non-blocking server and it is said, that first tests have proven that undertow is the fastest webserver written in Java.

As all of this sounds very promising, so let’s try to set up a simple websocket server. As usual we start by creating a simple java project and add the undertow maven dependency:

<dependency>
  <groupId>io.undertow</groupId>
  <artifactId>undertow-core</artifactId>
  <version>1.0.0.Beta20</version>
</dependency>

With undertow’s Builder API our buildAndStartServer() method looks like this:

public void buildAndStartServer(int port, String host) {
    server = Undertow.builder()
            .addListener(port, host)
            .setHandler(getWebSocketHandler())
            .build();
    server.start();
}

We just add a listener that specifies the port and host to listen for incoming connections and afterwards add a websocket handler. As the websocket handler code is a little bit more comprehensive, I have put it into its own method:

private PathHandler getWebSocketHandler() {
    return path().addPath("/websocket", websocket(new WebSocketConnectionCallback() {
        @Override
        public void onConnect(WebSocketHttpExchange exchange, WebSocketChannel channel) {
            channel.getReceiveSetter().set(new AbstractReceiveListener() {
                @Override
                protected void onFullTextMessage(WebSocketChannel channel, BufferedTextMessage message) {
                    String data = message.getData();
                    lastReceivedMessage = data;
                    LOGGER.info("Received data: "+data);
                    WebSockets.sendText(data, channel, null);
                }
            });
            channel.resumeReceives();
        }
    }))
    .addPath("/", resource(new ClassPathResourceManager(WebSocketServer.class.getClassLoader(), WebSocketServer.class.getPackage()))
            .addWelcomeFiles("index.html"));
}

Let’s go line by line through this code snippet. First of all we add a new path: /websocket. The second argument of the addPath() methods lets us specify what kind of protocol we want to use for this path. In our case we create a new WebSocket. The anonymous implementation has a onConnect() method in which we set an implementation of AbstractReceiveListener. Here we have a convenient method onFullTextMessage() that is called when a client has sent us a text message. A call of getData() fetches the actual message we have received. In this simple example we just echo this string back to client to validate that the roundtrip from the client to server and back works.

To perform some simple manual tests we also add a second resource under the path / which serves some static HTML and JavaScript files. The directory that contains these files is given as an instance of ClassPathResourceManager. The call of addWelcomeFiles() tells undertow which file to server when the client asks for the path /.

The index.html looks like this:

</pre>
<html>
<head><title>Web Socket Test</title></head>
<body>
  <script src="jquery-2.0.3.min.js"></script>
  <script src="jquery.gracefulWebSocket.js"></script>
  <script src="websocket.js"></script>
  <form onsubmit="return false;">
    <input type="text" name="message" value="Hello, World!"/>
    <input type="button" value="Send Web Socket Data" onclick="send(this.form.message.value)"/>
  </form>
  <div id="output"></div>
</body>
</html>
<pre>

Our JavaScript code is swapped out to the websocket.js file. We use jquery and the jquery-Plugin gracefulWebSocket to ease the client side development:

var ws = $.gracefulWebSocket("ws://127.0.0.1:8080/websocket");
ws.onmessage = function(event) {
    var messageFromServer = event.data;
    $('#output').append('Received: '+messageFromServer+'');
}

function send(message) {
    ws.send(message);
}

After having created a WebSocket object by calling $.gracefulWebSocket() we can register a callback function for incoming messages. In this method we only append the message string to the DOM of the page. The send() method is just a call to gracefulWebSocket’s send() method.

When we now start our application and open the URL http://127.0.0.1:8080/ in our webbrowser we see the following page:
undertow-websocket
Entering some string and hitting the “Send Web Socket Data” button sends the message to the server, which in response echos it back to the client.

Now that we know that everything works as expected, we want to protect our code against regression with a junit test case. As a websocket client I have chosen the library jetty-websocket:

<dependency>
    <groupId>org.eclipse.jetty</groupId>
    <artifactId>jetty-websocket</artifactId>
    <version>8.1.0.RC5</version>
    <scope>test</scope>
</dependency>

In the test case we build and start the websocket server to open a new connection to the websocket port. The WebSocket implementation of jetty-websocket allows us to implement two callback methods for the open and close events. Within the open callback we send the test message to the client. The rest of the code waits for the connection to be established, closes it and asserts that the server has received the message:

@Test
public void testStartAndBuild() throws Exception {
    subject = new WebSocketServer();
    subject.buildAndStartServer(8080, "127.0.0.1");
    WebSocketClient client = new WebSocketClient();
    Future connectionFuture = client.open(new URI("ws://localhost:8080/websocket"), new WebSocket() {
        @Override
        public void onOpen(Connection connection) {
            LOGGER.info("onOpen");
            try {
                connection.sendMessage("TestMessage");
            } catch (IOException e) {
                LOGGER.error("Failed to send message: "+e.getMessage(), e);
            }
        }
        @Override
        public void onClose(int i, String s) {
            LOGGER.info("onClose");
        }
    });
    WebSocket.Connection connection = connectionFuture.get(2, TimeUnit.SECONDS);
    assertThat(connection, is(notNullValue()));
    connection.close();
    subject.stopServer();
    Thread.sleep(1000);
    assertThat(subject.lastReceivedMessage, is("TestMessage"));
}

As usual you can find the source code on github.

Conclusion: Undertow’s Builder API makes it easy to construct a websocket server and in general an embedded webserver that fits your needs. This also eases automatic testing as you do not need any specific maven plugin that starts and stops your server before and after your integration tests. Beyond that the jquery plugin jquery-graceful-websocket lets you send and receive messages over websockets with only a few lines of code.

Testing HTML5 canvas applications with sikuli and arquillian

HTML5 introduces a great new element that can be used to draw arbitrary content on a pane: the canvas element. What has been a standard feature for fat client applications for decades is now introduced to the world of web applications. Web developers no longer need to use proprietary plugins to draw images or charts in their applications.

But when it comes for testing, this new feature imposes new challenges to the web development community. How to test that the canvas element is in an appropriate state at some point in time? Standard technologies like selenium focus on the markup that is generated by the web server and not on the pixels drawn on the canvas.

More promising in this field are technologies that use image processing to verify that an application renders its data correctly. One of these frameworks is sikuli. Sikuli is an open-source research project that was started at the MIT and is now maintained by Raimund Hocke.

To give a more practical introduction, let’s assume we have a simple web application that uses the HTML5 canvas element to implement some simple image processing functionality like a grayscale, a brighten and a threshold filter as well as an undo button (the code for this application can be found as usual on github):
html5-canvas-screenshot

The installation of sikuli is (of course) platform dependent. The installer, which can be downloaded from the sikuli download page, is a Java Swing application that asks you about your typical usage pattern. As we do not want to use the python IDE, we choose option 4 from the list of options. The actual jar file is then downloaded and prepared for our OS. After the installation process has finished, we find an OS dependent jar file within the installation directory. As our example project uses maven as build system, we have to introduce a system scope dependency after we have copied the library into the lib folder:

<dependency>
    <groupId>org.sikuli</groupId>
    <artifactId>sikuli</artifactId>
    <version>1.0</version>
    <scope>system</scope>
    <systemPath>${basedir}/lib/sikuli-java.jar</systemPath>
</dependency>

When sikuli is used for the first time, it extracts some native libraries into a new folder, here in our example into ${basedir}/lib/libs. This folder has to be added to the user’s path environment variable.

Now that we have installed sikuli, let’s setup arquillian so that we can write our first unit test. How to setup arquillian is described for example here. As I don’t want to repeat everything, in the following you will find only the unit test class:

@RunWith(Arquillian.class)
public class FilterTest {
    public static final String WEBAPP_SRC = "src/main/webapp";
    @ArquillianResource
    URL deploymentURL;
    private Screen screen;

    @Before
    public void before() throws URISyntaxException, IOException {
        screen = new Screen();
        if (Desktop.isDesktopSupported()) {
            Desktop.getDesktop().browse(deploymentURL.toURI());
        } else {
            fail();
        }
    }

    @Deployment
    public static WebArchive createDeployment() {
        return ShrinkWrap.create(WebArchive.class, "html5-sikuli-webapp.war")
                .addClasses(HomeBackingBean.class)
                .addAsWebResource(new File(WEBAPP_SRC, "home.xhtml"))
                .addAsWebResource(new File(WEBAPP_SRC, "resources/css/style.css"), "resources/css/style.css")
                .addAsWebResource(new File(WEBAPP_SRC, "resources/images/rom.jpg"), "resources/images/rom.jpg")
                .addAsWebResource(new File(WEBAPP_SRC, "resources/js/html5Sikuli.js"), "resources/js/html5Sikuli.js")
                .addAsWebResource(new File(WEBAPP_SRC, "resources/js/jquery-2.0.3.js"), "resources/js/jquery-2.0.3.js")
                .addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml")
                .setWebXML(new File(WEBAPP_SRC, "WEB-INF/web.xml"));
    }

The createDeployment() method sets up the war archive, which is deployed by arquillian to JBoss AS 7.1.1.Final (see arquillian.xml file). In our @Before method we use the SDK class Desktop to open the default browser and point it to the deployment URL. Here we also create an instance of sikuli’s class Screen. This class provides all methods needed to perform the interaction with our application. Let’s look at this in more detail:

@Test
@RunAsClient
public void testGrayScale() throws FindFailed {
    screen.wait(getFullPath("originalImage.png"));
    screen.find(getFullPath("btnUndo_disabled.png"));
    screen.click(getFullPath("btnGrayscale.png"));
    screen.find(getPattern("grayscaleImage.png", 0.9f));
    screen.click(getFullPath("btnUndo_enabled.png"));
    screen.click(getPattern("originalImage.png", 0.9f));
}

private Pattern getPattern(String path, float similarity) {
    Pattern p = new Pattern(getFullPath(path));
    return p.similar(similarity);
}

private String getFullPath(String path) {
    return "src/test/resources/img/" + path;
}

As sikuli is based on image processing, we can define where to click and what to verify with screenshots we have taken before. In this simple example I have stored all screenshots as png files within the src/test/resources/img folder of our project. More advanced projects may need a more sophisticated folder hierarchy. As you can see, we first wait for the application to show up. Once sikuli has found the first screenshot, we verify that the button “Undo” is disabled. This is done by calling the method find() with an image of the disabled button. Now we can click on the button “Grayscale” (again specified by an image of the button) and then verify that the grayscale version of the image is found on the screen.

Sikuli does not only compare both images pixel by pixel, but if you like it computes the similarity of the found screen region with the requested region. This helps when you need to be more tolerant (e.g. if you want to test the application in different browsers and these render the buttons slightly different). The default value for the similarity attribute is 0.7f, but if you increase it to 1.0f you have a simple pixel by pixel comparison.

But this is not all. With sikuli you can do nearly all things you could do as human interactor:

  • Type characters using screen.type()
  • Double click with screen.doubleClick()
  • Perform drag and drop operations with screen.dragDrop()
  • Use the mouse wheel

Conclusion: Sikuli is a powerful and easy to use tool to perform integration tests for web applications that heavily rely on HTML5’s canvas object. The same is of course true for standard fat client applications (Swing, JavaFX). Together with arquillian you can setup comprehensive test suites that cover a lot of “real” use cases.

Setting up your application server with maven

In many cases there is no way to deploy an application without the need to setup your application before. In JBoss AS 7.x you may want to configure for example your database connection. Or you have to configure a security realm. Maybe you also want to adjust the SLSB pool… In any of these cases all developers in the team have to share a common or at least a similar configuration.
Often this information can be found in sporadically sent emails or on some wiki page. But what happens sometime after a release, when you have to check out a branch to fix some bug or to add a new feature? You will have to reconstruct the configuration that was valid for this branch. So why not add the configuration files to your version control system and together with the mere configuration files, a maven configuration that sets up the whole application server?
Let’s try to keep it simple and use only public available and commonly used plugins. First of all let’s add all versions that we will need in the following to the properties part of the pom.xml:

	<properties>
		<jboss.install.dir>${project.build.directory}/jboss</jboss.install.dir>
		<jboss.version>7.2.0.Final</jboss.version>
		<app.version>${project.version}</app.version>
		<ojdbc.version>11.2.0.1.0</ojdbc.version>
	</properties>

We also define here the installation directory of the JBoss AS. This way we can change it, if we want, using the command line option -D. Now we add a new profile, such that we have to explicitly switch the setup procedure on and that it is not part of the normal build:

<profile>
	<id>setupAs</id>
	<build>
		<plugins>
		...
				</plugins
		</build>
</profile>

If we have the current JBoss version as a maven artifact deployed in our maven repository, we can use the maven-dependency-plugin to download and unpack the JBoss to the installation directory given above:

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-dependency-plugin</artifactId>
	<version>2.8</version>
	<executions>
		<execution>
			<id>unpack-jboss</id>
			<phase>package</phase>
			<goals>
				<goal>unpack</goal>
			</goals>
			<configuration>
				<artifactItems>
					<artifactItem>
						<groupId>org.jboss</groupId>
						<artifactId>jboss-as</artifactId>
						<version>${jboss.version}</version>
						<type>zip</type>
						<outputDirectory>${project.build.directory}/jboss</outputDirectory>
					</artifactItem>
				</artifactItems>
			</configuration>
		</execution>

Now that the application server is unpacked, we have to add the JDBC driver as well as our application (or anything else you need). We set this up by adding another execution block to the maven dependency plugin:

<execution>
	<id>copy</id>
	<phase>package</phase>
	<goals>
		<goal>copy</goal>
	</goals>
	<configuration>
		<artifactItems>
			<artifactItem>
				<groupId>our-company</groupId>
				<artifactId>our-application-ear</artifactId>
				<version>${app.version}</version>
				<type>ear</type>
				<outputDirectory>${jboss.install.dir}/jboss-as-${jboss.version}/standalone/deployments</outputDirectory>
			</artifactItem>
			<artifactItem>
				<groupId>com.oracle</groupId>
				<artifactId>ojdbc6</artifactId>
				<version>${ojdbc.version}</version>
				<outputDirectory>${jboss.install.dir}/jboss-as-${jboss.version}/standalone/deployments</outputDirectory>
				<destFileName>ojdbc6.jar</destFileName>
			</artifactItem>
		</artifactItems>
	</configuration>
</execution>

Last but not least, we also want to adjust the standard configuration files to our needs. We can use the maven-resources-plugin to substitute variable values within each file. Therefore we add templates for these files to the resources folder of our JBoss module and call the goal copy-resources:

<plugin>
	<artifactId>maven-resources-plugin</artifactId>
	<version>2.6</version>
	<executions>
		<execution>
			<id>copy-jboss-configuration</id>
			<phase>package</phase>
			<goals>
				<goal>copy-resources</goal>
			</goals>
			<configuration>
				<outputDirectory>${jboss.install.dir}/jboss-as-${jboss.version}/standalone/configuration</outputDirectory>
				<resources>
					<resource>
						<directory>src/main/resources/jboss/standalone/configuration</directory>
						<filtering>true</filtering>
					</resource>
				</resources>
			</configuration>
		</execution>
		<execution>
			<id>copy-jboss-bin</id>
			<phase>package</phase>
			<goals>
				<goal>copy-resources</goal>
			</goals>
			<configuration>
				<outputDirectory>${jboss.install.dir}/jboss-as-${jboss.version}/bin</outputDirectory>
				<resources>
					<resource>
						<directory>src/main/resources/jboss/bin</directory>
						<filtering>true</filtering>
					</resource>
				</resources>
			</configuration>
		</execution>
	</executions>
</plugin>

The values for the filtering can be given on the command line with the -D option. If the team has more than a few members, it is also possible to create for each user a properties file that contains his/her specific configuration values. If we use the OS user as filename, we can easily choose the file by the name of the currently logged in user. This way each team member can easily setup its own completely configured application server instance by simply running:

mvn clean install -PsetupAs

In order to prevent that the newly configured server is deleted with the next clean invocation, we disable the maven clean plugin for the normal build:

<plugin>
	<artifactId>maven-clean-plugin</artifactId>
	<version>2.5</version>
	<configuration>
		<skip>false</skip>
	</configuration>
</plugin>

Within the setupAs profile created above, we have to enable it of course, such that we can delete the whole installation just by calling “mvn clean -PsetupAs”. Now switching to an older branch is easy as we don’t lose any time searching for the right configuration…

JNDI load balancing with jboss-ejb-client on JBoss AS 7

Let’s assume we have a client application that accesses a remote stateful session bean (SFSB) on a JBoss Application Server AS 7. The SFSB is accessed via JNDI lookup as described here: EJB invocations via JNDI. The SFSB is clustered via the @Clustered annotation as described here: Clustered EJBs.

@Stateful
@Clustered
@Remote(TestRemote.class)
public class TestBean implements TestRemote {
...
}

Note that the infinispan cache used to cluster the SFSB is only started, when during the deployment one SFSB with the @Clustered annotation is found. If we want to start two JBoss servers on the same machine, we can do this with the following command line invocations:

standalone.bat -c standalone-ha.xml -Djboss.node.name=nodeA -b 127.0.0.1
standalone.bat -c standalone-ha.xml -Djboss.socket.binding.port-offset=100 -Djboss.node.name=nodeB -b 127.0.0.1

It is important to note, that both servers have to be bind to a specific ip address. If you bind both server with the option “-b 0.0.0.0” the clustering doesn’t start (see here). Both servers also do have to have a different node name.

The client uses the following properties file to access the SFSB via JNDI:

remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
invocation.timeout=10000
remote.connections=default

remote.connection.default.host=127.0.0.1
remote.connection.default.port=4447
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false

remote.clusters=ejb
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false
remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false

Two things are important. First of all we only have to define one of the two servers (here 127.0.0.1:4447). The other server is detected automatically via a topology message that the client receives after he has connected to the first server. It is also important to mention that this topology information is received with some latency, thus if you try to lookup all your SFSBs directly after the first lookup, your client program could be to fast to integrate the information about the second server and therefore all SFSB are executed on the first server. The name of the cluster has also to be defined (here with the property remote.clusters). Then for each defined cluster (here ejb) the SASL_POLICY as well as the SSL configuration is given.

If you now look up the remote bean with the following code, all invocations are load balanced to both server instances:

	private TestRemote lookupRemoteBean() throws NamingException {
		logger.info("Using jboss-ejb-client.");
		final Hashtable<String, String> jndiProperties = new Hashtable<String, String>();
		jndiProperties.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
		final Context context = new InitialContext(jndiProperties);
		final String appName = "jboss-ejb-client";
		final String moduleName = "server-ejb";
		final String distinctName = "";
		final String beanName = TestBean.class.getSimpleName();
		final String viewClassName = TestRemote.class.getName();
		String lookupString = "ejb:" + appName + "/" + moduleName + "/" + distinctName + "/" + beanName + "!"
				+ viewClassName + "?stateful";
		logger.debug(String.format("Looking up: %s", lookupString));
		return (TestRemote) context.lookup(lookupString);
	}

The appName and moduleName are chosen as described here.

There is also one more caveat: You can set a selector for the EJB client context to use programmatically:

final EJBClientConfiguration ejbClientConfiguration = new PropertiesBasedEJBClientConfiguration(
				clientConfigProps);
final ContextSelector<EJBClientContext> ejbClientContextSelector = new ConfigBasedEJBClientContextSelector(
				ejbClientConfiguration);
EJBClientContext.setSelector(ejbClientContextSelector);

Here the properties object clientConfigProps is created dynamically during runtime and contains in our example the same information as the properties file from above. If you set this selector directly before each lookup, the topology information is requested again and arrives too late, due to the latency mentioned before. Therefore again all clients are executed on the first server.