Saturday, October 24, 2009

Decouple modules with asynchronous event dispatching using Spring and task queues in GAE

To build applications that are maintainable and robust you should strive for decoupling between modules. To build applications that scale you will always benefit from asynchronism and parallellism.
Here we will look how to accomplish the above in GoogleAppEngine and with some help from springframework.
Lets say we have an application where users can register them self. When they do, the application creates a persistence instance of a User-object. But, we will also keep track of how many users we have registered on the site. Now, being in GAE with BigTable luring in the back, doing queries and calculations (as we are used to with a traditional database) isn't a good idea. So as an alternative we choose to have a separate Counter-object that we updates when ever a new user registers. Ok, nothing strange here. But, there are a couple of flaws here:
  1. The User module needs to know about the Counter module.
  2. The User module has to wait for the Counter module to finish when updating the counting.
Ok, lets solve the first by using spring's mechanism for ApplicationEvent's. First, let us put some aop magic to work to intercept the call to UserService.createUser and when it returns (and we have the transaction boundaries on service methods, so no exception, all went well) fire off an event. Spring config for the aop stuff:


<bean id="userListener" class="org.fornax.sculptor.UserListener"/>
<bean id="userAdvice" class="org.fornax.sculptor.UserAdvice"/>
<aop:config>
<aop:pointcut id="userCreationPointcut" expression="execution(public * org..UserService.createUser(..))"/>
<aop:advisor pointcut-ref="userCreationPointcut" ref="userCreationPointcut"/>
</aop:config>


Next, here is the advice:

public class UserAdvice implements MethodInterceptor, ApplicationContextAware {

private ApplicationContext ctx;

public Object invoke(MethodInvocation invocation) throws Throwable {
User user = (User) invocation.proceed();
fireNewUserEvent(user);
return user;
}

private void fireNewUserEvent(User user) {
ctx.publishEvent(new UserCreatedEvent(user));
}

public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
this.ctx = applicationContext;
}
}

The listener that is being notified:

public class UserListener implements ApplicationListener<UserCreatedEvent> {
@Autowired private CounterService counterService;
public void onApplicationEvent(UserCreatedEvent event) {
counterService.increment();
}
}

And the event being passed:

public class UserCreatedEvent extends ApplicationEvent {
public UserCreatedEvent(User user) {
super(user);
}
}

Ok, so now we are half way. We have the Observer pattern in place. But we still does everything synchronous.
Enter GAE's task queue's. Let us modify our UserListener:

public class UserListener implements ApplicationListener<UserCreatedEvent> {

public void onApplicationEvent(UserCreatedEvent event) {
TaskOptions task = url("/rest/admin/counter/user").method(POST);
Queue queue = QueueFactory.getDefaultQueue();
queue.add(task);
}
}

And by the wonders of task queue's, we now put a task on the queue and by that we do the counting job asynchronous. And of course, we dropped the reference to the CounterService. But we miss one piece here, right? What does the url in the task point at. Well, nothing strange here, it is just a spring mvc controller:

@Controller
public class CounterCountroller {
@Autowired private CounterService counterService;
@RequestMapping(value = "/admin/counter/user", method = RequestMethod.POST)
public void incrementCounter() throws IOException {
try {
counterService.increment();
} catch (Exception ignore) {
// doesn't matter if we get an exception here, just log it
log.error("Failed to increment counter!", ignore);
}
}
}
And now we have a more loosely coupled system that scales better. And with a little effort, the code can be generalized so more features are easy to add with the same pattern.
Of course, the downside of this kind of design is that error handling gets more complicated and you can't always trust it to be 'right'. But that is system design, you have to decide what's best for each situation.

Wednesday, October 21, 2009

Even Weird Naming Conventions are Good

The good thing with naming conventions are that they are toolable. The DBAs at my department have strong opinions about database naming. They have good reasons for that, even though I don't fully understand all of them :-)
  • Table names should be prefixed with application/component identifier.
  • Primary key id column should be prefixed with table name (without application prefix) and followed by _GID.
  • Underscore to separate words.
  • Foreign key column is concatenation of role name and primary key column name of target table, except when role and table have the same name.
Does this mean that we have to specify each and every name twice, once for Java and once for the database. Argh... NO, we are using Sculptor. With a straightforward customization I implemented these conventions in the generator and we could continue with natural (java point of view) naming and please the preferences of the DBAs without additional effort.

Naming conventions are important for software quality. Supporting the conventions with a tool is the best way to make sure that they are applied in a consistent way.

Monday, October 19, 2009

Sculptor and Agile

For us, the agile way has always been the way of getting things done. Even before there where fancy names for it we did things in a way that enabled us to deliver the right things on the right time. This hasn't change, but now we call it scrum :-)
Sculptor will help you if you want to work agile.

Things change
They do. There is nothing anybody can do about it. And when things change, there will be delays. Doing things in an agile manner helps minimize those delays. Using sculptor helps you even more.
I'll show you by an example. Lets say you have an application that is almost done for production. Testing is done. No more bugs (yeah, right). On the final demo, suddenly, one of the stake holders realize that the customer object needs another attribute, lets call it ICE (In Case of Emergency). And, of course according to the stake holder you can't go into production without it.
So, what are the changes that needs to be done?
From the bottom up:
  • Database schema
  • Persistence layer (JPA, Hibernate, etc) :
  • Domain object
  • Service layer (if we have any logic tied to the attribute)
  • Back office Client (for creating a customer object)
  • Public Client (for viewing a customer object)
Most of these changes are boiler plate code. Changing declarations etc. Without Sculptor these changes takes time and are error prone. If we assume that every change is a risk and then calculate and compare all changes we need to do with and without Sculptor we end up with the following table where every point is a code change needed to do to accomplish the last minute requirement:








Without SculptorWith Sculptor
model:01
db script:10
db migration script:11
property in domain object:10
jpa annotation för property:10
logic in service or domain object:11
back office client (create/update/list/view):70
public client (view):11
junit test (save, find):22
documentation (class diagrams):10
sum:166

So, being simple, lets say that each point is equally valued. By this assumption we reduce risk and time by 150%

This can be very valuable when change comes along. And it does, doesn't it...?

Sunday, October 4, 2009

Sculptor in the Cloud

Now you can use Sculptor to speed up and simplify development of applications running in the Google App Engine cloud.


Powered by App Engine


Let's start with a demo of how easy it is to create a new application and deploy it.



For this we are using Sculptor maven archetype for App Engine. Try it yourself:
  1. mvn archetype:generate -DarchetypeGroupId=org.fornax.cartridges -DarchetypeArtifactId=fornax-cartridges-sculptor-archetype-appengine -DarchetypeVersion=1.7.0-SNAPSHOT -DarchetypeRepository=http://www.fornax-platform.org/archiva/repository/snapshots/

  2. cd to the new directory

  3. mvn clean

  4. mvn generate-sources

  5. mvn eclipse:eclipse

  6. Import the project in Eclipse

Without any changes the new project is ready to run in the local development server or to be deployed at appspot.com. The sample in the demo is available here: http://sculptor-helloworld.appspot.com

The archetype creates a sample of of a RESTful Spring 3.0 Controller and JSP pages for the CRUD operations.




















The archetype also creates a simple sample model, from which Sculptor generates Entity, Repository and Service with the default CRUD operations; findById, findAll, save, and delete.

The model is defined in a textual DSL, with an intuitive syntax, from which Sculptor generates high quality Java code and configuration. It is not a one time shot. The application can be developed incrementally with an efficient round trip loop. The generator is part of the build process (maven).



Sculptor generates JPA mapping annotations for the domain objects defined in the design model. Relations are very limited in App Engine, since the datastore (BigTable) is not a relational database.

Owned and embedded associations are supported and mapped as ordinary JPA associations. They are specified with aggregate and BasicType in the Sculptor model.



Unowned associations are handled with id references and you must lookup the objects with findById when needed.



Services and Repositories are implemented as Spring components with @Autowired dependency injection. Spring AOP is used for error handling and transaction management.



Behavior is implemented with hand written code in subclass, separated from re-generated code in base class. In the above example the sayHello method is typically implemented in the Service by first using the generated findByKey method in the Repository. Note that the name attribute of the Planet is marked as key.

Sculptor also provides support for JUnit testing with the local App Engine environment. I will cover that in another article some day soon.

Thursday, October 1, 2009

Maven Archetype for App Engine

I have developed a maven archetype for Google App Engine projects. The generated project supports:
  • All dependency jar files are downloaded from maven repositories and copied to lib directory as required by App Engine Eclipse plugin, and local development server.

  • Eclipse project is created with mvn eclipse:eclipse. The resulting Eclipse project has the necessary settings for App Engine Eclipse plugin.

  • Entity classes are processed by DataNucleus enhancer in the build lifecycle.

  • JUnit tests with local App Engine environment can be run from maven.

Setting up all of this is not trivial and therefore I would like to share the solution and I hope you find it useful.

Eclipse Project
The maven eclipse plugin need a lot of configuration.
<build>
<outputDirectory>war/WEB-INF/classes</outputDirectory>
<plugins>
<plugin>
<artifactId>maven-eclipse-plugin</artifactId>
<version>2.5.1</version>
<configuration>
<!--
buildOutputDirectory doesn't work due to
http://jira.codehaus.org/browse/MECLIPSE-422 An workaround is the
outputDirectory at project/build level
<buildOutputDirectory>war/WEB-INF/classes</buildOutputDirectory>
-->
<testOutputDirectory>target/test-classes</testOutputDirectory>
<classpathContainers>
<classpathContainer>com.google.appengine.eclipse.core.GAE_CONTAINER</classpathContainer>
</classpathContainers>
<buildcommands>
<buildcommand>org.eclipse.jdt.core.javabuilder</buildcommand>
<buildcommand>com.google.gdt.eclipse.core.webAppProjectValidator</buildcommand>
<buildcommand>com.google.appengine.eclipse.core.enhancerbuilder</buildcommand>
<buildcommand>com.google.appengine.eclipse.core.projectValidator</buildcommand>
</buildcommands>
<additionalProjectnatures>
<projectnature>org.eclipse.jdt.core.javanature</projectnature>
<projectnature>com.google.appengine.eclipse.core.gaeNature</projectnature>
<projectnature>com.google.gdt.eclipse.core.webAppNature</projectnature>
</additionalProjectnatures>
<excludes>
<!-- Included in GAE_CONTAINER -->
<exclude>com.google.appengine:appengine-api-1.0-sdk</exclude>
<exclude>com.google.appengine:appengine-api-1.0-labs</exclude>
<exclude>com.google.appengine.orm:datanucleus-appengine</exclude>
<exclude>org.datanucleus:datanucleus-jpa</exclude>
<exclude>org.datanucleus:datanucleus-core</exclude>
<exclude>org.apache.geronimo.specs:geronimo-jpa_3.0_spec</exclude>
<exclude>org.apache.geronimo.specs:geronimo-jta_1.1_spec</exclude>
<exclude>javax.jdo:jdo2-api</exclude>
</excludes>
</configuration>
</plugin>


Some dependencies must be excluded, since they are part of GAE_CONTAINER, otherwise JUnit tests will not work when running inside Eclipse. The output directory is changed to war/WEB-INF/classes. There is a bug (MECLIPSE-422) which cause the test classes to not be separated if buildOutputDirectory is used. The local development server doesn't like the test classes. The trick is to define the output at the top build level and define testOutputDirectory.

Copy Dependencies
When running the local development server and deploying to App Engine all dependent jar files must be located in war/WEB-INF/lib. I have used the maven dependency plugin to copy the jar files during the maven clean phase.

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy</id>
<phase>clean</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${spring.version}</version>
<outputDirectory>war/WEB-INF/lib</outputDirectory>
</artifactItem>
<!-- more ... -->
<artifactItem>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-api-1.0-sdk</artifactId>
<version>${appengine.version}</version>
<outputDirectory>war/WEB-INF/lib</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-api-1.0-labs</artifactId>
<version>${appengine.version}</version>
<outputDirectory>war/WEB-INF/lib</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>com.google.appengine.orm</groupId>
<artifactId>datanucleus-appengine</artifactId>
<version>1.0.3</version>
<outputDirectory>war/WEB-INF/lib</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-jpa</artifactId>
<version>1.1.5</version>
<outputDirectory>war/WEB-INF/lib</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>1.1.5</version>
<outputDirectory>war/WEB-INF/lib</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-jpa_3.0_spec</artifactId>
<version>1.1.1</version>
<outputDirectory>war/WEB-INF/lib</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-jta_1.1_spec</artifactId>
<version>1.1.1</version>
<outputDirectory>war/WEB-INF/lib</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>javax.jdo</groupId>
<artifactId>jdo2-api</artifactId>
<version>2.3-eb</version>
<outputDirectory>war/WEB-INF/lib</outputDirectory>
</artifactItem>
</artifactItems>
<!-- other configurations here -->
</configuration>
</execution>
</executions>
</plugin>



DataNucleus Enhancer
Running the JUnit tests from maven was a primary goal as I would like to run tests from continous build server. Th JUnit tests are using local App Engine environment with in-memory datastore. Therefore the classes must be processed by DataNucleus enhancer after ordinary compilation.

<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>maven-datanucleus-plugin</artifactId>
<version>1.1.4</version>
<configuration>
<api>JPA</api>
<mappingIncludes>**/*.class</mappingIncludes>
<log4jConfiguration>${basedir}/src/main/resources/log4j.properties</log4jConfiguration>
<verbose>false</verbose>
</configuration>
<executions>
<execution>
<phase>process-classes</phase>
<goals>
<goal>enhance</goal>
</goals>
</execution>
</executions>
</plugin>


Archetype
All of this is packaged in a maven archetype. Try it like this.
  1. mvn archetype:generate -DarchetypeGroupId=org.fornax.cartridges -DarchetypeArtifactId=fornax-cartridges-sculptor-archetype-appengine -DarchetypeVersion=1.7.0-SNAPSHOT -DarchetypeRepository=http://www.fornax-platform.org/archiva/repository/snapshots/

  2. cd to the new directory

  3. mvn clean

  4. mvn eclipse:eclipse

  5. Import the project in Eclipse

As an extra bonus your new project is configured for Spring 3.0 with a sample of a RESTful controller.

Sculptor code generator tool is of course also configured and ready to be used in the new project. I will soon write another article about Sculptor's support for App Engine.