GAE Transactions

I'm trying to understand what we should do to make Sculptor compatible with Google App Engine (GAE).

I feel a bit sad when looking back to what I have just experienced, but I guess I should be happy, since I have learned a lot. In this post I will share my mistakes and insights to GAE transactions and Entity Groups.

Together with Andreas I'm developing a little sample that consists of 3 interacting applications. Customer, Supplier and Profile apps. User stories for the initial sprint:

• As a customer I want to specify a request for consultants so that I can allocate resources to my project.


• As a salesman (supplier) I want to be notified when a customer enters a request for consultants so that I quickly can create an offer to that request.


• As a salesman I want to offer consultants to a customer so that I can sell our services.


• As a customer I want to see up to date information in the profiles so that I know that it is not obsolete.

I was developing the form enter of the inquiry in the customer app. I saved the form data in an Inquiry object and sent the request to the supplier app using RestTemplate. No problems so far.

We are using the new REST features in Spring 3.0 and have done some adjustments to Sculptor to make it generate JPA code that is compliant with GAE datastore.

Since one inquiry should be sent to many suppliers it didn't feel very scalable to send them all in the form entry request. Therefore I separated the sending to a separate job, which would be invoked by the cron service (later, better with task queue). This is not only more scalable, it is also more fault tolerant, since supplier apps may not be available all the time. By separating it we can easily retry later.

I created a Supplier entity also. In the sendToSuppliers job I got the first problem:

IllegalArgumentException: can't operate on multiple entity groups in a single transaction

Since I had two entities, Inquiry and Supplier and I was using both in the transaction I assumed that it was not allowed to query the Suppliers and update the Inquiries in the same transaction. I based that on the GAE documentation:
All datastore operations in a transaction must operate on entities in the same entity group. This includes querying for entities by ancestor, retrieving entities by key, updating entities, and deleting entities.

That assumption was a fatal mistake that got me on the wrong track. I started to separate the the retrieval of Suppliers and update of Inquiries in separate transactions.

I learned from the documentation that it was possible to disable transactions, but that it was a temporary workaround.

After removing all code except the update of the Inquiries I realized that the Inquiry instances themselves belonged to separate entity groups. I was looping over all Inquiries that had not been sent to suppliers, i.e. I was updating several instances. Of course, they belong to separate entity groups, otherwise it would not scale when the number of objects increase.

Then I redesigned the sending job so that it would only send and update one Inquiry instance. The job will have to be run many times to send all Inquiries.

On the way I learned some more things about GAE datastore:
* A transaction is necessary for some operations, such flush, otherwise; "This operation requires a transaction yet it is not active"
* Queries also require a transaction, otherwise when iterating over the result;
"Object Manager has been closed"
* Modification several times; "can't update the same entity twice in a transaction or operation"

In the end I think the defaults for transactions in Sculptor are alright. Normally we define transaction boundary at the service layer. This is ok for many cases when using GAE also, but one have to design the operations so that they only update one instance (entity group).

There is probably a need for more fine grained transaction control at the repository level. E.g. starting a new transaction for some repository operations. I think we should implement this with @Transactional annotations. Is it possible to mix txAdvice (defaults) with @Transactional (deviations from default)?

Introducing a Type

An important building block when creating a high quality domain model is to create small type objects. In this article we will create a Length type for the diameter of the Planet of the helloworld application.



Alternative video format (mpg)

Length is a typical Quantity with a value and unit, e.g. meter, kilometer.

In the design model it looks like this:

BasicType Length {
 BigDecimal value min="0"
 -@LengthUnit unit
}

enum LengthUnit {
 cm, m, km
}

Entity Planet {
 gap
 scaffold
 String name key
 Long population min="0"
 -@Length diameter nullable
 -Set<@Moon> moons opposite planet

 Repository PlanetRepository {
     findByKey;
 }
}

We also need to convert between different units. The behaviour expressed as a JUnit test:

public class LengthTest {

  @Test
  public void shouldConvertFromMeterToKilometer() {
      Length length = new Length(new BigDecimal("31000"), m);
      Length lengthInKilometer = length.to(km);
      assertEquals(new Length(new BigDecimal("31"), km),
          lengthInKilometer);
  }

  @Test
  public void shouldConvertFromKilometerToMeter() {
      Length length = new Length(new BigDecimal("44"), km);
      Length lengthInMeter = length.to(m);
      assertEquals(new Length(new BigDecimal("44000"), m),
          lengthInMeter);
  }

  @Test
  public void shouldNotConvertSameUnit() {
      Length length = new Length(new BigDecimal("17"), km);
      Length length2 = length.to(km);
      assertSame(length, length2);
  }
}

BasicType objects may contain business logic in the same way as other domain objects. The following screencast illustrates how to implement the conversion.



Alternative video format (mpg)

BasicType is a stored in the same table as the Domain Object referencing it. It corresponds to JPA @Embeddable.

There are a lot of cases when it is a good idea to introduce types.

• Identifers, natural business keys. It is more readable to pass around an identifier type instead of a plain String or Integer
• Money
• Range
• Quantity
  

I can recommend reading When to Make a Type, Martin Fowler.

Testing is Simple

In previous articles we created a simple helloworld application without any tests. This article shows how to do integration testing of services with Sculptor.



Alternative video format (mpg)

We turn on generation of junit test and complete the failing tests from previous articles. For each Service there is a generated JUnit test class that we are encouraged to implement. It uses Spring transactional test fixtures and DbUnit.

Spring beans are injected in the test with ordinary @Autowired annotations.

public class PlanetServiceTest extends AbstractDbUnitJpaTests
    implements PlanetServiceTestBase {
  private PlanetService planetService;

  @Autowired
  public void setPlanetService(PlanetService planetService) {
    this.planetService = planetService;
  }

  @Test
  public void testFindById() throws Exception {
    Planet found = planetService.findById(getServiceContext(), 1L);
    assertEquals("Earth", found.getName());
  }

  @Test
  public void testFindAll() throws Exception {
    List<Planet> found = planetService.findAll(getServiceContext());
    assertEquals(2, found.size());
  }

  @Test
  public void testSave() throws Exception {
    int countBefore = countRowsInTable("PLANET");
    Planet planet = new Planet("Pluto");
    planet.setPopulation(0L);
    planetService.save(getServiceContext(), planet);
    assertEquals(countBefore + 1, countRowsInTable("PLANET"));
  }

  @Test
  public void testDelete() throws Exception {
    int countBefore = countRowsInTable("PLANET");
    Planet planet = planetService.findById(getServiceContext(), 2L);
    planetService.delete(getServiceContext(), planet);
    assertEquals(countBefore - 1, countRowsInTable("PLANET"));
  }
}

The initial test data is defined in DbUnit xml file. The database is refreshed for each test method.

<?xml version="1.0" encoding="UTF-8"?>

<dataset>
  <PLANET ID="1" NAME="Earth" POPULATION="7000000000" VERSION="0"/>
  <PLANET ID="2" NAME="Jupiter" POPULATION="0" VERSION="0"/>
  <MOON/>
</dataset>

Above tests only covers the normal cases so far and we should of course do more tests for exceptional cases and validation boundaries, such as negative population.

The tests illustrated here are kind of integration tests and you should do ordinary unit tests for domain objects and other classes of importance.

The World is Changing

In the previous article the simple helloworld application consisted of one single domain object, the Planet. In this article I will add some more features to the picture, including an association to the Moons of the Planet.



Alternative video format (mpg)

It is good to have a natural business key, which is used for equals and hashCode. The name of the Planet is a candidate. Let us add some properties for the population and diameter of the Planet also. Hibernate validator is supported and validations can be defined directly in the model, e.g. min="0".



We add the Moon Entity and its association to Planet.



Build and start Jetty. The changes are immediately reflected in the generated CRUD GUI.



The DSL and the code generation drives the development and is not a one time shot. The application can be developed incrementally with an efficient round trip loop.

Jump Start

This article illustrates the first basic steps of how to use Sculptor. We will create a simple hello world application. It will only have one simple domain object and some CRUD operations.



Alternative video format (mpg)

We start by firing off some maven archetype commands to create the business and presentation tier projects. The sculptor archetypes creates maven pom files with needed dependencies. We generate eclipse projects from those maven projects using the ordinary maven-eclipse-plugin. These projects are imported into Eclipse. You find the commands for this in the wiki.

We open the design model that is the input to the code generator. It is a textual DSL that defines the application structure. We define the Planet Entity in a Module.



We also need some CRUD operations to be able to do something with the Planet in the to be generated GUI. It is easiest to add those with the scaffold keyword. Note that the DSL editor has support for code completion, error highlight and outline view (including ctrl+O).

Now it is time to generate code, but first we will turn off one feature. Sculptor promotes writing junit tests, but for this demo we turn off that by adding a property to sculptor-generator.properties.

generate.test=false

We build the application by executing mvn install from the -parent project. The code generation is part of the maven build cycle.

We start the web application server by running mvn jetty:run from the -web project. Note that this is also a maven command, and no specific installation of an application server and database is required. Hsqldb inmemory database is used.

The generated CRUD GUI is available at http://localhost:8080/helloworld-web/



That was everything needed to create a minimal application. It illustrates how easy it is to get started. Within 15 minutes you can go from scratch to a running application, including build scripts, Eclipse projects, domain model, JPA persistence, services, Web Flow application and much more. Thereafter you can continue by evolving the design, add manual code and regenerate.

This was an extremely simple example. Sculptor can be used for much more advanced applications. The CRUD GUI is useful for administrative sections of the application or to serve as a scaffolding for your manually crafted user interface. The major strength of Sculptor is in the business tier when developing typical enterprise or web applications that benefit from a rich and persistent domain model.

What is Sculptor?

Sculptor is an open source productivity tool that applies the concepts from Domain-Driven Design and Domain Specific Languages.

You express your design intent in a textual DSL, from which Sculptor generates high quality Java code and configuration. It is not a one time shot. The application can be developed incrementally with an efficient round trip loop.

Sculptor is useful when developing typical enterprise or web applications that benefit from a rich and persistent domain model. Sculptor also provides a sophisticated CRUD GUI for administrative sections of the application or to serve as a scaffolding for your manually created pages.

The generated code is based on well-known frameworks, such as JPA, Hibernate, Spring Framework, Spring Web Flow, JSF, RCP, and Java EE.

Product characteristics:

• Easy to learn, intuitive syntax of the textual DSL, editor with error highlight, code completion, and outline

• Quick development round trip, it is not a one time generation

• Existing IDE tools, such as refactoring, code assist and debugger will continue to be of service to you

• High quality of generated code, based on well known frameworks, best practices, and a lot of experience

• Great extensibility and customization options

• Easy to remove the tool, no runtime magic

Visit the Fornax Sculptor web site to learn more and try it.