Sculptor Team Blog

Sculptor 2.1.0 Released

2012-01-21T23:49:00.003+01:00

One very useful new feature is the generated finder methods. It is now possible to specify the query in the model file and let sculptor generate the code for the repository operations. In some cases the query can be derived from the method signature only.

This will generate a query returning all persons with the specified first name.


Entity Person {
  String firstName
  String lastName
  Date birthDate

  PersonRepository {
    findBy(String firstName);
  }
}

More advanced conditions can also be specified



  PersonRepository {
    findBornBefore(Date bd) condition="birthDate < :bd" orderBy="firstName";
    findByName(String name) condition="lastName i= :name or firstName i= :name";
  }

Sculptor generates findByCondition expressions and will report at generation time if there are any errors in the queries.

Another nice addition is the generated builder classes that provides a fluent interface to build domain objects in a manner that can be easier to work with and read.


    Book book = book()
        .createdBy("me")
        .createdDate(now)
        .title("Ender's Game")
        .isbn("Some-ISBN")
        .addMediaCharacter(mediaCharacter()
            .name("Ender")
            .build())
        .build();

Sculptor 2.0 is out

2011-03-14T21:24:00.002+01:00

This is the 11th major release of Sculptor.

Three new and noteworthy features:

1. The new REST support in Sculptor makes it very easy to expose restful services by using conventions and easy delegation to Services to reduce boilerplate coding. Spring MVC is the underlaying REST framework.

2. Mixin composition is something Java developers are missing, but with Sculptor it becomes available.

3. The services can now easily be used with Spring Remoting. This is convenient for the RCP client, but can be used for other clients also.

In addition to these and several other new features we have also improved the technical quality.

We have upgraded support for latest versions of most of the underlaying tools and frameworks. Most important is Xtext 1.0, Eclipse Helios, Maven 3, Spring 3.0.

The logging API has been changed from Commons Logging with Log4j to SLF4J with Logback. SLF4J allow many backends and also have special bridges for many (legacy) logging frameworks. SLF4J with Logback is considered to be best logging framework available.

Mixin Composition

2011-02-03T22:00:00.003+01:00

I think one of the best features in Scala is traits. Using traits it is possible to compose small pieces of behavior and state in an elegant way. I miss traits when I use Java. To mitigate that problem I have implemented support for traits in Sculptor. This article illustrates how this tool can be used for designing good, rich, domain models with traits.

Traits provide a mixin composition mechanism that is missing in Java. Similar to interfaces in Java, traits are used to define object types by specifying the signature of the supported methods. Unlike interfaces, traits can be partially implemented; i.e. it is possible to define implementations for some methods. Similar to abstract classes, but you don't have to wast your single inheritance opportunity.

How many times have you stared at java code like this:
if (p1.compareTo(p2) <= 0)

Why not spell it out, to make the code more readable:
if (p1.lessThanOrEquals(p2))

We seldom do that, because implementing 4 methods, lessThan, lessThanOrEquals, greaterThan and greaterThanOrEquals, in each and every class that needs to be compared is too much work. We stick to the cryptical compareTo.

What if I say you only have to implement them once and then you can easily mixin the methods in each and every class that needs to be compared or sorted in some way.

I hope I don't have to elaborate on how bad idea it is to try to use inheritance (base class) for these kind of reusable pieces of code.

In Sculptor's textual DSL model traits are defined like this:


        Trait Ordered {
            def boolean greaterThan(@Ordered other);
            def boolean greaterThanOrEquals(@Ordered other);
            def boolean lessThan(@Ordered other);
            def boolean lessThanOrEquals(@Ordered other);
            def abstract protected int compare(@Ordered other);
        }
        
        Entity Product with Ordered {
            String name
        }

After code generation this means that you can implement the 4 comparison methods once, in the Ordered trait, and they will be available in Product and other domain objects that are defined 'with Ordered'. The compare method must still be implemented in Product, because it is there you know what to compare with. The internal, generated, implementation is based on delegation, i.e. no magic.

Let us define another trait, which illustrates that traits also can hold state.


        Trait PriceTag {
            - protected Money normalPrice
            protected double discount
            def Money price;
            def Money price(String currency);
        }

        Entity Product with PriceTag {
            String name
        }

        BasicType Money with Ordered {
            BigDecimal amount
            String currency
        }

This means that normalPrice and discount will be mixed in to Product. You implement the price methods in the PriceTag trait. Product and all other domain objects that are defined 'with PriceTag' will have those fields and methods.

Wouldn't it be nice to be able to compare product by price? Let us do that by combining the two traits. First add the compare method in PriceTag, so that you only have to implement it at one place.


        Trait PriceTag {
            - protected Money normalPrice
            protected double discount
            def Money price;
            def Money price(String currency);
            def protected int compare(Ordered other);
        }

Then mixin both traits into Product


        Entity Product with Ordered with PriceTag {
            String name
        }

That's it. We have designed products with a rich price and compare interface.
Note that compareTo is no longer implemented in Product, only in PriceTag.

Try this new feature in latest Sculptor 2.0.0-SNAPSHOT.

EDA in Massive Web Sites

2011-01-27T08:32:00.005+01:00

This is a comment to @h3nk3's latest blog post on Concurrency, Parallelism and Actors.

I totally agree that there is no single solution for everything. Regarding massive web sites I would like to add a few things to your conclusions, which makes asynchronous event driven solutions important for web sites also.

You should try to do as little as possible in the request thread, i.e. minimize latency. Threads in them selves are a scarce resource. You can do the essential validation and then hand off the rest of the work to a task queue (e.g. Flickr). You can precompute everything and cache it so that serving the requests becomes trivial (e.g. Reddit). Both these techniques can benefit from using event driven solutions in the backend.

We see great improvements in the area of truly asynchronous web solutions, which means that you don't have to deliver a synchronous response to every request. The result can be pushed to the clients asynchronously. Then it becomes natural to use event driven solutions in the server. The reasons for using Actors it is not only scalability and concurrency, it is also much about using a simpler concurrency model than threads and locks.

Of course it is also important to use HTTP as designed, since HTTP caching is a key ingredient in highly scalable web sites.

Event Sourcing with Sculptor - Snapshots

2010-10-29T22:00:00.002+02:00

Yesterday I described how events can be used as storage mechanism. Instead of storing current state we store each change as a Domain Event. Current state is reconstructed by loading and replaying all historical events. For Entities with a long life cycle it can be too many events. This can be solved with an optimization that is based on periodically storing a rolling snapshot of current state.

Let us look at the code in the Sculptor port of the Simple CQRS Example to understand what this means in practice.

When loading InventoryItem we start by applying latest snapshot, if any, and thereafter replaying the events after the snapshot. This is the code in the Repository:


    @Override
    public InventoryItem findByKey(String itemId)
            throws InventoryItemNotFoundException {
        InventoryItem result = super.findByKey(itemId);

        loadFromHistory(result);

        return result;
    }

    private void loadFromHistory(InventoryItem entity) {
        InventoryItemSnapshot snapshot = getInventoryItemSnapshotRepository()
                .getLatestSnapshot(entity.getItemId());
        entity.applySnapshot(snapshot);
        long snapshotVersion = snapshot == null ? 0 : snapshot.getVersion();

        List<InventoryItemEvent> history = getInventoryItemEventRepository()
                .findAllAfter(entity.getItemId(), snapshotVersion);
        entity.loadFromHistory(history);
    }

That is how the snapshots are used when loading InventoryItem. Let us see how they are saved. We define the Snapshot Value Object for InventoryItem like this


        ValueObject InventoryItemSnapshot {
            String itemId index
            boolean activated
            Long version
            
            Repository InventoryItemSnapshotRepository {
                @InventoryItemSnapshot getLatestSnapshot(String itemId);
                protected findByCondition(PagingParameter pagingParameter);
                save;
            }
        }

Here we store the state as explicit attributes, which is simple with Sculptor, since we got the persistence (to MongoDB or JPA) for free, but it can also be stored as a blob (ecoded as xml, protobuf, or whatever you prefer). In this example the only state is the activated flag, but it can be much more in a real application.

The storage of the snapshot is triggered by a subscriber on the ordinary Domain Event flow. Simply defined as this in the model:


        Service InventoryItemSnapshotter {
            subscribe to inventoryItemTopic
            inject @InventoryItemRepository
            inject @InventoryItemSnapshotRepository
        }

The implementation calculates how many events has passed since previous snapshot by comparing version numbers. When the delta exceeds a threshold (e.g. 100 events) a snapshot is created and saved.


    public void receive(Event event) {
        if (!(event instanceof InventoryItemEvent)) {
            return;
        }

        InventoryItemEvent inventoryItemEvent = (InventoryItemEvent) event;
        String itemId = inventoryItemEvent.getItemId();

        InventoryItemSnapshot snapshot = getInventoryItemSnapshotRepository()
                .getLatestSnapshot(itemId);
        long snapshotVersion = snapshot == null ? 1 : snapshot.getVersion();
        long eventVersion = inventoryItemEvent.getAggregateVersion() == null ? 1
                : inventoryItemEvent.getAggregateVersion();
        if (eventVersion - snapshotVersion >= VERSION_DELTA) {
            takeSnapshot(itemId);
        }
    }

    private void takeSnapshot(String itemId) {
        InventoryItem item;
        try {
            item = getInventoryItemRepository().findByKey(itemId);
        } catch (InventoryItemNotFoundException e) {
            log.warn("takeSnapshot failed: " + e.getMessage());
            return;
        }

        InventoryItemSnapshot snapshot = item.createSnapshot();
        getInventoryItemSnapshotRepository().save(snapshot);
    }

By using snapshots we can dramatically improve performance for loading Entities with many historical changes. However, you can always start development without snapshotting and add it later, as a performance enhancement.

Also, note that snapshots and event are immutable and therefore we have great opportunities for using caching for improving performance.

The complete source code for this example is available here: http://github.com/patriknw/sculptor-simplecqrs/

Event Sourcing with Sculptor

2010-10-28T14:40:00.001+02:00

A while a go Greg Young published the Super Simple CQRS Example. There are also some good documents of how to use events as storage mechanism at the CQRS Info Site.

I have implemented the same example with Java and Sculptor. In this post I will describe how the Event Sourcing is implemented. Even though my implementation is facilitated by using Sculptor and MongoDB the design can easily be done with other techniques, such as JPA or plain JDBC, and without Sculptor.

A domain model typically holds current state of the world. Event Sourcing makes it possible to see how we got to this state. Essentially it means that we have to capture all changes to an application state as a sequence of events. These events can be used to reconstruct current and past states.

The sample application is a simplified inventory of items. In line with CQRS it has a strict separation of commands and queries. Changes to the InventoryItem domain object are published as Domain Events. The InventoryItem can be modified with a few commands that I have represented as operations in a Service.


    Service InventoryFacade {
        inject @InventoryItemRepository
        createInventoryItem(String itemId, String name);
        deactivateInventoryItem(String itemId);
        renameInventoryItem(String itemId, String newName);
        checkInItemsToInventory(String itemId, int count);
        removeItemsFromInventory(String itemId, int count);
    }

All these commands are handled in the same way, except createInventoryItem, which is slightly different. The Service looks up the Entity and calls corresponding method on the domain object.


    public void deactivateInventoryItem(String itemId) {
        InventoryItem item = tryGetItem(itemId);
        item.deactivate();
        getInventoryItemRepository().save(item);
    }

The Entity performs validation and creates Domain Event for state changes.


    public void deactivate() {
        if (!isActivated())
            throw new IllegalStateException("already deactivated");
        applyChange(new InventoryItemDeactivated(new Date(), getItemId()));
    }

In this case InventoryItem has a flag (state) indicating that the item is activated or not. Note that this state is not changed directly, but it is changed later as a result of the InventoryItemDeactivated event. All changes are done with Domain Events, which is important, because we don't save current state, we only save the Domain Events. The activated flag is not stored explicitly.

The DomainEvent is applied and added to a list of changes, which later will be stored and published.


    private void applyChange(InventoryItemEvent event) {
        applyChange(event, true);
    }
    
    private void applyChange(InventoryItemEvent event, boolean isNew) {
        DynamicMethodDispatcher.dispatch(this, event, "apply");
        if (isNew) {
            changes.add(event);
        } else {
            setVersion(event.getAggregateVersion());
        }
    }

    public void apply(InventoryItemDeactivated event) {
        setActivated(false);
    }

When saving, the InventoryItem instance is saved, but only as a key and version. The version is used for detecting concurrent modifications (optimistic locking). Additionally, the changes, the Domain Events are stored. The version number of InventoryItem is stored in Each Domain Event. An additional sequence number is also used to ensure the correct order of the events.

We need to override the default save method in the Repository to handle these sequence numbers and also save the events.


    @Override
    public InventoryItem save(InventoryItem entity) {
        InventoryItem saved = super.save(entity);

        List<InventoryItemEvent> changes = entity.getUncommittedChanges();
        changes = applyVersionToChanges(changes, saved.getVersion());
        for (InventoryItemEvent each : changes) {
            getInventoryItemEventRepository().save(each);
        }
        entity.markChangesAsCommitted();

        return saved;
    }

    private List<InventoryItemEvent> applyVersionToChanges(
            List<InventoryItemEvent> changes, long version) {
        List<InventoryItemEvent> result = new ArrayList<InventoryItemEvent>();
        long sequence = version * 1000;
        for (InventoryItemEvent each : changes) {
            result.add(each.withAggregateVersion(version).withChangeSequence(
                    sequence));
            sequence++;
        }
        return result;
    }

When saving the Domain Events they are also published to a topic, which the read side subscribes on. That is handled with the publish/subscribe mechanism in Sculptor. In the model we simply need to specifiy publish on the save method.


        abstract DomainEvent InventoryItemEvent {
            persistent
            String itemId index
            Long aggregateVersion nullable
            Long changeSequence nullable
            
            Repository InventoryItemEventRepository { 
                save publish to inventoryItemTopic;
                List<@InventoryItemEvent> findAllForItem(String itemId);
                protected findByCondition;
            }
        }

        DomainEvent InventoryItemDeactivated extends @InventoryItemEvent {
        }

In the read side we add subscribers to this topic.


    Service InventoryListView {
        subscribe to inventoryItemTopic
        inject @InventoryItemListRepository
    }
    
    Service InventoryItemDetailView {
        subscribe to inventoryItemTopic
        inject @InventoryItemDetailsRepository
    }

Alright, then we are almost done. One more thing though, when retrieving a InventoryItem we must replay all events to recreate current state. We do that by overriding the default findByKey method in the Repository.


    @Override
    public InventoryItem findByKey(String itemId)
            throws InventoryItemNotFoundException {
        InventoryItem result = super.findByKey(itemId);

        loadFromHistory(result);

        return result;
    }

    private void loadFromHistory(InventoryItem entity) {
        List<InventoryItemEvent> history = getInventoryItemEventRepository()
                .findAllForItem(entity.getItemId());
        entity.loadFromHistory(history);
    }

To retrieve all events we use a simple query in the InventoryItemEventRepository.


    public List<InventoryItemEvent> findAllForItem(String itemId) {
        List<ConditionalCriteria> criteria = criteriaFor(
                InventoryItemEvent.class).withProperty(itemId()).eq(itemId)
                .orderBy(changeSequence()).build();
        return findByCondition(criteria);
    }

The loaded events are applied to the InventoryItem Domain Object.


    public void loadFromHistory(List<InventoryItemEvent> history) {
        for (InventoryItemEvent each : history) {
            applyChange(each, false);
        }
    }

    private void applyChange(InventoryItemEvent event, boolean isNew) {
        DynamicMethodDispatcher.dispatch(this, event, "apply");
        if (isNew) {
            changes.add(event);
        } else {
            setVersion(event.getAggregateVersion());
        }
    }

    public void apply(InventoryItemCreated event) {
        setActivated(true);
    }

    public void apply(InventoryItemDeactivated event) {
        setActivated(false);
    }

    public void apply(Object other) {
        // ignore
    }

In this example we have used a naive approach when loading the InventoryItem by replaying all historical events. For Entities with a long life cycle it can be too many events. Then we can use a snapshot technique, which I will describe in a separate blog post.

The complete source code for this example can be found here: http://github.com/patriknw/sculptor-simplecqrs/tree/event_sourcing_without_snapshots

Switching an existing app to MongoDB

2010-10-21T17:27:00.002+02:00

Ron has switched project from using Hibernate/MySQL to MongoDB and wrote notes on the changes and gotchas.
Read it: Switching an existing app to MongoDB

Photo from San Francisco

2010-09-18T09:36:00.001+02:00

Karsten saw this in San Francisco :-)

EDA CQRS Betting Sample

2010-09-09T21:00:00.000+02:00

For todays Xtext/Sculptor seminar we have prepared a new example that illustrates some of the new event-driven features in Sculptor. It highlights a good case for using an architecture in line with Command and Query Responsibility Segregation (CQRS) pattern.

Consider an online betting system. It receives a massive amount of bets. A simple model for that could be:


        ValueObject Bet {
            String betOfferId
            String customerId
            Double amount
            
            Repository BetRepository {
                save;
            }
        }
        
        Service BettingEngine {
            inject @BetRepository
            
            placeBet(@Bet bet);
        }

The BettingEngine process the Bet and stores the information.

Questions pop up:

How many bets have been done by customer A?

Which customers place the higest bets, on average?

What are the top 10 high stakes?

It would be rather easy to develop support for those kind of queries in the master betting engine domain, but problems will soon bubble up.

Poor performance - the structure of the domain model is not optimized for all types of queries.

Not scalable - single centralized database will become a bottleneck.

Hard to change - too much functionality in one monolithic system.

Command-Query Responsibility Segregation (CQRS) comes to the rescue. Simplified it is about separating commands (that change the data) from the queries (that read the data). Separate subsystems take care of answering queries (reporting) and the domain for processing and storing updates can stay focused. The result of the commands are published to query subsystems, each optimized for its purpose.

Back to the betting sample. We are aiming for a design as illustrated in next drawing. Command side on the left (green) and Query side on the right (blue).

We publish domain events when bets are placed. There are several ways to do that in Sculptor, but one easy way is to mark a Service operation with the publish keyword in the model.


        DomainEvent BetPlaced {
            - Bet bet
        }

        Service BettingPublisher {
            publishEvent(@BetPlaced betEvent) publish to jms:topic:bet; 
        }

Then, in the hand-written java code simply invoke this method. In this case a one-liner in BettingEngine.

That's all for the command side, now let us take a look at the query side. We define a new module or even better a completely new business component.


   Module customer {
        Consumer BettingConsumer {
            inject @CustomerStatisticsRepository
            subscribe to jms:topic:bet
        }
        
        Service BettingQueryService {
            getHighBetters => CustomerStatisticsRepository.findHighAverageCustomers; 
        }
   
        Entity CustomerStatistics {
            gap
            String customerId key
            int numberOfBets
            double averageAmount index
            
            Repository CustomerStatisticsRepository { 
                findByKey;
                save;
                List<@CustomerStatistics> findHighAverageCustomers(double limit);
                protected findByCondition;
            }
        }
   }

We receive the events published from the betting engine by defining the subscribe keyword in the BettingConsumer.

In the query side we use domain objects that are optimized for the views and reports that are needed. We denormalize the data to minimize the number of joins needed when retrieving the data. Data is calculated ahead of time. In this case we calculate the average amount for the CustomerStatistics.

Since we are using publish/subscribe via a message bus, it might take a while until the query side is updated with the latest data, but that is not a problem. Most systems can be eventually consistent on the query side.

The full source code for the example is available here: http://github.com/patriknw/sculptor-betting

In the example we have chosen MongoDB as persistence store, Camel together with ActiveMQ as event bus. It is a matter of simple configuration to use something else, such as Oracle with JPA/Hibernate and Spring Integration for the event bus.

I was pretty impressed myself when I counted the number of lines of code that was required to implement this example. 70 lines! 40 in the model and 30 lines of hand written java code. User interface not counted.

Sculptor Presentation

2010-09-09T20:50:00.003+02:00

Today we had a Sculptor presentation as part of the seminar 'Developing Domain-Specific Languages with Xtext'. The slides from this presentation is available at slideshare: http://www.slideshare.net/patriknw/sculptor

Free Seminar: Developing DSLs with Xtext

2010-09-02T08:28:00.003+02:00

Next Thursday, September 09, in Stockholm, we will talk about Sculptor in the context of how to develop textual Domain-Specific Languages with Xtext. In this talk Sven Efftinge, original creator of Xtext, will present Xtext. I'm looking forward to listen to Sven's presentation.

More information and registration here: http://jwsdsl09sep.eventbrite.com/

EDA Akka as EventBus

2010-08-26T10:00:00.001+02:00

Some time since last entry in the EDA-sequence, but here we are again. Today we are going to do something really interesting. A friend and colleague of us, Jonas Bonér, is the creator of a super interesting framework called Akka.
Akka is an event driven platform for constructing highly scalable and fault tolerant applications. It is built with Scala, but also have a rich API for java. It follows the Actor Model and together with Software Transactional Memory (STM), it raises the abstraction level and provides an easy to use tool for building highly concurrent applications.

So, today we are going to take advantage of the java API in Akka to do our own EventBus implementation.

First, update your pom with the stuff needed for Akka (repo's and dependency):

<repository>
<id>Akka</id>
<name>Akka Maven2 Repository</name>
<url>http://www.scalablesolutions.se/akka/repository/ </url>
</repository>

<repository>
<id>Multiverse</id>
<name>Multiverse Maven2 Repository</name>
<url>http://multiverse.googlecode.com/svn/maven-repository/releases/</url>
</repository>

<repository>
<id>GuiceyFruit</id>
<name>GuiceyFruit Maven2 Repository</name>
<url>http://guiceyfruit.googlecode.com/svn/repo/releases/ </url>
</repository>

<repository>
<id>JBoss</id>
<name>JBoss Maven2 Repository</name>
<url>https://repository.jboss.org/nexus/content/groups/public/ </url>
</repository>

...

<dependency>
<groupId>se.scalablesolutions.akka</groupId>
<artifactId>akka-core_2.8.0</artifactId>
<version>0.10</version>
</dependency>

Second, create your implementation of the bus, AkkaEventBus.java:


package org.foo;

import org.fornax.cartridges.sculptor.framework.event.Event;
import org.fornax.cartridges.sculptor.framework.event.EventBus;
import org.fornax.cartridges.sculptor.framework.event.EventSubscriber;

import se.scalablesolutions.akka.actor.ActorRef;
import se.scalablesolutions.akka.actor.ActorRegistry;
import se.scalablesolutions.akka.actor.UntypedActor;
import se.scalablesolutions.akka.actor.UntypedActorFactory;

public class AkkaEventBus implements EventBus {

 public boolean subscribe(final String topic, final EventSubscriber subscriber) {
  UntypedActor.actorOf(new UntypedActorFactory() {
   public UntypedActor create() {
    return new ActorListener(topic, subscriber);
   }
  }).start();
  
  return true;
 }

 public boolean unsubscribe(String topic, EventSubscriber subscriber) {
  // TODO : implement mapping between arbitrary subscriber and actor in registry
  return true;
 }

 public boolean publish(String topic, Event event) {
  ActorRef[] actorsForTopic = ActorRegistry.actorsFor(topic);
  for (int i = 0; i < actorsForTopic.length; i++) {
   actorsForTopic[i].sendOneWay(event);
  }
  return true;
 }

 @SuppressWarnings("unchecked")
 private class ActorListener extends UntypedActor {
  final String topic;
  final EventSubscriber subscriber;

  ActorListener(String topic, EventSubscriber subscriber) {
   this.topic = topic;
   this.subscriber = subscriber;
   this.getContext().setId(topic);
  }

  @Override
  public void onReceive(Object message) throws Exception {
   this.subscriber.receive((Event) message);
  }
 
 }
}

As you can see, I lack the unsubscribe implementation and I left out equals and hashCode overrides, but that I leave to you.

Third, add it as our bus implementation through spring config:


<bean id="akkaEventBus" class="org.foo.AkkaEventBus"/>
<alias name="akkaEventBus" alias="eventBus"/>

What we have done now is built an highly scalable and concurrent EventBus that dispatches event asynchronously.

Pretty easy, right? :-)

It doesn't run over the network, but Akka has some nice modules for that as well, so that is our task for next time.

EDA events over system boundaries - with camel

2010-08-11T18:07:00.000+02:00

Last time we examined how we could make our events travel over system boundaries with the help of JMS and Spring-integration.

Today we will do exactly the same but we will use Apache Camel as event engine instead. The task is to enable application domain events over system boundaries through JMS. If you don't remember our model, here it is again:

Application Universe {
basePackage=org.helloworld

Module milkyway {
  Service PlanetService {
    @BigLandingSuccess landOnPlanet(String planetName, String astronautName)
      publish to milkywayChannel;
  }

  DomainEvent BigLandingSuccess {
    String planetName
    String astronautName
  }
}
Module houston {
  Service GroundControlService {
    subscribe to milkywayChannel
    bringOutTheChampagne(int noOfBottles);
  }
}
}

To use Apache Camel as event engine, you need to specify it in the sculptor-generator.properties file:

integration.product=camel

And add the needed dependencies to the projects pom-file:


<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jms</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-camel</artifactId>
<version>5.3.2</version>
</dependency>
<!-- xbean is required for ActiveMQ broker configuration in the spring xml file -->
<dependency>
<groupId>org.apache.xbean</groupId>
<artifactId>xbean-spring</artifactId>
<version>3.7</version>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.1</version>
</dependency>

Regenerate and you will have a new camel.xml file in src/main/resources. This file is only generated once, and you can use it to add configuration for Camel.

To do the same as we did with spring-integration, i.e. put a domain event on a jms topic, we just have to add route rule to the camel.xml file:


<camel:route>
 <camel:from uri="direct:shippingChannel"/>
 <camel:to uri="jms:topic:shippingEvent"/>
</camel:route>

And that's it. Now we are publishing our domain event to an ActiveMQ topic. Yes, a bit strange that it works, but it gets clearer if we look in camel.xml and see whats was already there:


<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:camel="http://camel.apache.org/schema/spring" xmlns:broker="http://activemq.apache.org/schema/core" xsi:schemaLocation="         http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd         http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd         http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd         http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean id="camelEventBusImpl" class="org.fornax.cartridges.sculptor.framework.event.CamelEventBusImpl"/>
<alias name="camelEventBusImpl" alias="eventBus"/>

<camel:camelContext id="camel">
  <camel:package>org.sculptor.shipping</camel:package>
  <camel:template id="producerTemplate"/>
</camel:camelContext>
<!--
  Camel ActiveMQ to use the ActiveMQ broker
 -->
<bean id="jms" class="org.apache.activemq.camel.component.ActiveMQComponent">
  <property name="brokerURL" value="tcp://localhost:61616"/>
</bean>
</beans>

We have a bean that wires the jms api's to the actual ActiveMQ instance together. So you have to have an ActiveMQ instance up and running listening on port 61616.

All for today, next time...we'll see what the subject is...

EDA events over system boundaries

2010-08-02T22:00:00.001+02:00

In the last post we explained why we have the 'event bus' notion. And earlier we have seen how to use it to publish and subscribe, both through the dsl in the model, or through plain java code.

Today we thought we should look how you can make domain events part of the public api of your business component and publish them to the rest of the world.

To do this we will use the event bus implementation that is based on spring integration. We will use its JMS outbound channel adapter to publish the 'BigLandingSuccess' domain event to a jms topic that the rest of the world can listen to.

If you don't remember, here is the model:

Application Universe {
 basePackage=org.helloworld

 Module milkyway {
   Service PlanetService {
     @BigLandingSuccess landOnPlanet(String planetName, String astronautName)
       publish to milkywayChannel;
   }

   DomainEvent BigLandingSuccess {
     String planetName
     String astronautName
   }
 }
 Module houston {
   Service GroundControlService {
     subscribe to milkywayChannel
     bringOutTheChampagne(int noOfBottles);
   }
 }
}

But first, how do we replace the default event bus implementation with the spring-integration based?

In sculptor-generator.properties, add the property:

integration.product=spring-integration

And in the project pom file, add the dependency:

<dependency>
 <groupId>org.springframework.integration</groupId>
 <artifactId>org.springframework.integration</artifactId>
 <version>1.0.4.RELEASE</version>
</dependency>

Regenerate, and you are home. So, now all event routing will use spring-integration as engine. So without any other changes, it works exactly as before.
Now you have to decide how your public api should look like, and in our case we decides that the 'BigLandingSuccess' domain event should be made available for other applications to listen to.

And now we take advantage of the power of spring-integration. When we regenerated with spring-integration enabled, we ended up with the file:
src/main/resources/spring-integration.xml

This is the place where all spring-integration definitions lands. It is generated once, so you are now free to edit it. Open the file and add:

<jms:outbound-channel-adapter id="publicMilkywayChannel"  destination="publicMilkywayTopic" channel="milkywayChannel"/>

You will need a spring jms bean named 'publicMilkywayTopic' and that bean needs to map to an actual message broker instance, for example ActiveMQ. But the spring bean and message broker setup is out scope for this blog entry, so that one we leave to you.

The only caveat now is that we are exposing our java objects in the message. This can be cured with a transformation step before the jms adapter. So add a transformer to your spring-integration configuration:

<object-to-string-transformer input-channel="milkywayChannel" output-channel="milkywayMessagesAsStringChannel"/>

And change the jms outbound channel to wire up with the new channel:

<jms:outbound-channel-adapter id="publicMilkywayChannel"  destination="publicMilkywayTopic" channel="milkywayMessagesAsStringChannel"/>

Spring-integration has a lot of transformers you can use. The above just transforms a java object to a string through its toString method. That is of course not always what you want, but it works for this example.

So that was all that had to be done to make a domain event part of your public api. Of course, you need to have proper documentation to give others a fair chance of finding your event.

If we take one step back and consider what we have done.
First, we declared the 'BigLandingSuccess' domain event and publishes it internally.

Second, we add an adapter that listens to the channel and in its turn publishes it on a jms topic, i.e. makes it public.

Third, we added a transformation step before doing a public publish to remove the dependency to our classes.

Now, this is nice, isn't it? We can have a lot of domain events internally in our application and by that take advantage of all the nice attributes of EDA. And we can by choice make a domain event public with 'just' configuration changes.

Quite long post, but we got pretty much done. Next time we will look how to do the same with Apache Camel

EDA why the event bus in sculptor?

2010-08-01T15:26:00.000+02:00

We have come to our seventh entry about Event Driven Architecture. We are in the topic of how Sculptor supports EDA. In previous posts we have covered how to publish and subscribe, both through the dsl in the model, and through plain java code.

Today we will talk a bit more on the thin layer we call 'the event bus'.

As stated before, there are a lot of different approaches to EDA. You can use/implement it locally or apply it too the entire enterprise. But also, in its simplest form its about the observer pattern, regardless if we talk about big or small implementations. This leads us to the main motivations of our event bus abstraction.

We want to keep it simple, but at the same time still have the power of doing event handling big and small. In the Enterprise, or locally. And doing this with the same programming interfaces, i.e. keeping it simple.

So, based on the above we have the event bus api. And with the risk of repeating my self, it is a very simple one. Methods for publishing, subscribing, and un-subscribing.

It also ships with a default implementation named (you got it) simple event bus. This one is really easy (hey, I found another word instead of simple) to use and fully functional on its own. Though, if you need to integrate with another system you are going to need another implementation.

And that is one of our other motivation for the event bus, it should be easy to swap implementations.

Beside the default, we currently have two implementations based on Spring-integration and Apache Camel. Both of these are what's called "lightweight integration frameworks". Supporting these two frameworks brings a lot of power to the solution when it comes to integration.

And that brings us to the next topic of this series. Some examples of how to bring your events to life over system boundaries, i.e. integration stuff.

EDA pub/sub with plain java

2010-07-30T15:01:00.001+02:00

In a couple of posts we have seen how to publish and subscribe to events. I showed how to do that through the dsl in the model. But you have of course also the possibility to do pub/sub by plain java code as well. The key thing is to have a handle to the event bus.

The easiest way to explain it is to show an example, so, a simple service that publish an event as part of it's business logic:


@Service
public class GroundControlService {

 @Autowired
 @Qualifier("eventBus")
 private EventBus eventBus;

 @Override
 public void receive(Event event) {
   if (event instanceof BigLandingSuccess) {
     eventBus.publish("earthChannel", new Celebration("We did it!"));
     this.bringOutTheChampagne(ALL);
     eventBus.publish("supplierChannel", new MissingResource("Champagne"));
   }
 }
}

And for the other end, the subscriber:


@Service
public class SupplierService {

@Autowired
@Qualifier("eventBus")
private EventBus eventBus;

public void init() {
  eventBus.subscribe("supplierChannel", new EventSubscriber() {
    @Override
    public void receive(Event event) {
      if (event instanceof MissingResource) {
        raceForContract(event);
      }
    }
  });
}
}

Not so hard.

Now, here and there I've used the term 'event bus', and you have also seen it in java code above. Next time we will look closer into the bus. Why is there an abstraction and what can you do with it?

EDA a simple example with Sculptor continued

2010-07-29T14:45:00.004+02:00

Lets continue with our simple example from the last post. We declared a domain event in the model and we marked a service to publish that event to a channel.

Now, what about getting notified when such an event is published?

Continuing with our model from last time, its just a matter of editing it. Lets create another module with a service that is interested of the event:

Application Universe {
 basePackage=org.helloworld

 Module milkyway {
   Service PlanetService {
     @BigLandingSuccess landOnPlanet(String planetName, String astronautName)
       publish to milkywayChannel;
   }

   DomainEvent BigLandingSuccess {
     String planetName
     String astronautName
   }
 }
 Module houston {
   Service GroundControlService {
     subscribe to milkywayChannel
     bringOutTheChampagne(int noOfBottles);
   }
 }
}

The result is that the GroundControl service will implement the EventSubscriber interface and be marked with @Subscribe annotation. That means that the GroundControl service will automatically be added as subscriber to milkywayChannel. It will be notified, receive method called, when events are published to that channel.

You will get a stub of the receive method of the EventSubscriber interface.


@Override
public void receive(Event event) {
 // TODO Auto-generated method stub
 throw new UnsupportedOperationException("receive not implemented");
}

So that you will need to implement with your logic:


@Override
public void receive(Event event) {
 if (event instanceof BigLandingSuccess) {
   this.bringOutTheChampagne(999999);
 }
}

Ok, so now we covered some very simple notations for publishing and subscribing to events. Next time I'll show how to do the same but in plain java code.

EDA a simple example with Sculptor

2010-07-26T18:00:00.003+02:00

Now we have gone through EDA in general, and briefly covered how we support it in Sculptor. It's time to see how it can be used.

We start with a simple example where we will show how to declare a domain event and how that event is published.

Let us use our old, well known, hello world example.


Application Universe {
basePackage=org.helloworld

Module milkyway {
 Service PlanetService {
   void landOnPlanet(String planetName, String astronautName);
 }

}
}

That is our model. So, what might the rest of the 'universe' be interested of here? Well, if NASA sent out a space ship with an astronaut on it who's mission was to land on some far far away planet, wouldn't they be interested in the event of a landing? I think so. So, when a astronaut lands on a planet, we would like the service to publish an event of this happening.

Lets start with re-defining the model with the event:

Application Universe {
basePackage=org.helloworld

Module milkyway {
 Service PlanetService {
   void landOnPlanet(String planetName, String astronautName);
 }

 DomainEvent BigLandingSuccess {
   String planetName
   String astronautName
 }
}
}

DomainEvents may contain attributes and references in the same way as ValueObjects and Entities. DomainEvents are always immutable and not persistent.

Events are about something happening at a point in time, so it's natural for events to contain time information. Sculptor automatically adds two timestamps, occurred and recorded. The time the event occurred in the world and the time the event was noticed.

Ok, so now we have the event defined, now we want it to be published. The easiest way of doing this is to mark the service in the dsl (you can of course publish events programmatically, but more about that in a later blog entry) . So, once again, lets re-define our model:

Application Universe {
basePackage=org.helloworld

Module milkyway {
 Service PlanetService {
   @BigLandingSuccess landOnPlanet(String planetName, String astronautName)
      publish to milkywayChannel;
 }

 DomainEvent BigLandingSuccess {
   String planetName
   String astronautName
 }
}
}

The operation must return a DomainEvent or take a DomainEvent as parameter. That event is published to the defined channel when the operation has been invoked.

As an alternative the DomainEvent instance can be created from the return value or parameters. The DomainEvent must have a matching constructor.

The result of the above declarative way of publishing events is a generated annotation @Publish on the method. It will trigger the Spring AOP advice PublishAdvice that is part of Sculptor framework.

Recap: We have declared an event in our model, and further marked our service to publish this event when it happens. As an API, I think this is very neat. Everyone who is interested can see that when an astronaut lands on a planet, he or she can be notified.

And that is what we will talk about next time:

How can I be notified when things happens?

EDA sculptor support

2010-07-19T18:00:00.000+02:00

In the previous post we gave an overview of Event Driven Architecture. It is a very big area, and you can use it to a lot of things.
It is a very good complement to DDD, and is a corner stone when building scalable systems.
When constructing systems it is a very nice match to accomplish loosely coupled modules and bounded contexts, i.e. business components. And much more.

We strive for simplicity, but at the same time not restricting us.

So, how can you use with Sculptor?
In the 1.9.0 release, we have focused on support for Publish/Subscribe, Comman-Query Responisbility Segregation (CQRS) and Event Sourcing.

The most useful part is of course the pub/sub support. Event sourcing is an architectural style that has its niche. CQRS is also an architectural style that there has been some publicity around lately. CQRS uses pub/sub and can with advantage be constructed to use Event Sourcing.

To support the above, there are three central parts in our implementation:

An event bus abstraction
DomainEvent
CommandEvent

"The bus"
The event bus is an extremely simple API, with three different implementations (in 1.9.0), "Simple", Spring Integration and Apache Camel. The idea is that the central parts, i.e. pub/sub, should be easy to use. The only thing you have to work with is an event bus where you publish and subscribes to and from events. And if you need some non functional behavior (asynchronism, over the wire, etc) for your events, you plug in an event bus that can handle this requirements.
And as stated above, the easiest way to accomplish that with the 1.9.0 release is to use Spring Integration or Apache Camel. But you can also choose to implement your own event bus.
You can publish and subscribe to and from the bus either declarative through the DSL, or programatically through the event bus API.

DomainEvent vs CommandEvent

CommandEvent is an instruction for something to happen. The system processes a CommandEvent and takes appropriate actions.

DomainEvent states fact - that something has happen. This fact is published to the rest of the world and the publisher just lets it go with no further interest in what happens to the event, i.e. who receives it and what they do.

If you compare it to an application API, CommandEvent could be the input API, while the DomainEvent is the output API for the application. I.e. the CommandEvent is what you can make the application perform, while the DomainEvent is what the application reports has happen.

As you probably figured out, CommandEvent is for implementing CQRS and Event Sourcing.

Finally, let us take quick look at how the notion for DomainEvents look like in the DSL.

DomainEvent ShipHasArrived {
  - ShipId ship
  - UnLocode port
}
   
DomainEvent ShipHasDepartured {
  - ShipId ship
  - UnLocode port
}

As you can see, you define a domain event in the same way as for entities or value objects.

And to declare pub/sub:

Service TrackingService {
  @ShipHasArrived recordArrival(DateTime occurred, @Ship ship, @Port port)
      publish to shippingChannel;
}
Service Statistics {
  subscribe to shippingChannel
  int getShipsInPort(@UnLocode port);
  reset;
}


That was all for today, next time will it be more concrete with examples of how to use it.

EDA overview

2010-07-15T18:00:00.004+02:00

As stated in the last post, we would like to talk a bit (very shortly) about EDA in a broader sense to give an overview of it and to give you a sense for what we read into the term EDA. This is quite important as a background when we go further and explain our interpretation and implementation of it.

Event Driven Architecture is a very broad area. But in a short sentence its about:

Applications and systems that produce, consume and reacts on events.

And even if there is a lot of attention on EDA as means of implementing integration between applications and systems, EDA can be applied within applications and even in parts of applications.

The benefits of an EDA is:

loosely coupled systems (or internals of a system)
high performance (fire and forget)
high scalability

Of course this comes with a trade-off. An extra abstraction is added, i.e. it becomes more complex.

In it's simplest form, EDA is about the Observer Pattern. Something happens somewhere, and 0 to n parties is interested in that happening. From this simple pattern, all interpretations, implementations and usages of EDA are spawn.

So, before we go into how we have implemented support for EDA in Sculptor, we will give you some examples of what shapes we think EDA can take. Big and small...here we go:

GUI's

Swing uses the Observer pattern. Many different GUI frameworks uses an event driven approach. Java Server Faces works with events, event handlers and event actions. Etc...

DomainEvent

A Domain Event is registration of something that has happen. It might be of interest to 0 to n consumers. The thing is; The producer doesn't care, it just tell the world that this thing has happen and happily goes on with it's life.

PubSub

Publisher and Subscribers is a central part of all event driven integrations. It is an implementation of the Observer pattern. It often comes with persistent events. Topics in the Java/JMS tech domain is an implementation of it.

SOA2.0

Now when SOA has been around for some years the next thing is SOA2.0. In the 2.0 version of SOA events plays a central role as a complement to services.

An event-driven system typically consists of event emitters (or agents) and event consumers (or sinks). Sinks have the responsibility of applying a reaction as soon as an event is presented. The reaction might or might not be completely provided by the sink itself. For instance, the sink might just have the responsibility to filter, transform and forward the event to another component or it might provide a self contained reaction to such event. The first category of sinks can be based upon traditional components such as message oriented middleware while the second category of sinks (self contained online reaction) might require a more appropriate transactional executive framework, for example an ESB.

Event servers

Event servers are servers that are specialized in processing events. They often provide filter/query capabilities for events. Weblogic Event Server is an example of this kind of product.

Event sourcing

All changes to an application is recorded as a series of events. An applications current state can be queried. But not only that, an application state can be rolled forward or backwards as you wish, giving you a lot of power. Martin Fowler has written all about it here. Also, Patrik has written a couple of blog entries of how he has played with it and now also added support for it in Sculptor, see here and here.

CQRS

Command and Query Responsibility Segregation is a very cool area. I will not try to explain it in detail, it is very well done here.

Simplified it is about separating commands (that change the data) from the queries (that read the data). Separate subsystems take care of answering queries (reporting) and the domain for processing and storing updates can stay focused. The result of the commands are published to query subsystems, each optimized for its purpose.

Ok, that was probably a very short and incomplete list of EDA related topics, but it gave you a taste for it and a ground for further reading. We will use it as a foundation when we talk about how we think about and implements EDA.

If you think we have missed an important topic, tool, concept, etc, around EDA, please add a comment about it.

Next up, how we support EDA in Sculptor.

EDA intro

2010-07-12T18:00:00.002+02:00

Both Patrik and I have always been believers of Event Driven Architecture. EDA has been around for many years, but recently the attention towards it has increased. The reasons for that is probably many, but I guess one of the most important is that EDA will help you build more concurrent and scalable system.

Therefore, as we strive to support popular technologies, we take the opportunity to implement support for EDA in the newly released 1.9.0 version of Sculptor.

Patrik has written a couple of interesting posts (here and here) about event driven architecture in a special implementation model, event sourcing.
Event sourcing is a powerful tool in certain circumstances. Though, not always a perfect match.

Also, many people reads SOA2.0 or ESB when you say EDA. This is for sure true in many cases, but just a small set of the truth.

EDA is much more than event sourcing, SOA2.0 or ESB implementations.

EDA plays a big role in application architecture and design (or even module design) as well.

Our approach is that you as a developer should be able to choose how and where you implement EDA.

That is what we will talk about in a sequence of posts.

EDA in general.

EDA, how we see it.

EDA, how we support it.

So, stay tuned for more about EDA.

What's Next, after 1.9.0

2010-07-11T20:25:00.000+02:00

Next release will mostly be a technical upgrade. Xtext and Xpand version 1.0 were released together with Eclipse Helios. We will upgrade to that.

Spring 3.0 and JPA 2.0 are also important upgrades.

Oliver will contribute with more cool stuff. He has some almost finished things that he will share with us (hopefully) soon.

The original design of Sculptor has worked fine and we will not change it, but 3 years of additions of different combinations of target implementations is starting to hurt some templates. Therefore we will remove some old (probably unused) target implementations. Those have been deprecated in 1.9.0 and in case you are using any of it you should start migration to some of the more modern alternatives:

EJB/Spring combination => migrate to EJB3 or Spring

EJB 2 => migrate to EJB3 or Spring

Hibernate mapping with XML => migrate to JPA/Hibernate with annotations

Spring definitions in XML => migrate to Spring with annotations

Hibernate without JPA => migrate to JPA/Hibernate

This makes a good reason to use version number 2.0.0 for next release. Personally I get bad vibrations when Open Source project go to 2.0. That often means total redesign and no backwards compatibility. That will not be the case for Sculptor 2.0. It will be a cleanup, and some pieces are not supported any more, but most of it will stay intact and be compatible (with some migration steps, as usual).

Exploring new features in 1.9.0

2010-07-11T20:15:00.000+02:00

This post is dedicated for those of you who are already using Sculptor and would like to have a quick overview of some selected features in the release. I will not bring up bug fixes and all improvements. Please refer to the complete list in issue tracker for that.

I have written a separate teaser about the MongoDB and EDA features of the new release.

Generated Documentation

In the model you can write documentation as a quoted string in front of almost every element (attribute, reference, entity, operation, ...). From the model Sculptor generates HTML documentation of all domain objects including their attributes and associations. Documentation of Services are also included. The generated result is located in src/generated/resources/DomainModelDoc.html

Improved Graphviz Visualization

Several diagrams with different focus and level of detail are generated. They are also included in the generated documentation. Samples here. There is a new maven plugin fornax-graphviz-m2-plugin that generates images (.png) from the .dot files.

Syntax Diagrams

We have mainly focused on sample based documentation. Some users have asked for more formal syntax description of the DSL. We have created railroad syntax diagrams.

Aggregate Syntax

Aggregates can now be defined with the new belongsTo keyword. It is more informative than the previous !aggregateRoot.


        Entity Cargo {
            - TrackingId trackingId key
            - Location origin required
            - Location destination required
            - Itinerary itinerary nullable inverse opposite cargo
        }

        ValueObject Itinerary {
            belongsTo Cargo
            not optimisticLocking
            not immutable
            - Cargo cargo nullable opposite itinerary
            - List legs inverse
        }

        "An itinerary consists of one or more legs."
        ValueObject Leg {
            belongsTo Cargo 
            - CarrierMovement carrierMovement;
            - Location from;
            - Location ^to;
        }

Nullable in key

It is now allowed to have some nullable fields as part a composite natural key.


    ValueObject HandlingEvent {
        - Type type key
        - CarrierMovement carrierMovement nullable key
        - Location location key
        DateTime completionTime key
        DateTime registrationTime
        - Cargo cargo key opposite events
    }

ToStringStyle

We are using commons-lang for toString of domain objects. It has a style parameter, which is now possible to define in sculptor-generator.properties. You can choose between several styles. I like this format:


toStringStyle=SHORT_PREFIX_STYLE

You can also override this for individual DomainObject with hint:


    ValueObject Foo { 
        hint="toStringStyle=MULTI_LINE_STYLE" 
        String aaa 
        String bbb 
    }

databaseJoinTable

It is possible to define many-to-many join table with databaseJoinTable and its columns with databaseColumn at both sides of the bidirectional association:


- Set<@Media> existsInMedia opposite mediaCharacters
        databaseJoinTable="MED_CHR" databaseColumn="CHR"

BTW opposite doesn't have to be defined last any more.

Those keywords are also useful for unidirectional to-many associations. Additionally databaseJoinColumn is used, since there is no opposite side to define the column on.


- Set<@Person> playedBy databaseJoinTable="CHR_PERS" 
        databaseColumn="PERS" databaseJoinColumn="CHR"

Clob/Blob

Clob and Blob are now supported with default java types String and byte[]. They are mapped with JPA @Lob.

GUI with Not Persistent Value Objects

The structure of the persistent domain model might not always match the presentation, for example you might want a dialog for editing several domain objects in one single screen. Then you can create a non-persistent ValueObject with a corresponding Service. The application service handles the transformation between the presentation object and the persistent domain objects. More documentation and sample is available here.

Sculptor 1.9.0 - Support for MongoDB and Event-Driven Architecture

2010-07-11T20:00:00.000+02:00

The Sculptor development team has a track record delivering around 3 releases per year. We have delivered 10 releases in total. That means that the core pieces are rock solid. I have personally used it successfully together with 15 other developers on daily basis for about a year now. It simply works very well.

We care about bugfixing and making small improvements. At the same time we are excited about learning new technology and using emergent design. This release contains two new features in the area of scalability. Persistence backed with MongoDB and support for Event-Driven Architecture.

MongoDB bridges the gap between key-value stores (which are fast and highly scalable) and traditional RDBMS systems (which provide rich queries and deep functionality). I think this makes MongoDB very interesting for applications that need high-performance and/or scalability, but also prefer using a rich persistent domain model with complex associations. The schema less structure is attractive from a developer productivity perspective, which is one of the two goals with Sculptor (quality is the other).

Sculptor generates data mapper classes that converts domain objects to/from MongoDB data structures, DBObjects. This makes it easy to use a domain model à la DDD with automatic mapping to MongoDB data structures.

Sculptor provides generic repository operations for use with MongoDB. This includes operations such as save, delete, findById, findByKey, findByCondition, and some more. You get CRUD operations, and GUI, for free.

Queries can be expressed with a slick fluent api that support code completion and refactoring.

Rich support for associations. Aggregates are stored as a embedded documents. Other associations are stored with referring ids. In the domain objects there are generated getters that lazily fetch associated objects from the ids. This means that you don't have to work with the ids yourself, you can follow associations as usual.

Read more about how to use MongoDB with Sculptor here.

Event-Driven Architecture (EDA) is a good complement to Domain-Driven Design. We think EDA is an important ingredient for building scalable systems. It is also an enabler for designing loosely coupled modules and bounded contexts.

For this Sculptor makes it possible to define Domain Events in the model in similar way as Entities and Value Objects. Sculptor also provide a mechanism to publish and subscribe through a simple event bus. This is done either in a declarative way in the model, or programatically with a simple API.

Implementations of the event bus that integrates with Apache Camel and Spring Integration are available out-of-the-box.

In addition to publish/subscribe Sculptor also has support for CQRS and EventSourcing.

CQRS is about separating commands (that change the data) from the queries (that read the data). Separate subsystems take care of answering queries (reporting) and the domain for processing and storing updates can stay focused.

Event Sourcing makes it possible to see how we got to the current state and query how the state looked liked in the past. Essentially it means that we have to capture all changes to an application state as a sequence of events. Sculptor provides a default implementation of EventSourcing, which can be extended and customized to fit specific needs.

Read more about how to use the new event features of Sculptor here. We will blog more about this topic later, so stay tuned.

New Home Page

2010-06-04T22:12:00.005+02:00

New Sculptor home page is live: http://sculptor.fornax-platform.org/

I have also updated the original TSS article: Improving Developer Productivity with Sculptor.

Event Sourcing Snapshots

2010-06-01T20:30:00.003+02:00

I got a relevant question on yesterdays post. How does the snapshot mechanism work?

The easiest way to reconstruct a historical state is to start with a clean database, maybe populated with static reference data, and then apply all events in order up to the point you are interested in. This will take forever if your system has been running for a while, it contains a lot of events.

The trick to make it work is to periodically take a snapshot of the complete database. When doing replay it can start with the preceding snapshot and only replay events after the snapshot time up to the time you are interested in.

Taking a snapshot can be implemented in various ways. I have tried an approach that aims at being able to take a snapshot without stopping/locking normal operation, i.e. inflow of events and processing of them can continue.

The steps to create snapshots are illustrated by the following figure. It also illustrates a query to replay events to a specific point in time.

How do I copy database? For MongoDB there is a simple copydb command, which can be executed from the application. Exactly how to do this with other databases is vendor specific, but I guess it is possible somehow. At least in command line admin mode. Since I always copy from a read-only (snapshot) database it is should be a consistency safe operation.

These operations will of course not be super fast and temporary databases for historical queries must probably be prepared in advanced to have decent response times. If you need to do a lot of historical queries you need something more, such as a separate system responsible for answering such queries. It may be a data warehouse, a denormalized database, or whatever tailored for the purpose. This separate system is fed from the same original event stream. The design is called CQRS.

Prototyping Event Sourcing

2010-05-31T18:30:00.002+02:00

In Domain Language Newsletter March 2010 we could read Eric Evans writeup on Domain Events.

Over the last few years it has become clear that it is very useful to add a new pattern to the DDD "Building Blocks" (Entities, Value Objects, Services, etc.) -- Domain Events. This pattern has been around for a long time. Martin Fowler wrote a good description.

Andreas and I have lately been prototyping on some mechanism to better support Event-Driven Architecture in Sculptor. In this article I will describe how to do Event Sourcing.

A domain model typically holds current state of the world. Event Sourcing makes it possible to see how we got to this state and query how the state looked liked in the past. Essentially it means that we have to capture all changes to an application state as a sequence of events. These events can be used to reconstruct current and past states.

In the prototype we have used Martin Fowler's shipping system example. The full source code for the sample is available in Subversion. I have developed Event Sourcing with existing Sculptor mechanisms. Next step will be to move the general pieces into the tool so that it can be easily used. I would like to share the ideas right away in this blog.

The domain model for the shipping system is very simple. Ships that carry cargo and move between ports. It looks like this in Sculptor DSL:

Entity Ship {
   gap
   - ShipId shipId key
   String name
   - Port port nullable
   - Set<@Cargo> cargos

   Repository ShipRepository {
       gap
       inject @CargoRepository
       save;
       findByKey;
   }
}

Entity Port {
   - UnLocode unlocode key;
   String city
   - Country country

   Repository PortRepository {
       save;
       findByKey;
   }
}

Entity Cargo {
   gap
   String cargoId key
   boolean hasBeenInCanada

   Repository CargoRepository {
       save;
       findByKey;
   }
}

BasicType ShipId {
   String identifier key
}

"United nations location code."
BasicType UnLocode {
   String identifier key
}

enum Country {
   US,
   CANADA
}

When a ship arrives at port it is registered in the TrackingService, recordArrival operation. Instead of taking that information and directly save it to current domain objects, the TrackingService creates a DomainEvent and pass it in to the processor for further execution.

public void recordArrival(DateTime occured, Ship ship, Port port) {
   DateTime now = new DateTime();
   ArrivalEvent event = new ArrivalEvent(occured, now, ship.getShipId(), port.getUnlocode());
   getDomainEventProcessor().process(event);
}

The DomainEventProcessor stores the event, which is important for audit and possibility to replay events to reconstruct past states. The DomainEventProcessor is generic, knows nothing about the shipping domain. It dispatches the event to an EventHandler that knows how to process the shipping specific events. As a start I use simple Spring dependency injection to select EventHandler, but we will probably use a slightly more sophisticated mechanism later. The important thing is that the generic DomainEventProcessor doesn't know how to process events, so it delegates to application specific event handlers.

A useful utility for dispatching events to separate methods is available in commons beanutils. The ShippingEventHandler looks like this (error handling removed):


public void handleEvent(DomainEvent event) {
   dispatch(event);
}

/**
* Runtime dispatch to handle method with correct event parameter type
*/
protected void dispatch(DomainEvent event) {
   MethodUtils.invokeMethod(this, "handle", new Object[] { event });
}

public void handle(ArrivalEvent event) {
   Ship ship = getShipRepository().findByKey(event.getShip());
   Port port = getPortRepository().findByKey(event.getPort());
   ship.arrival(port);
   getShipRepository().save(ship);
}

public void handle(DepartureEvent event) {
   Ship ship = getShipRepository().findByKey(event.getShip());
   Port port = getPortRepository().findByKey(event.getPort());
   ship.departure(port);
   getShipRepository().save(ship);
}

The events are also defined in Sculptor DSL. So far I have used ordinary ValueObjects for the events but we will add a special syntax for defining events like this:

DomainEvent ArrivalEvent {
   - ShipId ship
   - UnLocode port
}

DomainEvent DepartureEvent {
   - ShipId ship
   - UnLocode port
}

What have we done? We have made the design more complicated by an intermediate event step, but we have also a foundation for all the exciting things that can be done with Event Sourcing.

We can query the state of a ship for a specific point in time, and it is not only the state of the ship, the complete domain model can be used as usual at a specific point in time.


    DateTime to = travelStart.plusDays(15);
    ReplaySpecification replaySpec = new ReplaySpecification().withTo(to).withTarget(tmpDb);
    domainEventProcessor.replay(replaySpec);
    DbManager.setThreadInstance(tmpDb);
    Ship ship = referenceDataService.getShip(shipId);
    Port port = ship.getPort();
    Set<Cargo> cargos = ship.getCargos();

When doing this prototype we have used MongoDB, which is excellent for these kind of features because:

it is schema-less, so any event type can be stored without predefined schema
it has low latency, so replay of events are fast
it is easy to make new database instances and copy database, which make replay and snapshot mechanisms simple

Note that in the above example we replay the events into a temporary database (tmpDb), a Parallel Model. This is a full featured MongoDB instance and therefore we have all the ordinary query capabilities, i.e. exactly the same domain model including repositories can be used with the temporary database as with the ordinary database.

The generic DomainEventProcessor provides several methods to operate on the events. For example to speed up replay of large data volumes it is possible to create snapshots. Latest preceding snapshot is used as base when replaying, i.e. only the events after the snapshot time need to be processed.

The event processor module is defined like this in the prototype. The idea is that it will be provided automatically by Sculptor. Of course with customization possibilities to support variations.

Module event {
   Service DomainEventProcessor {
       inject @DomainEventRepository
       inject @SnapshotRepository

       process(@DomainEvent event);
       getAllEvents => DomainEventRepository.findAll;
       replayAll;
       replayUpTo(DateTime timePoint);
       replay(@ReplaySpecification spec);
       save(List<@DomainEvent> events);
       String createSnapshot(DateTime timePoint);
   }

   abstract ValueObject DomainEvent {
       gap
       DateTime occured index;
       DateTime recorded;

       Repository DomainEventRepository {
           save;
           save(List<@DomainEvent> entities);
           findAll(PagingParameter pagingParameter);
           findBetween(DateTime from, DateTime to, PagingParameter pagingParameter);
           protected findByCondition(PagingParameter pagingParameter);
       }
   }

   ValueObject ReplaySpecification {
       not persistent
       DateTime from nullable
       DateTime to nullable
       DB target nullable   
   }

   ValueObject SnapshotInfo {
       DateTime timePoint index
       String name

       Repository SnapshotRepository {
           @SnapshotInfo findLatest(DateTime timePoint);
           String snapshotName(DateTime timePoint);
           @SnapshotInfo createSnapshot(DateTime timePoint);
           copyDb(String fromDbName, String toDbName) => CopyDbAccessObject;
           protected findByCondition;
           protected save;
       }
   }
}

We are also prototyping how to add other event mechanisms, such as publish/subscribe and integration with various products, such as Spring Integration, Apache Camel and Akka. More about that later.

Generating Syntax/Railroad diagrams from Xtext

2010-05-04T16:18:00.025+02:00

Problem

Some months ago we started to speak with Patrik about syntax (railroad) diagrams for Sculptor. Syntax of Sculptor DSL is very rich now and we have no formal reference documentation. In Advanced and Developer doc many details of language are described but pure syntax description would be very helpful.

Solution

I started looking for some tools. I found following generator for syntax diagrams:

http://www-cgi.uni-regensburg.de/~brf09510/syntax.html

Little bit old but output looks good. Now we need only BNF definition for our language. I looked at Xtext definition and start to manually change it. For example this Xtext:

DslValueObject :
  (doc=STRING)?
  (abstract?="abstract")? "ValueObject" name=ID ("extends" (("@"extends=[DslValueObject]) | (extendsName=DslJavaIdentifier)))? "{"
    ("package" "=" package=DslJavaIdentifier )?
    (((notOptimisticLocking?=NOT "optimisticLocking") | ("optimisticLocking")) |
     ((notImmutable?=NOT "immutable") | ("immutable")) |
     ((cache?="cache") | (NOT "cache")) |
     ((gapClass?="gap") | (noGapClass?="nogap")) |
     (scaffold?="scaffold") |
     ("hint" "=" hint=STRING) |
     ("databaseTable" "=" databaseTable=STRING) |
     ("discriminatorValue" "=" discriminatorValue=STRING) |
     ("discriminatorColumn" "=" discriminatorColumn=STRING) |
     ("discriminatorType" "=" discriminatorType=DslDiscriminatorType) |
     ("discriminatorLength" "=" discriminatorLength=STRING) |
     ("inheritanceType" "=" inheritanceType=DslInheritanceType) |
     ("validate" "=" validate=STRING) |
     ((notPersistent?=NOT "persistent") | ("persistent")))*
    ((attributes+=DslAttribute) |
     (references+=DslReference))*
    (repository=DslRepository)?
  "}";

have to be transformed to this BNF notation:

DslValueObject :
  STRING?
  "abstract"? "ValueObject" ID ("extends" (("@"DslValueObject) | DslJavaIdentifier))? "{"
    ("package" "=" DslJavaIdentifier )?
    ((NOT? "optimisticLocking") |
     (NOT? "immutable") |
     (NOT? "cache") |
     ("gap" | "nogap") |
     "scaffold" |
     ("hint" "=" STRING) |
     ("databaseTable" "=" STRING) |
     ("discriminatorValue" "=" STRING) |
     ("discriminatorColumn" "=" STRING) |
     ("discriminatorType" "=" DslDiscriminatorType) |
     ("discriminatorLength" "=" STRING) |
     ("inheritanceType" "=" DslInheritanceType) |
     ("validate" "=" STRING) |
     (NOT? "persistent"))*
    (DslAttribute |
     DslReference)*
    DslRepository?
  "}";

As you can see, it is not that different. In first round I did it manually but I found that it will be possible automatize with short sed script. Store following to convertToBNF.sed:

s/[A-Za-z]\+.=//g
s/\[//g
s/\]//g
s/""/" "/g
s/(\([A-Za-z"]\+\))/\1/g
s/^enum *//
s/^terminal *//
/^grammar /d
/^generate /d

You can apply it to Xtext file by:

sed -f convertToBNF.sed Sculptordsl.xtext > Sculptordsl.bnf

Now you can copy and paste content of Sculptordsl.bnf to text area on previous mentioned site:

http://www-cgi.uni-regensburg.de/~brf09510/syntax.html

For making it nice we will use bigger font and than resize to 50% with anti-aliasing. Set font size to 22 and image size to double of expected result image. For me with our example it's width 2000 and height 2000. For big grammars more images are generated on page. Just save whole page to some directory. All new browsers always store html file and all other resources like images, styles, javascripts to separate directory. Go to this resource directory and run following command (this need ImageMagic convert tool):

for i in *.png; do convert $i -trim -resize 52% ${i%.png}-S.png; done

Now you have small version of original syntax diagrams with nice anti-aliasing. Here is my result:

Conclusion

As you can see, generating nice looking syntax diagrams from Xtext grammar isn't that difficult. In next days we will make page with all Sculptor DSL syntax diagrams. Question is: "Can I apply it for my Xtext grammar too?". Sed tool is working with pure text. It's possible to confuse sed with strange declaration but for normal grammars it should work. It will be better to have special tool which will parse Xtext notation, maybe some transformation and simplification for nicer results and generate BNF notation directly, but this is bigger job and sed work nice with Sculptor Xtext. Here is full version of sed script which I'm using now for transforming Xtext grammar to BNF notation. You can see also some transformations at end of sed script which do beautification of BNF:

s/[A-Za-z]\+.=//g
s/\[//g
s/\]//g
s/""/" "/g
s/(\([A-Za-z"]\+\))/\1/g
s/^enum *//
s/^terminal *//
/^grammar /d
/^generate /d
$i \
ID :\
   LETTER (LETTER | NUMBER)*\
\
STRING :\
  """" CHAR+ """";\
\
LETTER :\
  "A-Z" | "a-z";\
\
CHAR :\
  LETTER | NUMBER | " ~!@#$%&*()_+-={}[]:|\';/.,;<>?";\
\
INTEGER :\
  NUMBER+;\
\
NUMBER :\
  '0-9';\

# Syntactic sugars
# X | (Y X) => Y? X
s/\([A-Za-z"]\+\) *| *(\([A-Za-z"]\+\) *\1)/\2? \1/g

# X | (X Y) => X Y?
s/\([A-Za-z"]\+\) *| *(\1 *\([A-Za-z"]\+\))/\1 \2?/g

# (X Y) | Y => X? Y
s/(\([A-Za-z"]\+\) *\([A-Za-z"]\+\)) *| *\2/\1? \2/g

# (X Y) | X => X Y?
s/(\([A-Za-z"]\+\) *\([A-Za-z"]\+\)) *| *\1/\1 \2?/g

MongoDB with Sculptor - Repository

2010-04-28T21:20:00.002+02:00

This post is part of a series of articles describing the support for MongoDB in Sculptor. This post explains how Repositories with useful operations are defined and how queries can be expressed in terms of the domain.

In the visualization of the blog sample you maybe noticed that there are Services with findById, findAll, save and delete even though they were not explicitly defined in the model. They come from that the domain objects are marked with with scaffold. That automatically generates some predefined CRUD operations in the Repository and corresponding Service.


    Entity BlogPost {
            scaffold
            - Blog inBlog
            String slug key
            String title
            String body
            DateTime published nullable
            Set<String> tags
            - Author writtenBy
            - List<Comment> comments opposite forPost
    }

You can also define the needed operations explicitly like this:


    Entity BlogPost {
            - Blog inBlog
            String slug key
            String title
            String body
            DateTime published nullable
            Set<String> tags
            - Author writtenBy
            - List<Comment> comments opposite forPost

            Repository BlogPostRepository { 
                save;
                delete;
                findAll(PagingParameter pagingParameter);
                findById;
                findByKey;
             List<@BlogPost> findPostsInBlog(@Blog blog);
             List<@BlogPost> findPostsWithComments => AccessObject;
             List<@BlogPost> findPostsWithTags(Set<String> tags);
             protected findByCondition;
            }

You don't need to do any manual coding for the built in operations, such as save, delete, findAll, findById, findByKey.
findByKey is for the natural key, i.e. the attributes marked with key (slug above).

There is good support for pagination and sorting.

As you see in the above sample it is also possible to define your own repository operations, such as findPostsWithTags.

findByCondition

The most interesting finder is findByCondition. Queries can expressed with an internal DSL that support code completion and refactoring. It is best illustrated with a few samples.


    public List<BlogPost> findPostsInBlog(Blog blog) {
        List<ConditionalCriteria> condition = criteriaFor(BlogPost.class)
            .withProperty(inBlog()).eq(blog)
            .orderBy(published()).descending()
            .build();
        return findByCondition(condition);
    }


    public List<BlogPost> findPostsWithGreatComments() {
        List<ConditionalCriteria> condition = criteriaFor(BlogPost.class)
            .withProperty(comments().title()).ignoreCaseLike(".*great.*")
            .and().withProperty(published()).isNotNull()
            .orderBy(published()).descending().build();
        return findByCondition(condition);
    }

The ConditionalCriteriaBuilder has support for defining queries with eq, like, between, lessThan, greaterThan, in, and, not, orderBy, and some more.

You can of course also work directly with the underlaying MongoDB DBCollection and DBObject for doing advanced queries that might not be supported by findByCondition.

MongoDB with Sculptor - Associations

2010-04-28T21:10:00.001+02:00

This post is part of a series of articles describing the support for MongoDB in Sculptor. This post explains how associations and inheritance are managed.

MongoDB is not a relational database and there is no such thing as joins. You can still use associations between domain objects. Associations can be of two main categories, either embedded or reference by id.

Aggregates

Aggregates are one of the core building blocks in Domain-Driven Design (DDD). MongoDB has perfect support for aggregates, since an associated object can be stored as an embedded document, i.e. it belongs to parent object and cannot be shared between several objects.

Let us repeat what DDD says about aggregates:

An Aggregate is a group of associated objects which are considered as one unit with regard to data changes. The Aggregate is demarcated by a boundary which separates the objects inside from those outside. Each Aggregate has one root. The root is an Entity, and it is the only object accessible from outside. The root can hold references to any of the aggregate objects, and the other objects can hold references to each other, but an outside object can hold references only to the root object.
- from DDD Quickly

Sculptor will validate the reference constraints described in the quote above. Repositories are only available for aggregate roots. Aggregates are defined with belongsTo or not aggregateRoot in the owned DomainObjects.

A typical aggregate in the blog sample is that Comment belongs to BlogPost


        Entity BlogPost {
            String slug key
            String title
            String body
            DateTime published nullable
            - List comments opposite forPost
        }

        ValueObject Comment {
            not aggregateRoot
            - BlogPost forPost opposite comments
            String title
            String body
        }

An aggregate can of course include several classes as in this sample:


        Entity Cargo {
            - TrackingId trackingId key;
            - Location origin required;
            - Location destination required;
            - Itinerary itinerary nullable opposite cargo;
            - Set<HandlingEvent> events opposite cargo;

        }

        BasicType TrackingId {
            String identifier key
        }

        ValueObject Itinerary {
            belongsTo Cargo
            - Cargo cargo nullable opposite itinerary
            - List<Leg> legs
        }

        ValueObject Leg {
            belongsTo Cargo
            - CarrierMovement carrierMovement
            - Location from 
            - Location to
        }

In above sample the TrackingId, Itinary and Leg are all stored toghether with the Cargo. BasicTypes are also stored as embedded documents.

Reference by Id

The other alternative is to store ids of the referred objects. In the domain objects there are generated getters that lazily fetch associated objects from the ids. This means that you don't have to work with the ids yourself, you can follow associations as usual, but be aware that an invocation of such a getter might need to query the database.


    Set writers = blog.getWriters();

In the same way you can modify unowned associations by working with objects rather than ids.


        Author pn = new Author("Patrik");
        pn = authorService.save(getServiceContext(), pn);
        blog.addWriter(pn);
        Author ak = new Author("Andreas");
        ak = authorService.save(getServiceContext(), ak);
        blog.addWriter(ak);
        blogService.save(getServiceContext(), blog);

Referential integrity of the stored ids are not enforced. It must be handled by your program. Lazy getters of associations will not fail if referred to object is missing, they will return null for single value references and ignore missing objects for collection references. This means that you can cleanup dangling references by fetching objects, populate associations by invoking the getters and then save the object.

Inheritance
The MongoDB feature of Sculptor has full support for inheritance. It is even possible to do polymorphic queries.


        abstract Entity Media {
            String title !changeable
            
            Repository MediaRepository {
                List<@Media> findByTitle(String title);
                protected findByCondition;
            }
        }
        
        Entity Book extends @Media {
            String isbn key length="20"
        }
        
        Entity Movie extends @Media {
            String urlIMDB key
            Integer playLength
            - @Genre category nullable
        }

MongoDB with Sculptor - Data Mapper

2010-04-28T21:00:00.001+02:00

This post is part of a series of articles describing the support for MongoDB in Sculptor. This post explains how the mapping between domain objects and MongoDB data objects is performed.

When working with mongoDB Java API the DBObject plays a central role. It is like a key value Map. Values can be of most types and also collections and other DBObjects for nested documents. Sculptor also adds support for Enums and Joda date time classes.

Implementing mapping of domain objects to DBObjects is a tedious and error prone programming task that is a perfect candidate for automation. Sculptor comes in handy for this. Sculptor generates data mapper classes that converts domain objects to DBObjects. It is useful to not have to write those mappers by hand.

An early definition of a BlogPost in the blog sample might look like this in Sculptor model:


        Entity BlogPost {
            String slug key
            String title
            String body
        }

From this Sculptor generates domain object class in java with appropriate getters, setters, constructors, equals, etc. Sculptor also generates a mapper for converting the BlogPost to/from DBObject. The generated code looks like this:


    public DBObject toData(BlogPost from) {
        if (from == null) {
            return null;
        }

        DBObject result = new BasicDBObject();

        if (from.getId() != null) {
            ObjectId objectId = ObjectId.massageToObjectId(from.getId());
            result.put("_id", objectId);
        }

        result.put("slug", from.getSlug());
        result.put("title", from.getTitle());
        result.put("body", from.getBody());
        
        return result;
    }

and in the other direction:


    public BlogPost toDomain(DBObject from) {
        if (from == null) {
            return null;
        }

        String slug = (String) from.get("slug");

        BlogPost result = new BlogPost(slug);

        if (from.containsField("_id")) {
            ObjectId objectId = (ObjectId) from.get("_id");
            String idString = objectId.toStringMongod();
            IdReflectionUtil.internalSetId(result, idString);
        }

        if (from.containsField("title")) {
            result.setTitle((String) from.get("title"));
        }
        if (from.containsField("body")) {
            result.setBody((String) from.get("body"));
        }

        return result;
    }

I'm happy that I don't have to write and especially maintain that kind of code. The generation is not a one time shot. When you change the model the domain objects and the mappers are regenerated.

Domain objects can of course also have behavior, otherwise it wouldn't be a rich domain model. The behavior logic is always written manually. Separation of generated and manually written code is done by a generated base class and manually written subclass, a gap class. It is in the subclass you add methods to implement the behavior of the domain object. The subclass is also generated, but only once, it will never be overwritten by the generator.

You might need to add some hand written code to customize the mappers. That is easy. In the model you can specify that you need a subclass that you can implement yourself. This is very useful for doing data migration.

It can also be good to know that fields marked with transient are are not stored, but they are loaded if they exist in the retrieved documents. This can also be used for data migration.

By default the names in the data store are the same as the names in the Java domain objects. In case you need to use other names it is possible to define that in model with databaseTable and databaseColumn.


        Entity BlogPost {
            databaseTable="BlogEntries"
            String slug key
            String title
            String body databaseColumn="content"
        }

You have probably selected MongoDB for its low latency. Then you want the data mapper to be fast also. The mappers provided by Sculptor are generated code that runs at full speed. Alternative solutions using reflection or intermediate String JSON format is probably not as fast.

MongoDB with Sculptor - Introduction

2010-04-27T11:30:00.004+02:00

MongoDB bridges the gap between key-value stores (which are fast and highly scalable) and traditional RDBMS systems (which provide rich queries and deep functionality).

Would you like to?

Use a rich persistent domain model à la DDD with automatic mapping to MongoDB data structures.
Use associations even though the underlaying data store is not relational.
Express queries in refactoring safe terms of the domain model.
Get CRUD operations, and GUI, for free.

Then you should try Sculptor with the brand new MongoDB target implementation.

In the next few days I will describe the features one-by-one, but first an brief overview of what Sculptor provides for efficient development of a typical enterprise or web application with MongoDB as persistent store.

Sculptor is a productivity tool that let you express your design intent in a textual DSL, from which Sculptor generates high quality Java code and configuration. Read more in What is Sculptor?

Sculptor generates data mapper classes that converts domain objects to/from MongoDB data structures, DBObjects.

Sculptor provides generic repository operations for use with MongoDB. This includes operations such as save, delete, findById, findByKey, findByCondition, and some more.

Queries can be expressed with a slick fluent api that support code completion and refactoring.

Rich support for associations. Aggregates are stored as a embedded documents. Other associations are stored with referring ids. In the domain objects there are generated getters that lazily fetch associated objects from the ids. This means that you don't have to work with the ids yourself, you can follow associations as usual.

A sample blog application may look like this when defined in Sculptor textual DSL:

Visualization, generated by Sculptor, of the above model looks like this:

Read the other posts that describes the features in more detail:

More hands on instructions of how to use Sculptor is available in the Sculptor MongoDB Tutorial.

Support for mongoDB started

2010-03-31T21:30:00.004+02:00

I think NoSQL datastores are a very interesting complement to relational databases. I think mongoDB is a good choice for the rich persistent domain models that are typical for a Sculptor application. Today's article at InfoQ about mongoDB mention the relation between document stores and DDD.

We have started prototyping and it looks promising. The idea is that that Sculptor will provide:

Access objects (implementation of repository operations) that interact with mongoDB.
Good query support (e.g. findByCondition).
Automatic mapping between domain objects and mongoDB documents (DBObject) will be generated.
Different types of association will be supported. Aggregates are typically embedded documents and other associations are stored as id references. Maybe we will provide a mechanism to lazy load associated documents.

Improved Graphviz Visualization

2010-03-20T22:10:00.015+01:00

I have improved the visualization with several new features. You can use this new diagram generator right away as described in the end.

Several diagrams with different focus and level of detail are generated. Below are samples of the different types of diagrams. You can click on the images below to see them in full scale.

Module dependencies:

Overview:

One separate diagram for each module:

Diagram with focus on elements marked as core domain (hint="umlgraph=core"). Note that core domain objects also have another font color (and background color) in all diagrams.

And finally, a diagram with full detail of all objects.

All colors can be configured using generator properties file and it is also possible to set the color for individual elements using a hint in model file (hint="umlgraph.bgcolor=FFCC99"). It is also possible to hide elements using hint="umlgraph=hide".

You can generate the new diagrams in your Sculptor project right away. Version 1.8.1-SNAPSHOT includes this, but even if you need to use an older version (not snaphot) you can drop in the new generator templates in your project. Download from trunk

umlgraphhelper.ext to your src/main/resources/extensions/ and

UMLGraph.xpt to your src/main/resources/templates/

To generate images (png) from the generated Graphviz dot files you can use the new fornax-graphviz-m2-plugin. Add the following to your pom.xml:


  <plugin>
   <groupId>org.fornax.toolsupport</groupId>
   <artifactId>fornax-graphviz-m2-plugin</artifactId>
   <version>1.0.0</version>
   <executions>
    <execution>
     <phase>generate-resources</phase>
     <goals><goal>run</goal></goals>
    </execution>
   </executions>
  </plugin>

You also need to install the excellent Graphviz tool.

Success Story - Using my Own Tool

2010-03-11T19:07:00.004+01:00

I’m the founder and lead developer of Sculptor. Why would I write a success story? They should be written by users. Right, I’m a user of my own tool. This is maybe the key to why the tool is working as good as it does.

Since May 2009 I have been working with a customer in the logistics business. I’m involved in several projects but in my “main” project we develop a Track & Trace system. We receive events with parcel state information from several production systems in different countries. Volumes around 500 million tracking events during 6 months, which is the period we need to support track&trace searching.

In this department JEE was greenfield development. They have been using Java before, but not with application servers and modern frameworks. We established guidelines and tools for the new platform.

The introduction of Sculptor started with a technical prototype of the domain model. About 20 entities were developed in 2 days. Test driven. Of course this was only an initial draft, but it was important to quickly be able to try our ideas in practice. We have done a lot of refactoring of the domain model since then and Sculptor doesn’t sit in the way for doing changes. It simplifies change.

When evaluating Sculptor for the prototype my team developers stressed the importance of being able to remove the tool and continue without starting from scratch. It was good to see that the developers were able to quickly grasp the concepts and be able to use the tool immediately without any lengthy education.

When I and the other developers in my team had used Sculptor for a while another team wanted to try it also. I helped with converting existing code. The conversion and initial learning was smooth.

We listed the following benefits of using Sculptor for all new projects at the department:

Improved productivity. Generate code from high level description. Less manual mistakes. Relieved the developers from repeating and tedious programming tasks.
High quality of generated code. As if written manually, but better. We actually found several flaws in the hand written code when converting it to Sculptor.
Consistent design and structure. The department was running several projects in parallel, with different developers, and varying experience of JEE. Consistency between solutions was considered very important from a maintenance perspective.
Easy to learn. The developers that tried Sculptor found it intuitive to use. “It’s nothing magic”.
Great possibilities for customization. We did some adjustments to fit the database naming conventions.
Good design. The design of the generated code match our guidelines, JEE, JPA, DDD.
Easy to remove the tool. No dynamic runtime magic.

We have continued to use Sculptor successfully. We use Sculptor for all new projects. Several new developers have joined the teams and quickly learnt how to use Sculptor. We are 11 developers in 4 projects right now.

Bonus Material

2010-02-21T21:11:00.006+01:00

For my customer's project I created a small addition to Sculptor, that might be useful for you also. The domain model is important for communication, and we need to document it. The difficult thing with documentation is to keep it up to date with changes. Placing the descriptions close to the code makes it easier to do the changes to the documentation at the same time as when you change the code. Since we use Sculptor it was a natural choice to write the descriptions in the model file and generated the documentation.


     "A location is our model is stops on a journey, such as cargo
     origin or destination, or carrier movement endpoints."
     Entity Location {
         not optimisticLocking
         - @UnLocode unLocode key
         'Actual name of this location, e.g. "Stockholm"'
         String name not changeable
      
         Repository LocationRepository {
             @Location find(@UnLocode unLocode) throws LocationNotFoundException;
             findAll;
             protected findByKeys;
         }
     }

From the model we generate HTML documentation of all domain objects including their attributes and associations.

Try it if you like. It is easy to understand and adjust to fit your needs.

Download this generation template and place it in src/main/resources/templates/
In SpecialCases.xpt you add


«AROUND templates::Root::Root FOR Application»
   «targetDef.proceed()»
   «EXPAND DomainModelDoc::start»
«ENDAROUND»

The generated result is located in src/generated/resources/DomainModelDoc.html

Sculptor 1.8.0 Released

2010-02-16T20:45:00.001+01:00

Only two weeks after the big 1.7.0 we are happy to deliver another major release - 1.8.0. This time it is no new functionality, but the usability of DSL editor is much, much better.

We have migrated from openArchitectureWare 4.3.1 to Xtext/Xpand/MWE 0.7.2 at Eclipse.org. Many thanks to Todd Ferrell for doing most of this migration.

Now, content assist (ctrl+space) works as you would expect.

Another useful feature is formatting (pretty printing) of model files (ctrl+shift+F).

Watch the DSL editor in action in the demo below, or even better try it yourself.

Customization of webflows

2010-02-09T07:42:00.000+01:00

With the new 1.7 release of sculptor the possibilities to customize flows for the web client is much better.
With our Library example here is what you can do.
For example, lets say you want to implement filter functionality for the library list feature. Here is the steps to do that:

1) Specify that you want gap-files for the list library feature, so in your model.guidesign:


gui Library for Library {
Module for media {
    ListTask for Library {
        gap
    }
}
}

Now you have a bunch of files for the feature:

src/generated/java/org...library/ListLibraryActionBase.java
src/main/java/org...library/ListLibraryAction.java -> gap
src/generated/java/org...library/ListLibraryForm.java
src/WEB-INF/generated/flows/media/listLibrary/listLibrary-base.xml
src/WEB-INF/generated/flows/media/listLibrary/list_include.xhtml
src/WEB-INF/flows/media/listLibrary/listLibrary-flow.xml -> gap
src/WEB-INF/flows/media/listLibrary/list.xhtml -> gap

2) Edit the media/listLibrary flow:



<?xml version="1.0" encoding="UTF-8"?>
<flow xmlns="http://www.springframework.org/schema/webflow"
     xmlns:ns0="http://www.w3.org/2001/XMLSchema-instance"
     ns0:schemaLocation="http://www.springframework.org/schema/webflow
     http://www.springframework.org/schema/webflow/spring-webflow-2.0.xsd"
     parent="media/listLibraryBase">
 <view-state id="list">
     <transition on="filterLibrary" to="listByFilter" />
 </view-state>
 <view-state id="listByFilter" model="listLibraryForm"
     view="/WEB-INF/flows/media/listLibrary/list.xhtml" parent="media/listLibraryBase#list">
     <on-render>
         <evaluate
             expression="listLibraryAction.findByFilter(flowRequestContext)" />
     </on-render>
 </view-state>
</flow>

3) Edit the ListLibraryAction, add the method:



public String findByFilter(RequestContext ctx) {
    getRepository().clear();

    List<library> allLibraries = getLibraryService().findAll(ServiceContextStore.get());
    List<library> filtered = new ArrayList<library>();
    String filter = ctx.getRequestParameters().get("libraryFilter");
    for (Library library : allLibraries) {
        if (library.getName().startsWith(filter)) {
            filtered.add(library);
        }
    }
    formObject(ctx).setAllLibraries(filtered);
    return "success";
}

4) Add a simple form with a text field and a button to the media/listLibrary/list.xhtml-file:


<html xmlns="http://www.w3.org/1999/xhtml" xmlns:ui="http://java.sun.com/jsf/facelets"
 xmlns:f="http://java.sun.com/jsf/core" xmlns:t="http://myfaces.apache.org/tomahawk"
 xmlns:h="http://java.sun.com/jsf/html" xmlns:c="http://java.sun.com/jstl/core"
 xmlns:a="ApplicationTaglib">
 <body>
     <ui:composition template="/WEB-INF/common/template.xhtml">
         <ui:define name="content">
             <h1>
                 <h:outputFormat value="#{msg['list.header']}">
                     <f:param
                         value="#{msgMedia['model.DomainObject.Library.plural']}" />
                 </h:outputFormat>
             </h1>
             <h:form xmlns="http://www.w3.org/1999/xhtml"
                 xmlns:ui="http://java.sun.com/jsf/facelets" xmlns:f="http://java.sun.com/jsf/core"
                 xmlns:t="http://myfaces.apache.org/tomahawk"
                 xmlns:h="http://java.sun.com/jsf/html" xmlns:c="http://java.sun.com/jstl/core"
                 xmlns:a="ApplicationTaglib">
             <div>
                 <label for="_libraryFilter">#{msgMedia['model.DomainObject.Library.filter']}: </label>
                 <input type="text" value="#{requestParameters.libraryFilter}" name="libraryFilter" id="_libraryFilter"/>
                 <h:commandButton value="#{msgMedia['model.DomainObject.Library.filterButton']}" action="filterLibrary" />
             </div>
             </h:form>
             <ui:include
                 src="/WEB-INF/generated/flows/media/listLibrary/list_include.html" />
         </ui:define>
     </ui:composition>
 </body>
</html>

Note that the default generated table showing the list result is used by including the generated file. In case you need to modify the generated content you can either overwrite the code generation templates in WebSpecialCases.xpt or simply copy the generated file and maintain it manually

That's it.

Two Success Stories from Factory4Solutions

2010-02-02T20:20:00.001+01:00

Introduction
I work as architect at company Factory4Solutions (www.f4s.sk). Our
primary customers are banks, insurance companies, telecommunication
sector. We was using MDA approach long before. We started around 1997
with our own home made solution based on RationalRose, XML and custom
generator. Than we switched to AndroMDA. Tool was nice but stalled in
development around 2007. Then I start investigate another MDA tools.
We were looking for tool based on open technologies, universal and able
to generate JEE code. After playing little bit with proof of concept
projects we got first real project in which we could apply it.

Insurance company
Decision was not easy. We have to migrate existing portal with huge
amount of existing data to new portal. Persistent layer was hand
written in Oracle TopLink and front-end was in JSPs. I expected many
problems because hand written persistent layer is usually full of
nonstandard approaches which are hard to achieve in generic MDA tool.

However for me as an architect it was important to achieve
standardization because we had complicated deployment mode. Because of
security, portal was split to two. First was 3 tier extranet deployed
on WebLogic cluster and internal part which had similar functionality
but was running only on single server. Both portals were using same
database. For easier development I decided go with only one code base.
At build time we split the application and created more WAR files (Full
intranet, Extranet service tier, Extranet web tier).

I was really afraid about MDA approach. I started with SQL DDL and
created first version of model design file for Sculptor. Then I started
to tweak model file to fit existing database. After 2 days I was ready

with our model and first version of important services.

Thanks to strict Domain Driven Design which Sculptor follow I moved
code to developers to finish services and develop web frontend. Thanks
to Spring Remoting it took just 1 day to develop build procedure for 3
different WAR files. Except some small problems when developer used
some utility methods from frontend on backend. The architecture is stable,
system is maintainable and extendable. Today we still use one code
base also for demo system, batch processing system and Web Service
layer. Flexibility of Sculptor is endless. When you reach the border
with what Sculptor allow, you are still free to override parts of
generated code thanks to Xpand language.

Project facts:

6 developers + 0.5 architect + 1 project manager + 0.5 auality assurance + 0.5 analyst for 9 month
108.000 lines of java code, 34.000 lines of JSP code
Processing big reports from customers take 5-6 minutes (10 MB XML)
20.000 reports a day are send in peaks

Investment company
We had advantage that we should start from scratch. Now project
consist of 40 entities and 26 services. Very complex security and
entity processing by complicated business processes involving bank,
customers, internal users, stock exchange, e-mail processing and
advisor networks. It was developed fully with Sculptor for entity and
service tier.

This is our first project where we used Sculptor also for generating
web frontend. We used smartclient (Sculptor allow to generate different
frontends). We are satisfied with speed of development and nice GUI
which users appreciate.

We also used very unique concept of waving rule engine - Drools

in front of service tier. This way we validate objects, sending

announcement, sending business alerts and also for complex price

list processing which consist of 8 different payment tables applied

at complex conditions. Result is very robust system which allow at

one side dynamic change but on other side very stable.

We are now running User Acceptance Tests at client side.

Project facts:

2 developers + 0.5 architect + 0.5 quality assurance during 6 month
9.000 lines of hand written code, 3.600 lines of test code, 49.000 lines of generated code

Conclusion
Today we are using Sculptor for all our projects. We master it to nice
extend and we now participate in development of new versions. I'm very
happy with new 1.7 release. This brings some nice features.
Specifically it's Smartclient frontend, hints for custom generation of
code and condition criteria builder with nice Java DSL (type and name
checking and IDE autocompletion).

This story was written by Pavel Tavoda, Factory4Solutions

An Old Success Story

2010-02-02T20:10:00.001+01:00

It was a completely new project started from scratch. The main goal of a project was processing of a big files and then generates reports. Despite the fact that installation of a product includes configuration for personal and network use, we choose choose architecture of app that includes Tomcat and Hibernate. After some investigation I found Sculptor. It uses well known (by me) technologies and best practices for that moment. When main functionality was implemented in a draft state Sculptor helped to start with a simple domain model. It was not really big model. This model includes about 50 entities and services (Users, Reports, Settings, Schedule, Directory objects, ...). It was implemented by a team with a different experience in JEE technologies. And almost all of them started to use Sculptor model very quickly.

Our team used SCRUM as a development process. So, it was really exciting "Agile" process implementation. Even Domain Model was agile! There were several serious refactorings that includes changes in Domain Model and Sculptor framework supported this in several ways. First, It requires tests implementation, so our changes was tested just after applying. Second, the way Sculptor uses to separate generated and hand-written code doesn't prevent refactorings. Sure, there were some minor problems, e.g. with using "lazy" initialization. But they were successfully solved and now product is on the market.

Regards,

Vladimir

December 2008

Sculptor 1.7.0 - GAE, Smartclient, EclipseLink, DataNucleus, JEE

2010-02-02T20:00:00.001+01:00

Sculptor 1.7.0 has been released. In this version the list of supported technologies has grown with several popular alternatives.

Sculptor has many built in customization options to pick and choose from. The major new target implementation options in this release:

Google App Engine
EclipseLink and DataNucleus JPA Provider
Smartclient GWT
Pure EJB3 (without Spring)
Web Services with JAX-WS

Sculptor also has good possibilities for extensibility if you prefer a design or framework that is not supported out-of-the-box. Sculptor is using openArchitectureWare code generation platform, which makes it possible to change the code generation templates using Aspect-Oriented Programming (AOP) features.

The Pure EJB3 implementation, one of the new features in 1.7.0, has already been successfully used in several projects, i.e. it is well tested and production ready. This means that not only Spring users can gain the productivity and quality benefits of Sculptor, but also those who develop on the standard JEE stack.

Enable scala in your appengine/sculptor project

2010-01-28T22:00:00.001+01:00

I've recently started to interest myself in the Scala language. It is a language that runs on the jvm. It's very rich in its syntax and it has a very pleasant mix of OO- and functional programming.

But since I always starts my explorations with the solid foundation of maven the first problem I encountered was how to integrate scala in my build process in a project that already mixes GAE and Sculptor. I.e. I start off from a project that has been created with our maven appengine archetype. So this is a log of my steps for making this happen.

1) In your pom, add a property for the version of Scala:


<scala.version>2.7.7</scala.version>

2) In your pom, add repository for scala:


<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>

3) In your pom , add pluginRepository:


<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>

4) In your pom, add dependency:


<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>

5) In your pom, add entry in maven-dependency-plugin in the build section:


<artifactItem>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
<outputDirectory>war/WEB-INF/lib</outputDirectory>
</artifactItem>

6) In your pom, add dependency to specs-library (if you want it):


<dependency>
<groupId>org.specs</groupId>
<artifactId>specs</artifactId>
<version>1.4.3</version>
<scope>test</scope>
</dependency>

7) In your pom, add the scala-plugin to the build section:


<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
 <id>scala-compile-first</id>
 <phase>process-resources</phase>
 <goals>
   <goal>add-source</goal>
   <goal>compile</goal>
 </goals>
</execution>
<execution>
 <id>scala-test-compile</id>
 <phase>process-test-resources</phase>
 <goals>
   <goal>testCompile</goal>
 </goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<args>
 <arg>-target:jvm-1.5</arg>
</args>
</configuration>
</plugin>

8) In your project file structure, add a simple scala file: src/main/scala/org/foo/App.scala


package org.foo

/**
* Hello world!
*
*/
object App extends Application {
println( "Hello World!" )
}

9) In your project file structure, add a simple scala test: src/test/scala/org/foo/AppTest.scala


package org.foo

import org.junit._
import Assert._

@Test
class AppTest {

 @Test
 def testOK() = assertTrue(true)

}

10) In the root of your project file structure, run mvn install to see that everything builds ok

How to Remove Sculptor

2010-01-26T23:00:00.001+01:00

When taking the decision to try a tool, such as Sculptor, it is important to know what it takes to stop using it if doesn't turn out as expected. Therefore I will in this post explain how to remove Sculptor. You don't have to do all steps and you don't have to do everything at once.

Step 1

The first step is to remove the code generation. This will take you less than 10 minutes.

a) In the maven pom.xml you remove the following:

plugin fornax-oaw-m2-plugin
plugin maven-clean-plugin
plugin fornax-checksum-m2-plugin
dependency fornax-cartridges-sculptor-generator

b) Remove these files in src/main/resources:

model files, .btdesign, .guidesign
sculptor-generator.properties
templates/SpecialCases.xpt
extensions/SpecialCases.ext

c) Move generated files

src/generated/java -> src/main/java
src/generated/resources -> src/main/resources
src/test/generated/java -> src/test/java
src/test/generated/resources -> src/test/resources

d) Run mvn eclipse:eclipse to generate eclipse project without generated source directories

Step 2
You don't need to do more right away, but later you would probably like to refactor the gap classes. Separation of generated and manually written code is done by a generated base class and manually written subclass. E.g. Person and PersonBase. You would probably like to collapse those into one class. To do this you can use the 'Push Down' refactoring in Eclipse, or simply copy paste the code from the base class to the subclass, and remove the extension.

This takes a few minutes per class, so you should be able to do a whole project within a few hours. It is a trivial and safe refactoring.

Step 3

Sculptor is a development tool, but it also has a small runtime library, with classes of utility and general character. This blog post explains why we have a runtime library. You can continue to use those classes as is. If you feel uncomfortable with that you can copy the source and remove classes that you don't use. The runtime library is not advanced at all, and you will have no problems to understan, maintain and change it as you need.

Why we Combine Code Generator with Runtime Library

2010-01-21T20:30:00.003+01:00

Sculptor is a development tool, but it also has a small runtime library. Some people find it scaring to bring in yet another library in their software stack.

It contains classes of utility and general character. It is not advanced at all. We could have generated those also and have no runtime dependency at all. Instead, we try to only generate things that must be variable, such as application specific structure and glue code. Static content should not be generated.

If we would have decided to not use a runtime library I think it would be a high risk that we would have generated code with bad design, such as repeated, duplicated code. We think in terms of DRY for the generated code. We always try to produce code as if we would have written it by hand, without a code generator. There are some exceptions to this rule, for example separation of generated and hand written code in separate classes.

It is interesting to compare Spring Roo and Sculptor. I see a lot of similarities, but also differences. One difference related to this topic is that Roo doesn't have a runtime library (except ordinary Spring Framework). They generate everything. They are using this as a selling point, but I think they have fallen into the trap of generating duplicated code. Take a look at the persistence methods in the ITD for an Entity.

   @Transactional 
 public void Planet.remove() { 
     if (this.entityManager == null) this.entityManager = entityManager();     
     if (this.entityManager.contains(this)) {     
         this.entityManager.remove(this);         
     } else {     
         Planet attached = this.entityManager.find(Planet.class, this.id);         
         this.entityManager.remove(attached);         
     }     
 }

What happens if you decide to remove Roo? Then you will have to maintain a lot of duplicated code, that was generated initially. I think it illustrates how easy it is to forget or not bother about ordinary design principles when you solve everything with code generation. DRY should of course be used even when you have a code generator in your hands.

In Sculptor we have placed the general persistence operations in parameterized access objects, in the runtime library.

Pick 'n' Choose Target Implementation

2010-01-16T15:00:00.001+01:00

This article gives an overview of the options you have regarding the generated target implementation. We add new stuff all the time so this describes what is available at the time of this writing, i.e. in Sculptor 1.7.0.

The default target implementation is using:

JPA with Hibernate as provider
Spring with annotations, no EJBs
Web CRUD GUI client with JSF, Facelets, and Spring Web Flow
In-memory Hsqldb as database
Deployment as war in Jetty

The default choices gives a good starting environment with minimal requirements of installation of external software, such as database and application server.

For each of these areas there are alternative implementations that are supported by Sculptor out of the box. The features are selected with simple configuration.

If you have a different reference architecture than what is provided by Sculptor out of the box you can rather easily customize the generator to fit your needs. It is possible to add your own code generator templates or redefine sections of existing templates. All that is described in the Developer's Guide.

Test Scaffold

2010-01-16T13:39:00.004+01:00

I must admit that when I use Sculptor I often skip completing the JUnit tests for the ordinary CRUD operations. To my defense I would say that they are trivial and completely generated, but anyway I think that is a bad habit that should be changed. Therefore I wrote a small Eclipse template for them. Maybe useful for you also:


    @${testType:newType(org.junit.Test)}
    public void testFindById() throws Exception {
  ${DomainObject:localVar(org.fornax.cartridges.sculptor.framework.domain.AbstractDomainObject)} obj = ${service:field}.findById(getServiceContext(), 1L);
        assertNotNull(obj);
 }

    @${testType:newType(org.junit.Test)}(expected=${DomainObject}NotFoundException.class)
 public void testFindByIdWithNotFoundException() throws Exception {
  ${service}.findById(getServiceContext(), -1L);
 }

    @${testType:newType(org.junit.Test)}
    public void testFindAll() throws Exception {
        ${listType:newType(java.util.List)}<${DomainObject}> found = ${service}.findAll(getServiceContext());
        assertEquals(1, found.size());
    }

    @${testType:newType(org.junit.Test)}
    public void testSave() throws Exception {
        int countBefore = countRowsInTable(${DomainObject}.class);
        ${DomainObject} obj = new ${DomainObject}();
        // TODO complete...
        // obj.set${cursor}
        ${service}.save(getServiceContext(), obj);
        assertEquals(countBefore + 1, countRowsInTable(${DomainObject}.class));
    }

    @${testType:newType(org.junit.Test)}
    public void testDelete() throws Exception {
        int countBefore = countRowsInTable(${DomainObject}.class);
        ${DomainObject} obj = ${service}.findById(getServiceContext(), 1L);
        ${service}.delete(getServiceContext(), obj);
        assertEquals(countBefore - 1, countRowsInTable(${DomainObject}.class));
  }

Add it in Eclipse > Preferences > Java > Editor > Templates
Name: testScaffold
Context: Java type members
Description: JUnit test methods for Sculptor CRUD operations

Internal DSL for Criteria - part 3:3

2009-12-01T21:00:00.001+01:00

In previous 2 articles I have described the basics of the new internal DSL for JPA/Hibernate criteria. Let us dive in to an advanced topic. We would like to be able to specify conditional expressions with logical operators, and with grouping of nested conditions.


  // A and (B or C)
  criteriaFor(Person.class).withProperty(aaa()).eq("A").and().lbrace()
      .withProperty(bbb()).eq("B").or().withProperty(ccc()).eq("C").rbrace()
      .build();

That should build a syntax tree like:

This means that I need to hold state of current and previous not completed operators in the builder. I hold the current operator in a stack and give braces a special meaning. The expression builder looks like this:


public class ConditionalCriteriaBuilder<T> {

   private enum ExpressionOperator {
       Not, Or, And, LBrace
   }

   private ConditionalCriteriaBuilder() {
   }

   public static <T> ConditionRoot<T> criteriaFor(Class<T> clazz) {
       return new ConditionalCriteriaBuilder<T>().new RootBuilderImpl();
   }

   public interface ConditionRoot<T> {
       List<ConditionalCriteria> build();
       CondditionProperty<T> withProperty(Property<T> property);
       ConditionRoot<T> and();
       ConditionRoot<T> or();
       ConditionRoot<T> not();
       ConditionRoot<T> lbrace();
       ConditionRoot<T> rbrace();
   }

   public interface CondditionProperty<T> {
       ConditionRoot<T> eq(Object value);
   }

   public class RootBuilderImpl implements ConditionRoot<T> {
       private final SimpleStack<ConditionalCriteria> criteriaStack = new SimpleStack<ConditionalCriteria>();
       private final SimpleStack<ExpressionOperator> operatorStack = new SimpleStack<ExpressionOperator>();

       /**
        * End the expression with this build
        */
       public List<ConditionalCriteria> build() {
           return criteriaStack.asList();
       }

       public CondditionProperty<T> withProperty(Property<T> property) {
           return new PropBuilderImpl(property.getName());
       }

       public ConditionRoot<T> and() {
           operatorStack.push(ExpressionOperator.And);
           return this;
       }

       public ConditionRoot<T> or() {
           operatorStack.push(ExpressionOperator.Or);
           return this;
       }

       public ConditionRoot<T> not() {
           operatorStack.push(ExpressionOperator.Not);
           return this;
       }

       public ConditionRoot<T> lbrace() {
           operatorStack.push(ExpressionOperator.LBrace);
           return this;
       }

       public ConditionRoot<T> rbrace() {
           operatorStack.pop();
           if (criteriaStack.isEmpty()) {
               return this;
           }
           ConditionalCriteria lastCriteria = popCriteria();
           pushCriteria(lastCriteria);
           return this;
       }

       private ConditionalCriteria popCriteria() {
           return criteriaStack.pop();
       }

       private void pushCriteria(ConditionalCriteria criteria) {
           ExpressionOperator currentOperator = operatorStack.peek();
           if (currentOperator == ExpressionOperator.Or || currentOperator == ExpressionOperator.And) {
               ConditionalCriteria compositeCriteria;
               if (currentOperator == ExpressionOperator.Or) {
                   compositeCriteria = ConditionalCriteria.or(popCriteria(), criteria);
               } else {
                   compositeCriteria = ConditionalCriteria.and(popCriteria(), criteria);
               }
               criteriaStack.push(compositeCriteria);
               operatorStack.pop();
           } else if (currentOperator == ExpressionOperator.Not) {
               ConditionalCriteria notCriteria = ConditionalCriteria.not(criteria);
               criteriaStack.push(notCriteria);
               operatorStack.pop();
               if (!operatorStack.isEmpty() && !criteriaStack.isEmpty()) {
                   pushCriteria(criteriaStack.pop());
               }
           } else if (currentOperator == ExpressionOperator.LBrace) {
               criteriaStack.push(criteria);
           } else {
               criteriaStack.push(criteria);
           }
       }

       private class PropBuilderImpl implements CondditionProperty<T> {
           String propertyName;

           PropBuilderImpl(String name) {
               this.propertyName = name;
           }

           public ConditionRoot<T> eq(Object value) {
               pushCriteria(ConditionalCriteria.equal(propertyName, value));
               return RootBuilderImpl.this;
           }
       }

   }
}

The magic is in pushCriteria, which is invoked at the end of a condition, such as eq(), and also when right brace is reached. It is not a trivial problem, but thanks to the stack metaphor the solution is not that complicated.

A final thought about the syntax. Look at the expression again:


  // A and (B or C)
  criteriaFor(Person.class).withProperty(aaa()).eq("A").and().lbrace()
      .withProperty(bbb()).eq("B").or().withProperty(ccc()).eq("C").rbrace()
      .build();

I use special keywords for left brace and right brace. Why not use grouping with an single method instead and write the nested expression as method parameter?


  // A and (B or C)
  criteriaFor(Person.class).withProperty(aaa()).eq("A").and().group(
      criteriaFor(Person.class).withProperty(bbb()).eq("B").or().withProperty(ccc()).eq("C"))
      .buildSingle();

That would be possible, and probably easier to implement, but I see a few problems from end user perspective:

There are already much clutter of braces due to the methods and it is hard to see and write the end brace of the group operation.
You loose context and have to start the nested expression from scratch, i.e. with static criteriaFor method.

That was all of this exercise. I hope you find it useful as a practical example of how to develop a small Internal DSL. You find the full source code in subversion, including JUnit test.

Internal DSL for Criteria - part 2:3

2009-11-30T22:00:00.001+01:00

In previous article I introduced the new internal DSL for JPA/Hibernate criteria. Let us look at how to handle the property names.


criteria().prop("lastName").eq("Svensson").build();

The problem with the above expression is that the property lastName is a String. Refactoring of Person.lastName will not be automatically detected. Another issue is that you have to remember/lookup that the property name is "lastName" when writing this expression, the IDE will not help you.

We, who are using Sculptor, define the domain object in the design model like this:


Entity Person {
   String firstName
   String lastName
}

From this Sculptor generates the Person class. We can also easily generate meta data of the properties of Person, which makes it possible to write expressions like this:


List<ConditionalCriteria> conditionalCriteria = ConditionalCriteriaBuilder.criteriaFor(Person.class)
    .withProperty(PersonProperties.lastName()).eq("Svensson").build();

PersonProperties is generated and it contains static methods for each property, returning a Property<Person> object, which internally defines the strings. The builder is also parameterized, using the class defined in the criteriaFor factory method. This means that the withProperty method only accepts properties of correct type, i.e. Property<Person>.

Static imports can be used to make the expression more compact and readable:


import static org.fornax.cartridges.sculptor.framework.accessapi.ConditionalCriteriaBuilder.criteriaFor;
import static org.library.person.domain.PersonProperties.lastName;

List<ConditionalCriteria> conditionalCriteria = criteriaFor(Person.class)
  .withProperty(lastName()).eq("Svensson").build();

Note that the eclipse keyboard shortcut for static import is <ctrl+shift+m> (mac: <cmd+shift+m>).

The expression builder looks like this:


public class ConditionalCriteriaBuilder<T> {

    private final List<ConditionalCriteria> criteriaList = new ArrayList<ConditionalCriteria>();

    private ConditionalCriteriaBuilder() {
    }

    public static <T> ConditionalCriteriaBuilder<T> criteriaFor(Class<T> clazz) {
        return new ConditionalCriteriaBuilder<T>();
    }

    public List<ConditionalCriteria> build() {
        return criteriaList;
    }

    private void addCriteria(ConditionalCriteria criteria) {
        criteriaList.add(criteria);
    }

    public CondditionProperty<T> withProperty(Property<T> property) {
        return new PropBuilderImpl(property.getName());
    }

    public interface CondditionProperty<T> {
        ConditionalCriteriaBuilder<T> eq(Object value);
    }

    private class PropBuilderImpl implements CondditionProperty<T> {
        String propertyName;

        PropBuilderImpl(String name) {
            this.propertyName = name;
        }

        public ConditionalCriteriaBuilder<T> eq(Object value) {
            addCriteria(ConditionalCriteria.equal(propertyName, value));
            return ConditionalCriteriaBuilder.this;
        }
    }
}

Alright, let us refactor the model, and introduce a BasicType for the first and last name of the Person:


Entity Person {
  - @PersonName name
}

BasicType PersonName {
    String first
    String last
}

Generate, and you will immediately detect compilation error in the criteria expression. Fix it and it looks like this:


List<ConditionalCriteria> conditionalCriteria = criteriaFor(Person.class)
  .withProperty(name().last()).eq("Svensson").build();

The generated Properties classes defines refererences also, so it easy to navigate associations with full code completion support. Great, our goals of code completion and refactoring are fulfilled.

That is not the end of this series of articles. In next article I will illustrate some more advanced operators that requires some intellectual thoughts. See you tomorrow.

Internal DSL for Criteria - part 1:3

2009-11-29T20:30:00.003+01:00

Hibernate has support for criteria queries and that is an awaited feature in JPA 2.0. The criteria API is rather technical and doesn't read to well with your domain terms, your DDD Ubiquitous Language. Therefore I have developed a small internal DSL in Java to be able to express conditional criteria in human readable format. It is not intended to handle everything in the criteria API.

I use this to illustrate how to implement an internal DSL. It is not hard once you have learned the tools to play with. Java is not optimal for DSLs, but let us use it as good as possible.

The goal was to create a fluent api that supported code completion and refactoring in a good way. A simple criteria may look like:

criteriaFor(Person.class).withProperty(sex()).eq(Gender.FEMALE).build();

and a more advanced criteria:


criteriaFor(Person.class)
    .withProperty(sex()).eq(Gender.FEMALE).and()
    .withProperty(ssn().country()).eq(SWEDEN).and()
    .lbrace().withProperty(name().first()).like("A%")
        .or().withProperty(name().last()).like("A%").rbrace()
    .orderBy(name().last())
    .build();

I will show how a few pieces of the DSL was developed. I have mostly used Expression Builder and Method Chaining, but also a few other tricks.

The expression builder doesn't create the JPA/Hibernate criteria directly. It creates an intermediate structure of objects , which can be translated to criteria API. The main reason for that is decoupling and possibility to have vendor specific implementations.

First, let me state the obvious: When developing an internal DSL it is perfect to do it the TDD way. You will add more and more features and do a lot of refactoring until you are satisfied with the language.

Enough of the introductory talking, let us go hands on to a simple equals expression:

List<ConditionalCriteria> conditionalCriteria =   
    ConditionalCriteriaBuilder.criteria().prop("name").eq("Svensson").build();

This can be implemented in the builder as:


public class ConditionalCriteriaBuilder {
    private final List<ConditionalCriteria> criteriaList = new ArrayList<ConditionalCriteria>();
    private String propertyName;

    private ConditionalCriteriaBuilder() {
    }

    public static ConditionalCriteriaBuilder criteria() {
        return new ConditionalCriteriaBuilder();
    }

    public List<ConditionalCriteria> build() {
        return criteriaList;
    }

    public ConditionalCriteriaBuilder prop(String name) {
        propertyName = name;
        return this;
    }

    public ConditionalCriteriaBuilder eq(Object value) {
        addCriteria(ConditionalCriteria.equal(propertyName, value));
        return this;
    }

    private void addCriteria(ConditionalCriteria criteria) {
        criteriaList.add(criteria);
    }
}

But that is not very good because it allows invalid expressions, such criteria().eq("A").prop("aaa").eq("B").

Therefore I introduce an interface for the property level:


public PropBuilder prop(String name) {
    return new PropBuilderImpl(name);
}

public interface PropBuilder {
    ConditionalCriteriaBuilder eq(Object value);
}

private class PropBuilderImpl implements PropBuilder {
    String propertyName;

    PropBuilderImpl(String name) {
        this.propertyName = name;
    }

    public ConditionalCriteriaBuilder eq(Object value) {
        addCriteria(ConditionalCriteria.equal(propertyName, value));
        return ConditionalCriteriaBuilder.this;
    }
}

This also facilitates code completion (ctrl+space) in a good way, which was one of our goals.

In next post, tomorrow, I will take care of the string property names that are not very refactoring friendly.

How we do automated regression testing with Selenium and Hudson

2009-11-19T13:41:00.000+01:00

When developing a piece of software that has a lifecycle that spans over several years and periodically is released you have to do regression testing, i.e. making sure that previous features doesn't break because of new stuff.

And after a while when your software grows and you add new features, the amount of regression tests increases. To avoid drowning yourself with testing you need to automate as much as possible.

The Sculptor team is a bunch of guys that are driven by interest and are developing the software on there spare time during late nights. We doesn't have the time to do deep manual testing, hence automation is very attractive for us. And since we are geographically distributed and don't have a central CI environment we have to solve some practical problems locally.

What we do (amongst other things), involves having a local Hudson server running on our developing environment (i.e. my iMac) that (of course) builds all projects and runs unit tests.

But we also use Selenium to run automated functional tests to make sure that our example application works. I though I should show some more details about how we use Selenium.

As I said, we have an example application called Library. If you look in the source code for the library-web module you will find the directory:

src/main/webapp/selenium

In there you will find a bunch of tests. The root is the suite file:

___test-suite.xhtml

Selenium test can be written in various programming languages. We have chosen to keep it simple and implement the tests in html.

Having these tests enables us to run them as soon as the code base changes thanks to Hudson and Maven. In Hudson we just creates a new job that is triggered to run as soon as the Library projects is compiled and the unit tests passes. In the pom-file for the Library web project we have a profile that we can use to start a local instance of the Jetty server. Deploy the the application. Run our selenium tests. And finally stop the server. The Maven configuration for this is:

<profile>
<id>regression</id>
<build>
<plugins>
 <plugin>
   <groupId>org.mortbay.jetty</groupId>
   <artifactId>maven-jetty-plugin</artifactId>
   <version>6.1.11</version>
   <configuration>
     <scanIntervalSeconds>10</scanIntervalSeconds>
     <stopKey>foo</stopKey>
     <stopPort>9999</stopPort>
   </configuration>
   <executions>
     <execution>
       <id>start-jetty</id>
       <phase>pre-integration-test</phase>
       <goals>
         <goal>run</goal>
       </goals>
       <configuration>
         <scanIntervalSeconds>0</scanIntervalSeconds>
         <daemon>true</daemon>
       </configuration>
     </execution>
     <execution>
       <id>stop-jetty</id>
       <phase>post-integration-test</phase>
       <goals>
         <goal>stop</goal>
       </goals>
     </execution>
   </executions>
 </plugin>
 <plugin>
   <groupId>org.codehaus.mojo</groupId>
   <artifactId>selenium-maven-plugin</artifactId>
   <version>1.0</version>
   <configuration>
     <suite>src/main/webapp/selenium/___test-suite.xhtml</suite>
     <browser>*firefox</browser>
     <results>${project.build.directory}/target/selenium.html</results>
     <startURL>http://localhost:8080/${artifactId}</startURL>
   </configuration>
   <executions>
     <execution>
       <id>run-tests</id>
       <phase>integration-test</phase>
       <goals>
         <goal>selenese</goal>
       </goals>
     </execution>
   </executions>
 </plugin>
</plugins>
</build>
</profile>

But, since the amount of tests keeps growing and the time it takes to run them all also grows, there is a need to be able to run just a single test or a few of them. For instance, it is very convenient to have that possibility when developing new features, fixing bugs, or just doing a refactoring. To enable this we use the very competent Firefox plugin called SeleniumIDE. Beside recording test case, you can also load already defined test cases (or suites) and run them.

Promote Quality with Sculptor

2009-11-15T21:00:00.001+01:00

We have written an article that has been published in the paper magazine JayView Issue 20.

Without a vision of how to design applications within an organization the development can be compared to lawless Wild West. Development guidelines are often used, but seldom successful over the long haul. We suggest taking the architectural decisions one step further by automating them using a tool such as Sculptor.

When using a general purpose language, such as Java, and its big toolbox of APIs and frameworks there is a huge freedom of choice. This is a double-edged sword. We meet a lot of companies that have realized that they must narrow down the choices so that each new project doesn't invent its own unique system architecture and product suite. The benefits of a homogeneous architecture is obvious when looking at the big picture.

The reference architecture is often accomplished by writing guidelines and maybe a sample reference application. There are several problems with a reference architecture that is only promoted by documentation. We suggest automating some pieces of the development by using a code generator tool, such as Sculptor, to enforce consistency in the architecture.

Read more in the full article.

Mocking with App Engine and Spring

2009-11-03T21:00:00.001+01:00

In previous article I illustrated how easy it is to get started with unit testing with the local App Engine environment. In this article I will go in to more advanced interaction based testing, i.e. mocking.

The App Engine APIs are simulated in the local environment. Some local implementations are designed with testing in mind, such as the email API. It is possible to verify the emails that were sent.


LocalMailService localMailService = AppEngineTestHelper.getLocalMailService();
List<MailMessage> sentMessages = localMailService.getSentMessages();
assertEquals(2, sentMessages.size());

Some other local implementations are not suitable for unit testing, such as the URL fetch service, which executes a real remote request. To solve this you need to encapsulate usage of external communication and make it possible to replace it when unit testing.

Since we are using Spring for dependency injection it is possible to replace any Spring bean for testing purpose. In our customer-supplier sample the InquiryRepository in the customer application sends inquiries to the customer application with a REST post.

This can be replaced when testing by defining a stub implementation that overrides the method that sends then inquiries. This is done in spring xml configuration (more-test.xml):


<bean id="inquiryRepository"
  class="org.customer.inquiry.repositoryimpl.InquiryRepositoryStub"/>


public class InquiryRepositoryStub extends InquiryRepositoryImpl {
    @Override
    protected boolean sendInquiryToSupplier(Inquiry inquiry, Supplier supplier) {
        return true;
    }
}

Next step is to use a mocking framework instead. This makes it possible to verify the interaction, i.e. that the sendInquiryToSupplier method was invoked.

Then it is motivated to extract the sending to a separate class and interface. It is this interface that we want to mock.


public interface InquirySender {
    boolean sendInquiryToSupplier(Inquiry inquiry, Supplier supplier);
}

The real implementation is an ordinary Spring @Component, that is @Autowired in InquiryRepositoryImpl. It is this implementation we want to replace with a mock when testing.


@Component
public class InquirySenderImpl implements InquirySender {

We use the approach described in the first part of Mocking & Spring tests. The FactoryBean is included in Sculptor so we only need to add the xml definition (more-test.xml):


  <bean id="inquirySenderMockFactory"
    class="org.fornax.cartridges.sculptor.framework.test.MockitoFactory"
    primary="true" >
      <property name="type" value="org.customer.inquiry.repositoryimpl.InquirySender"/>
  </bean>

The junit test looks like this:


public class InquiryServiceTest extends AbstractAppEngineJpaTests
  implements InquiryServiceTestBase {

  @Autowired
  private InquiryService inquiryService;
  @Autowired
  private InquirySender inquirySenderMock;

  @Before
  public void initMock() {
      when(inquirySenderMock.sendInquiryToSupplier(any(Inquiry.class), any(Supplier.class)))
          .thenReturn(true);
  }

  @Before
  public void populateDatastore() {
      Inquiry inquiry1 = new Inquiry();
      inquiry1.setMessage("M1");
      inquiry1.setOwnerEmail("foo@gmail.com2");
      getEntityManager().persist(inquiry1);

      Supplier supplier1 = new Supplier("S1");
      supplier1.setUrl("http://localhost:8081/rest/inquiry");
      getEntityManager().persist(supplier1);

      Supplier supplier2 = new Supplier("S2");
      supplier2.setUrl("http://localhost:8081/rest/inquiry");
      getEntityManager().persist(supplier2);
  }

  @Test
  public void testSendInquiry() throws Exception {
      Key key = KeyFactory.createKey(Inquiry.class.getSimpleName(), 1L);
      boolean ok = inquiryService.sendInquiry(getServiceContext(), key);
      assertTrue(ok);
      // there are 2 suppliers
      verify(inquirySenderMock, times(2)).sendInquiryToSupplier(
          any(Inquiry.class), any(Supplier.class));
  }
}

Note that the mock is initialized in the @Before method and then verified last in the test method. In this case two messages should be sent, one for each supplier.

Maybe you have noticed that this approach is not at all specific for App Engine, it can be used for any Spring application. We need to learn a lot of new things when using App Engine, but some old knowledge still applies. :-)

Unit Testing with App Engine and Spring

2009-11-01T17:30:00.000+01:00

Sculptor makes it easy to write JUnit tests for Google App Engine. A test case looks like this:


public class SupplierServiceTest extends AbstractAppEngineJpaTests {

    @Autowired
    private SupplierService supplierService;

    @Before
    public void populateDatastore() {
        Supplier supplier1 = new Supplier("S1");
        getEntityManager().persist(supplier1);

        Supplier supplier2 = new Supplier("S2");
        getEntityManager().persist(supplier2);
    }

    @Test
    public void testFindAll() throws Exception {
        List<Supplier> all = supplierService.findAll(getServiceContext());
        assertEquals(2, all.size());
    }

    @Test
    public void testFindByName() throws Exception {
        Supplier found = supplierService.findByName(getServiceContext(), "S2");
        assertNotNull(found);
        assertEquals("S2", found.getName());
    }
}

Very natural!

It is interesting to take a look at the base class. It defines a few annotations and extends AbstractJUnit4SpringContextTests to initialize the Spring environment. This enables usage of ordinary @Autowire dependency injection directly in the test class.


@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations =  {"classpath:applicationContext-test.xml"})
public abstract class AbstractAppEngineJpaTests
    extends AbstractJUnit4SpringContextTests {

The embedded App Engine environment is initialized from a method annotated with @Before, i.e. invoked before each test method.


public static void setUpAppEngine(ApiProxy.Environment testEnvironment) {
    ApiProxy.setEnvironmentForCurrentThread(testEnvironment);

    ApiProxy.setDelegate(new ApiProxyLocalImpl(new File(".")) {
    });

    ApiProxyLocalImpl proxy = (ApiProxyLocalImpl) ApiProxy.getDelegate();
    proxy.setProperty(LocalDatastoreService.NO_STORAGE_PROPERTY, Boolean.TRUE.toString());
    clearSentEmailMessages();
}

public static void tearDownAppEngine() {
    ApiProxyLocalImpl proxy = (ApiProxyLocalImpl) ApiProxy.getDelegate();
    LocalDatastoreService datastoreService = (LocalDatastoreService) proxy.getService("datastore_v3");
    datastoreService.clearProfiles();
    clearSentEmailMessages();
}

It is initialized with in memory data store, i.e. it is empty before each test method. You may populate it with initial data in your subclass in a @Before method, see populateDataStore in the sample above.

I learned one thing when doing junit testing in the app engine environment. When working with ordinary databases I have found the Spring transactional test support useful, i.e. Spring executes each test method in a transaction, which is rolled back after the test mehtod. That is achieved with the following annotations and usage of the annotation @BeforeTransaction instead of the ordinary @Before.


@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations =  {"classpath:applicationContext-test.xml"})
@TestExecutionListeners(TransactionalTestExecutionListener.class)
@TransactionConfiguration(transactionManager = "txManager", defaultRollback = true)
@Transactional
public abstract class AbstractAppEngineJpaTests
    extends AbstractJUnit4SpringContextTests {

That was my initial approach also with app engine, but I realized that it was not good. Look at the following test. It will fail on the last assert when using the above transactional support.


   @Test
   public void testSave() throws Exception {
       int countBefore = countRowsInTable(Supplier.class);
       Supplier supplier3 = new Supplier("S3");
       supplierService.save(getServiceContext(), supplier3);
       int countAfter = countRowsInTable(Supplier.class);
       assertEquals(countBefore + 1, countAfter);
   }

The reason is that queries see a snapshot of the datastore as of the beginning of the transaction.

Data isolation between test methods is no problem, since the datastore is initialized (empty) before each test method.

That's all! Try it yourself by running the Maven Archetype for App Engine and fill in the details in the generated PlanetServiceTest.

mvn archetype:generate -DarchetypeGroupId=org.fornax.cartridges -DarchetypeArtifactId=fornax-cartridges-sculptor-archetype-appengine -DarchetypeVersion=1.7.0-SNAPSHOT -DarchetypeRepository=http://www.fornax-platform.org/archiva/repository/snapshots/
mvn clean eclipse:eclipse

Stay tuned, in next post I will describe how to mock.

Decouple modules with asynchronous event dispatching using Spring and task queues in GAE

2009-10-24T18:54:00.002+02:00

To build applications that are maintainable and robust you should strive for decoupling between modules. To build applications that scale you will always benefit from asynchronism and parallellism.
Here we will look how to accomplish the above in GoogleAppEngine and with some help from springframework.
Lets say we have an application where users can register them self. When they do, the application creates a persistence instance of a User-object. But, we will also keep track of how many users we have registered on the site. Now, being in GAE with BigTable luring in the back, doing queries and calculations (as we are used to with a traditional database) isn't a good idea. So as an alternative we choose to have a separate Counter-object that we updates when ever a new user registers. Ok, nothing strange here. But, there are a couple of flaws here:

The User module needs to know about the Counter module.
The User module has to wait for the Counter module to finish when updating the counting.

Ok, lets solve the first by using spring's mechanism for ApplicationEvent's. First, let us put some aop magic to work to intercept the call to UserService.createUser and when it returns (and we have the transaction boundaries on service methods, so no exception, all went well) fire off an event. Spring config for the aop stuff:


<bean id="userListener" class="org.fornax.sculptor.UserListener"/>
<bean id="userAdvice" class="org.fornax.sculptor.UserAdvice"/>
<aop:config>
<aop:pointcut id="userCreationPointcut" expression="execution(public * org..UserService.createUser(..))"/>
<aop:advisor pointcut-ref="userCreationPointcut" ref="userCreationPointcut"/>
</aop:config>

Next, here is the advice:


public class UserAdvice implements MethodInterceptor, ApplicationContextAware {

private ApplicationContext ctx;

public Object invoke(MethodInvocation invocation) throws Throwable {
  User user = (User) invocation.proceed();
  fireNewUserEvent(user);
  return user;
}

private void fireNewUserEvent(User user) {
  ctx.publishEvent(new UserCreatedEvent(user));
}

public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
  this.ctx = applicationContext;
}
}

The listener that is being notified:


public class UserListener implements ApplicationListener<UserCreatedEvent> {
@Autowired private CounterService counterService;
public void onApplicationEvent(UserCreatedEvent event) {
  counterService.increment();
}
}

And the event being passed:


public class UserCreatedEvent extends ApplicationEvent {
public UserCreatedEvent(User user) {
  super(user);
}
}

Ok, so now we are half way. We have the Observer pattern in place. But we still does everything synchronous.
Enter GAE's task queue's. Let us modify our UserListener:


public class UserListener implements ApplicationListener<UserCreatedEvent> {

public void onApplicationEvent(UserCreatedEvent event) {
  TaskOptions task = url("/rest/admin/counter/user").method(POST);
  Queue queue = QueueFactory.getDefaultQueue();
  queue.add(task);
}
}

And by the wonders of task queue's, we now put a task on the queue and by that we do the counting job asynchronous. And of course, we dropped the reference to the CounterService. But we miss one piece here, right? What does the url in the task point at. Well, nothing strange here, it is just a spring mvc controller:


@Controller
public class CounterCountroller {
@Autowired private CounterService counterService;
@RequestMapping(value = "/admin/counter/user", method = RequestMethod.POST)
public void incrementCounter() throws IOException {
  try {
    counterService.increment();
  } catch (Exception ignore) {
    // doesn't matter if we get an exception here, just log it
    log.error("Failed to increment counter!", ignore);
  }
}
}

And now we have a more loosely coupled system that scales better. And with a little effort, the code can be generalized so more features are easy to add with the same pattern.
Of course, the downside of this kind of design is that error handling gets more complicated and you can't always trust it to be 'right'. But that is system design, you have to decide what's best for each situation.

Even Weird Naming Conventions are Good

2009-10-21T22:00:00.001+02:00

The good thing with naming conventions are that they are toolable. The DBAs at my department have strong opinions about database naming. They have good reasons for that, even though I don't fully understand all of them :-)

Table names should be prefixed with application/component identifier.
Primary key id column should be prefixed with table name (without application prefix) and followed by _GID.
Underscore to separate words.
Foreign key column is concatenation of role name and primary key column name of target table, except when role and table have the same name.

Does this mean that we have to specify each and every name twice, once for Java and once for the database. Argh... NO, we are using Sculptor. With a straightforward customization I implemented these conventions in the generator and we could continue with natural (java point of view) naming and please the preferences of the DBAs without additional effort.

Naming conventions are important for software quality. Supporting the conventions with a tool is the best way to make sure that they are applied in a consistent way.

Sculptor and Agile

2009-10-19T14:22:00.000+02:00

For us, the agile way has always been the way of getting things done. Even before there where fancy names for it we did things in a way that enabled us to deliver the right things on the right time. This hasn't change, but now we call it scrum :-)

Sculptor will help you if you want to work agile.

Things change

They do. There is nothing anybody can do about it. And when things change, there will be delays. Doing things in an agile manner helps minimize those delays. Using sculptor helps you even more.
I'll show you by an example. Lets say you have an application that is almost done for production. Testing is done. No more bugs (yeah, right). On the final demo, suddenly, one of the stake holders realize that the customer object needs another attribute, lets call it ICE (In Case of Emergency). And, of course according to the stake holder you can't go into production without it.
So, what are the changes that needs to be done?
From the bottom up:

Database schema
Persistence layer (JPA, Hibernate, etc) :
Domain object
Service layer (if we have any logic tied to the attribute)
Back office Client (for creating a customer object)
Public Client (for viewing a customer object)

Most of these changes are boiler plate code. Changing declarations etc. Without Sculptor these changes takes time and are error prone. If we assume that every change is a risk and then calculate and compare all changes we need to do with and without Sculptor we end up with the following table where every point is a code change needed to do to accomplish the last minute requirement:

	Without Sculptor	With Sculptor
model:	0	1
db script:	1	0
db migration script:	1	1
property in domain object:	1	0
jpa annotation för property:	1	0
logic in service or domain object:	1	1
back office client (create/update/list/view):	7	0
public client (view):	1	1
junit test (save, find):	2	2
documentation (class diagrams):	1	0
sum:	16	6

So, being simple, lets say that each point is equally valued. By this assumption we reduce risk and time by 150%

This can be very valuable when change comes along. And it does, doesn't it...?

Sculptor in the Cloud

2009-10-04T22:00:00.003+02:00

Now you can use Sculptor to speed up and simplify development of applications running in the Google App Engine cloud.

Let's start with a demo of how easy it is to create a new application and deploy it.

For this we are using Sculptor maven archetype for App Engine. Try it yourself:

mvn archetype:generate -DarchetypeGroupId=org.fornax.cartridges -DarchetypeArtifactId=fornax-cartridges-sculptor-archetype-appengine -DarchetypeVersion=1.7.0-SNAPSHOT -DarchetypeRepository=http://www.fornax-platform.org/archiva/repository/snapshots/

cd to the new directory

mvn clean

mvn generate-sources

mvn eclipse:eclipse

Import the project in Eclipse

Without any changes the new project is ready to run in the local development server or to be deployed at appspot.com. The sample in the demo is available here: http://sculptor-helloworld.appspot.com

The archetype creates a sample of of a RESTful Spring 3.0 Controller and JSP pages for the CRUD operations.

The archetype also creates a simple sample model, from which Sculptor generates Entity, Repository and Service with the default CRUD operations; findById, findAll, save, and delete.

The model is defined in a textual DSL, with an intuitive syntax, from which Sculptor generates high quality Java code and configuration. It is not a one time shot. The application can be developed incrementally with an efficient round trip loop. The generator is part of the build process (maven).

Sculptor generates JPA mapping annotations for the domain objects defined in the design model. Relations are very limited in App Engine, since the datastore (BigTable) is not a relational database.

Owned and embedded associations are supported and mapped as ordinary JPA associations. They are specified with aggregate and BasicType in the Sculptor model.

Unowned associations are handled with id references and you must lookup the objects with findById when needed.

Services and Repositories are implemented as Spring components with @Autowired dependency injection. Spring AOP is used for error handling and transaction management.

Behavior is implemented with hand written code in subclass, separated from re-generated code in base class. In the above example the sayHello method is typically implemented in the Service by first using the generated findByKey method in the Repository. Note that the name attribute of the Planet is marked as key.

Sculptor also provides support for JUnit testing with the local App Engine environment. I will cover that in another article some day soon.

Maven Archetype for App Engine

2009-10-01T21:00:00.005+02:00

I have developed a maven archetype for Google App Engine projects. The generated project supports:

All dependency jar files are downloaded from maven repositories and copied to lib directory as required by App Engine Eclipse plugin, and local development server.

Eclipse project is created with mvn eclipse:eclipse. The resulting Eclipse project has the necessary settings for App Engine Eclipse plugin.

Entity classes are processed by DataNucleus enhancer in the build lifecycle.

JUnit tests with local App Engine environment can be run from maven.

Setting up all of this is not trivial and therefore I would like to share the solution and I hope you find it useful.

Eclipse Project
The maven eclipse plugin need a lot of configuration.

<build>
<outputDirectory>war/WEB-INF/classes</outputDirectory>
<plugins>
    <plugin>
        <artifactId>maven-eclipse-plugin</artifactId>
        <version>2.5.1</version>
        <configuration>
            <!--
                buildOutputDirectory doesn't work due to
                http://jira.codehaus.org/browse/MECLIPSE-422 An workaround is the
                outputDirectory at project/build level
                <buildOutputDirectory>war/WEB-INF/classes</buildOutputDirectory>
            -->
            <testOutputDirectory>target/test-classes</testOutputDirectory>
            <classpathContainers>
                <classpathContainer>com.google.appengine.eclipse.core.GAE_CONTAINER</classpathContainer>
            </classpathContainers>
            <buildcommands>
                <buildcommand>org.eclipse.jdt.core.javabuilder</buildcommand>
                <buildcommand>com.google.gdt.eclipse.core.webAppProjectValidator</buildcommand>
                <buildcommand>com.google.appengine.eclipse.core.enhancerbuilder</buildcommand>
                <buildcommand>com.google.appengine.eclipse.core.projectValidator</buildcommand>
            </buildcommands>
            <additionalProjectnatures>
                <projectnature>org.eclipse.jdt.core.javanature</projectnature>
                <projectnature>com.google.appengine.eclipse.core.gaeNature</projectnature>
                <projectnature>com.google.gdt.eclipse.core.webAppNature</projectnature>
            </additionalProjectnatures>
            <excludes>
                <!-- Included in GAE_CONTAINER -->
                <exclude>com.google.appengine:appengine-api-1.0-sdk</exclude>
                <exclude>com.google.appengine:appengine-api-1.0-labs</exclude>
                <exclude>com.google.appengine.orm:datanucleus-appengine</exclude>
                <exclude>org.datanucleus:datanucleus-jpa</exclude>
                <exclude>org.datanucleus:datanucleus-core</exclude>
                <exclude>org.apache.geronimo.specs:geronimo-jpa_3.0_spec</exclude>
                <exclude>org.apache.geronimo.specs:geronimo-jta_1.1_spec</exclude>
                <exclude>javax.jdo:jdo2-api</exclude>
            </excludes>
        </configuration>
    </plugin>

Some dependencies must be excluded, since they are part of GAE_CONTAINER, otherwise JUnit tests will not work when running inside Eclipse. The output directory is changed to war/WEB-INF/classes. There is a bug (MECLIPSE-422) which cause the test classes to not be separated if buildOutputDirectory is used. The local development server doesn't like the test classes. The trick is to define the output at the top build level and define testOutputDirectory.

Copy Dependencies
When running the local development server and deploying to App Engine all dependent jar files must be located in war/WEB-INF/lib. I have used the maven dependency plugin to copy the jar files during the maven clean phase.

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
    <execution>
        <id>copy</id>
        <phase>clean</phase>
        <goals>
            <goal>copy</goal>
        </goals>
        <configuration>
            <artifactItems>
                <artifactItem>
                    <groupId>org.springframework</groupId>
                    <artifactId>spring-core</artifactId>
                    <version>${spring.version}</version>
                    <outputDirectory>war/WEB-INF/lib</outputDirectory>
                </artifactItem>
                <!-- more ... -->
                <artifactItem>
                    <groupId>com.google.appengine</groupId>
                    <artifactId>appengine-api-1.0-sdk</artifactId>
                    <version>${appengine.version}</version>
                    <outputDirectory>war/WEB-INF/lib</outputDirectory>
                </artifactItem>
                <artifactItem>
                    <groupId>com.google.appengine</groupId>
                    <artifactId>appengine-api-1.0-labs</artifactId>
                    <version>${appengine.version}</version>
                    <outputDirectory>war/WEB-INF/lib</outputDirectory>
                </artifactItem>
                <artifactItem>
                    <groupId>com.google.appengine.orm</groupId>
                    <artifactId>datanucleus-appengine</artifactId>
                    <version>1.0.3</version>
                    <outputDirectory>war/WEB-INF/lib</outputDirectory>
                </artifactItem>
                <artifactItem>
                    <groupId>org.datanucleus</groupId>
                    <artifactId>datanucleus-jpa</artifactId>
                    <version>1.1.5</version>
                    <outputDirectory>war/WEB-INF/lib</outputDirectory>
                </artifactItem>
                <artifactItem>
                    <groupId>org.datanucleus</groupId>
                    <artifactId>datanucleus-core</artifactId>
                    <version>1.1.5</version>
                    <outputDirectory>war/WEB-INF/lib</outputDirectory>
                </artifactItem>
                <artifactItem>
                    <groupId>org.apache.geronimo.specs</groupId>
                    <artifactId>geronimo-jpa_3.0_spec</artifactId>
                    <version>1.1.1</version>
                    <outputDirectory>war/WEB-INF/lib</outputDirectory>
                </artifactItem>
                <artifactItem>
                    <groupId>org.apache.geronimo.specs</groupId>
                    <artifactId>geronimo-jta_1.1_spec</artifactId>
                    <version>1.1.1</version>
                    <outputDirectory>war/WEB-INF/lib</outputDirectory>
                </artifactItem>
                <artifactItem>
                    <groupId>javax.jdo</groupId>
                    <artifactId>jdo2-api</artifactId>
                    <version>2.3-eb</version>
                    <outputDirectory>war/WEB-INF/lib</outputDirectory>
                </artifactItem>
            </artifactItems>
            <!-- other configurations here -->
        </configuration>
    </execution>
</executions>
</plugin>

DataNucleus Enhancer
Running the JUnit tests from maven was a primary goal as I would like to run tests from continous build server. Th JUnit tests are using local App Engine environment with in-memory datastore. Therefore the classes must be processed by DataNucleus enhancer after ordinary compilation.

<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>maven-datanucleus-plugin</artifactId>
<version>1.1.4</version>
<configuration>
   <api>JPA</api>
   <mappingIncludes>**/*.class</mappingIncludes>
   <log4jConfiguration>${basedir}/src/main/resources/log4j.properties</log4jConfiguration>
   <verbose>false</verbose>
</configuration>
<executions>
   <execution>
       <phase>process-classes</phase>
       <goals>
           <goal>enhance</goal>
       </goals>
   </execution>
</executions>
</plugin>

Archetype
All of this is packaged in a maven archetype. Try it like this.

mvn archetype:generate -DarchetypeGroupId=org.fornax.cartridges -DarchetypeArtifactId=fornax-cartridges-sculptor-archetype-appengine -DarchetypeVersion=1.7.0-SNAPSHOT -DarchetypeRepository=http://www.fornax-platform.org/archiva/repository/snapshots/

cd to the new directory

mvn clean

mvn eclipse:eclipse

Import the project in Eclipse

As an extra bonus your new project is configured for Spring 3.0 with a sample of a RESTful controller.

Sculptor code generator tool is of course also configured and ready to be used in the new project. I will soon write another article about Sculptor's support for App Engine.

Customer Specific Addon: Deep Merge

2009-09-16T21:00:00.001+02:00

This article illustrates the possibility to add your own features to the Sculptor code generator.

In my customer project we have a need to merge two object graphs. We have a persistent domain model and we receive messages from production systems when changes occur. There are several production systems sending the data in slightly different format and semantics.

We designed this as a first step that converts the production messages to new transient domain object instances.

Next step is to merge that object graph with present persistent objects.

This feels like a tedious and repetitive programming task. If done manually it will require some maintenance when we do changes.

At first I took a look at Dozer, but pretty soon things got complicated and required a lot of XML mapping files. So we gave up that idea.

At home, it struck me that we already have the tool we need. We are already using Sculptor, and it should be a simple addition to generate the merge methods in the domain objects.

Next morning I implemented it like this...

The final java code to be generated looks like this in each domain object. It copies attributes and new associated objects. It traverses existing associations.

  public void deepMerge(Item other) {
    Set<Object> processed = new HashSet<Object>();
    deepMerge(other, processed);
}

public void deepMerge(Item other, Set<Object> processed) {
    if (processed.contains(this)) {
        return;
    }
    processed.add(this);

    if (other.getEstimatedTimeOfArrival() != null) {
        setEstimatedTimeOfArrival(other.getEstimatedTimeOfArrival());
    }

    deepMergeShipment(other, processed);

    deepMergeEvents(other, processed);

}

public void deepMergeShipment(Item other, Set<Object> processed) {
    Shipment currentValue = getShipment();
    if (other.getShipment() != null) {
        if (currentValue == null) {
            setShipment(other.getShipment());
        } else {
            currentValue.deepMerge(other.getShipment(), processed);
        }
    }
}

public void deepMergeEvents(Item other, Set<Object> processed) {
    for (TrackingEvent each : other.getEvents()) {
        if (getEvents().contains(each)) {
            TrackingEvent currentValue = eventForKey(other.getKey());
            currentValue.deepMerge(each, processed);
        } else {
            addEvent(each);
        }
    }
}

protected TrackingEvent eventForKey(Object key) {
    for (TrackingEvent each : getEvents()) {
        if (each.getKey().equals(key)) {
            return each;
        }
    }
    return null;
}

I developed this as a project specific addon, i.e. I invoked a code generation template from SpecialCases.xpt:

«AROUND templates::DomainObject::keyGetter FOR DomainObject»
«targetDef.proceed()»

«EXPAND templates::DeepMerge::deepMerge»
«ENDAROUND»

I started with the simple attributes.

«DEFINE deepMerge FOR DomainObject»

«EXPAND deepMergeMethod»

«ENDDEFINE»

«DEFINE deepMergeMethod FOR DomainObject»
public void deepMerge(«getDomainPackage()».«name» other) {
«EXPAND deepMergeAttribute FOREACH attributes.reject(e | !e.changeable)»
«ENDDEFINE»

«DEFINE deepMergeAttribute FOR Attribute»
if (other.«getGetAccessor()»() != null) {
set«name.toFirstUpper()»(other.«getGetAccessor()»());
}
«ENDDEFINE»

I generated and looked at the result.

I noticed that the auditable fields were included. Ok, then I can use the helper function isSystemAttribute() to skip those.

«EXPAND deepMergeAttribute FOREACH attributes
.reject(e | !e.changeable || e.isSystemAttribute())»

The tricky part is the associations and I could imagine that we would have some corner cases that wouldn't be covered by the generated pattern. Therefore I created separate methods for each association so that it will be possible to override the generated methods in gap classes and handle eventual special cases manually.

I added the templates for references. Starting with the to-one references:

«DEFINE deepMergeOneReference FOR Reference»
  public void deepMerge«name.toFirstUpper()»(«from.getDomainPackage()».«from.name» other) {
      «to.getDomainPackage()».«to.name» currentValue = get«name.toFirstUpper()»();
      if (other.get«name.toFirstUpper()»() != null) {
          if (currentValue == null) {
              set«name.toFirstUpper()»(other.get«name.toFirstUpper()»());
          } else {
              currentValue.deepMerge(other.get«name.toFirstUpper()»());
          }
      }
  }
«ENDDEFINE»

Continuing with the to-many case. It is a little bit more tricky, since we need to grab existing instance for collection. Added a helper method for that.

«DEFINE deepMergeManyReference FOR Reference»
  public void deepMerge«name.toFirstUpper()»(«from.getDomainPackage()».«from.name» other) {
      for («getTypeName()» each : other.get«name.toFirstUpper()»()) {
          if (get«name.toFirstUpper()»().contains(each)) {
              «to.getDomainPackage()».«to.name» currentValue = «name.singular()»ForKey(other.getKey());
              currentValue.deepMerge(each);
          } else {
              add«name.toFirstUpper().singular()»(each);
          }
      }
  }

  protected «to.getDomainPackage()».«to.name» «name.singular()»ForKey(Object key) {
      for («to.getDomainPackage()».«to.name» each : get«name.toFirstUpper()»()) {
          if (each.getKey().equals(key)) {
              return each;
          }
      }
      return null;
  }
«ENDDEFINE»

I was testing this using the Library sample in Sculptor. I noticed problem with extended objects, such as Book, Movie that extends Media. The getKey method is not defined in Media. However, I just ignore this for now, since we don't have that kind of association in our model, and the intention is not to develop a general purpose solution.

Not completely done yet. We have the classical case with circular references. To avoid infinite recursion I added a collection that was passed as parameter to keep track of which objects that have been processed.

All this took me 2 hours to implement, probably much less than implementing it manually in all domain objects. The big benefit is that it is much less risk of manual faults and requires zero maintenance when making changes to the domain objects.

The final template file below, in case you are interested in implementing something similar:

«IMPORT sculptormetamodel»
«EXTENSION extensions::helper»
«EXTENSION extensions::dbhelper»
«EXTENSION extensions::properties»


«DEFINE deepMerge FOR DomainObject»
    «IF !isImmutable()»
        «EXPAND deepMergeMethod»

        «EXPAND deepMergeOneReference FOREACH references.select(r | !r.many).reject(e | !e.changeable)»
        «EXPAND deepMergeManyReference FOREACH references.select(r | r.many)»
    «ENDIF»
«ENDDEFINE»

«DEFINE deepMergeMethod FOR DomainObject»
    public void deepMerge(«getDomainPackage()».«name» other) {
        java.util.Set<Object> processed = new java.util.HashSet<Object>();
        deepMerge(other, processed);
    }

    public void deepMerge(«getDomainPackage()».«name» other, java.util.Set<Object> processed) {
        if (processed.contains(this)) {
            return;
        }
        processed.add(this);
        
        «EXPAND deepMergeAttribute FOREACH attributes.reject(e | !e.changeable || e.isSystemAttribute())»
        
        «FOREACH references.reject(e | !e.changeable) AS ref»
        deepMerge«ref.name.toFirstUpper()»(other, processed);
        «ENDFOREACH»
    }
«ENDDEFINE»



«DEFINE deepMergeAttribute FOR Attribute»
    «IF isPrimitive() -»
    set«name.toFirstUpper()»(other.«getGetAccessor()»());
    «ELSE-»
    if (other.«getGetAccessor()»() != null) {
        set«name.toFirstUpper()»(other.«getGetAccessor()»());
    }
    «ENDIF-»
«ENDDEFINE»


«DEFINE deepMergeOneReference FOR Reference»
    public void deepMerge«name.toFirstUpper()»(«from.getDomainPackage()».«from.name» other, java.util.Set<Object> processed) {
        if (other.get«name.toFirstUpper()»() != null) {
            «IF to.isImmutable()»
                if (!other.get«name.toFirstUpper()»().equals(get«name.toFirstUpper()»())) {
                    set«name.toFirstUpper()»(other.get«name.toFirstUpper()»());
                }
            «ELSE»
            «to.getDomainPackage()».«to.name» currentValue = get«name.toFirstUpper()»();
            if (currentValue == null) {
                set«name.toFirstUpper()»(other.get«name.toFirstUpper()»());
            } else {
                currentValue.deepMerge(other.get«name.toFirstUpper()»(), processed);
            }
            «ENDIF»
        }
    }
«ENDDEFINE»

«DEFINE deepMergeManyReference FOR Reference»
    public void deepMerge«name.toFirstUpper()»(«from.getDomainPackage()».«from.name» other, java.util.Set<Object> processed) {
        for («getTypeName()» each : other.get«name.toFirstUpper()»()) {
            if (get«name.toFirstUpper()»().contains(each)) {
            «IF to.isImmutable()»
                // replace
                remove«name.toFirstUpper().singular()»(each);
                add«name.toFirstUpper().singular()»(each);
            «ELSE»
                «to.getDomainPackage()».«to.name» currentValue = «name.singular()»ForKey(each.getKey());
                currentValue.deepMerge(each, processed);
            «ENDIF»
            } else {
                add«name.toFirstUpper().singular()»(each);
            }
        }
    }
    
    protected «to.getDomainPackage()».«to.name» «name.singular()»ForKey(Object key) {
        for («to.getDomainPackage()».«to.name» each : get«name.toFirstUpper()»()) {
            if (each.getKey().equals(key)) {
                return each;
            }
        }
        return null;
    }
«ENDDEFINE»

Screencast: Introduction to Sculptor

2009-08-27T07:00:00.002+02:00

During the summer I have published a series of articles that illustrate basic usage of Sculptor. They include screencasts so that you get a feeling of what it looks like when using Sculptor.

If you are totally new to Sculptor you might need to read What is Sculptor? before looking at the practical example.

The series illustrates the following, step-by-step:

Jump Start - Initial creation of maven and eclipse projects. Persistent entity and CRUD GUI are created in a few minutes.

The World is Changing - Adding more to the application. Quick development round trip, short feedback loop, it is not a one time shot.

Testing is Simple - Testability is crucial and is of course supported.

Adding Behaviour - Generated code is well separated from hand written code.

Say Hello - Entity, Repository and Service are some of the available building blocks. Yes, it is real DDD-style.

Introducing a Type - Developing a high quality domain model is the core of Sculptor. Small type objects are typically part of a good domain model.

Refactoring - How is refactoring done when having a mix of hand written and generated code?

GAE Transactions

2009-08-25T08:45:00.000+02:00

I'm trying to understand what we should do to make Sculptor compatible with Google App Engine (GAE).

I feel a bit sad when looking back to what I have just experienced, but I guess I should be happy, since I have learned a lot. In this post I will share my mistakes and insights to GAE transactions and Entity Groups.

Together with Andreas I'm developing a little sample that consists of 3 interacting applications. Customer, Supplier and Profile apps. User stories for the initial sprint:

As a customer I want to specify a request for consultants so that I can allocate resources to my project.

As a salesman (supplier) I want to be notified when a customer enters a request for consultants so that I quickly can create an offer to that request.

As a salesman I want to offer consultants to a customer so that I can sell our services.

As a customer I want to see up to date information in the profiles so that I know that it is not obsolete.

I was developing the form enter of the inquiry in the customer app. I saved the form data in an Inquiry object and sent the request to the supplier app using RestTemplate. No problems so far.

We are using the new REST features in Spring 3.0 and have done some adjustments to Sculptor to make it generate JPA code that is compliant with GAE datastore.

Since one inquiry should be sent to many suppliers it didn't feel very scalable to send them all in the form entry request. Therefore I separated the sending to a separate job, which would be invoked by the cron service (later, better with task queue). This is not only more scalable, it is also more fault tolerant, since supplier apps may not be available all the time. By separating it we can easily retry later.

I created a Supplier entity also. In the sendToSuppliers job I got the first problem:

IllegalArgumentException: can't operate on multiple entity groups in a single transaction

Since I had two entities, Inquiry and Supplier and I was using both in the transaction I assumed that it was not allowed to query the Suppliers and update the Inquiries in the same transaction. I based that on the GAE documentation:
All datastore operations in a transaction must operate on entities in the same entity group. This includes querying for entities by ancestor, retrieving entities by key, updating entities, and deleting entities.

That assumption was a fatal mistake that got me on the wrong track. I started to separate the the retrieval of Suppliers and update of Inquiries in separate transactions.

I learned from the documentation that it was possible to disable transactions, but that it was a temporary workaround.

After removing all code except the update of the Inquiries I realized that the Inquiry instances themselves belonged to separate entity groups. I was looping over all Inquiries that had not been sent to suppliers, i.e. I was updating several instances. Of course, they belong to separate entity groups, otherwise it would not scale when the number of objects increase.

Then I redesigned the sending job so that it would only send and update one Inquiry instance. The job will have to be run many times to send all Inquiries.

On the way I learned some more things about GAE datastore:
* A transaction is necessary for some operations, such flush, otherwise; "This operation requires a transaction yet it is not active"
* Queries also require a transaction, otherwise when iterating over the result;
"Object Manager has been closed"
* Modification several times; "can't update the same entity twice in a transaction or operation"

In the end I think the defaults for transactions in Sculptor are alright. Normally we define transaction boundary at the service layer. This is ok for many cases when using GAE also, but one have to design the operations so that they only update one instance (entity group).

There is probably a need for more fine grained transaction control at the repository level. E.g. starting a new transaction for some repository operations. I think we should implement this with @Transactional annotations. Is it possible to mix txAdvice (defaults) with @Transactional (deviations from default)?

Refactoring

2009-08-25T07:30:00.000+02:00

Sometimes I get the question "How is refactoring done when having a mix of hand written and generated code?" It is a good question, since refactoring is very important. The intention is that existing IDE refactoring tools will continue to serve you when using Sculptor.

When doing initial prototyping and you don't have any (or little) hand written code you can easily change in the model and re-generate. When you have hand written code you start with using the refactoring tools in the IDE, as you are used to. Thereafter you do corresponding change in the model and re-generate.

Eventual mistakes will normally be caught by the compiler and JUnit tests.

The following screencast illustrates how to rename Planet to Planet2.

Alternative video format (mpg)

Sculptor doesn't make it more difficult to do refactoring. Sometimes it makes refactoring easier, when the change only affects generated code.

Introducing a Type

2009-08-20T07:45:00.000+02:00

An important building block when creating a high quality domain model is to create small type objects. In this article we will create a Length type for the diameter of the Planet of the helloworld application.

Alternative video format (mpg)

Length is a typical Quantity with a value and unit, e.g. meter, kilometer.

In the design model it looks like this:

BasicType Length {
 BigDecimal value min="0"
 -@LengthUnit unit
}

enum LengthUnit {
 cm, m, km
}

Entity Planet {
 gap
 scaffold
 String name key
 Long population min="0"
 -@Length diameter nullable
 -Set<@Moon> moons opposite planet

 Repository PlanetRepository {
     findByKey;
 }
}

We also need to convert between different units. The behaviour expressed as a JUnit test:

public class LengthTest {

  @Test
  public void shouldConvertFromMeterToKilometer() {
      Length length = new Length(new BigDecimal("31000"), m);
      Length lengthInKilometer = length.to(km);
      assertEquals(new Length(new BigDecimal("31"), km),
          lengthInKilometer);
  }

  @Test
  public void shouldConvertFromKilometerToMeter() {
      Length length = new Length(new BigDecimal("44"), km);
      Length lengthInMeter = length.to(m);
      assertEquals(new Length(new BigDecimal("44000"), m),
          lengthInMeter);
  }

  @Test
  public void shouldNotConvertSameUnit() {
      Length length = new Length(new BigDecimal("17"), km);
      Length length2 = length.to(km);
      assertSame(length, length2);
  }
}

BasicType objects may contain business logic in the same way as other domain objects. The following screencast illustrates how to implement the conversion.

Alternative video format (mpg)

BasicType is a stored in the same table as the Domain Object referencing it. It corresponds to JPA @Embeddable.

There are a lot of cases when it is a good idea to introduce types.

Identifers, natural business keys. It is more readable to pass around an identifier type instead of a plain String or Integer
Money
Range
Quantity

I can recommend reading When to Make a Type, Martin Fowler.

Say Hello

2009-08-15T06:30:00.000+02:00

In previous article our Planet is capable of constructing a greeting message. This article shows how to make it possible for a client application to say hello to the Planet.

Alternative video format (mpg)

Let us create a PlanetService to expose the sayHello method to clients. We lookup the Planet from its name using the built in findByKey repository operation. In Sculptor model file this looks like this:

      Service PlanetService {
         String sayHello(String planetName) throws PlanetNotFoundException;
     }

     Entity Planet {
         gap
         scaffold
         String name key
         Long population min="0"
         Long diameter min="0" nullable
         -Set<@Moon> moons opposite planet
      
         Repository PlanetRepository {
             findByKey;
         }
     }

All hand written java code we need to add is for testing and two trivial lines in PlanetServiceImpl:

    public String sayHello(ServiceContext ctx, String planetName)
       throws PlanetNotFoundException {

       Planet planet = getPlanetRepository().findByKey(planetName);
       return planet.greeting();
   }

Adding Behaviour

2009-08-11T08:00:00.000+02:00

In previous articles we created a simple helloworld application without any hand written code. This article shows how to add some hand written business logic.

Alternative video format (mpg)

The behaviour to implement is that the Planet should be able to construct a greeting message based on its population. Test for this behaviour looks like this:

public class PlanetTest {

  @Test
  public void shouldSayHelloWhenHasPopulation() {
      Planet earth = new Planet("Earth");
      earth.setPopulation(7000000000L);
      String message = earth.greeting();
      assertEquals("Hello from Earth", message);
  }

  @Test
  public void shouldBeQuietWhenNoPopulation() {
      Planet pluto = new Planet("Pluto");
      String message = pluto.greeting();
      assertEquals("", message);
  }
}

The video illustrates how to implement this.

As you see nothing special, you add the business logic in Java as usual.

Separation of generated and manually written code is done by a generated base class and manually written subclass, a gap class. It is in the subclass you add methods to implement the behavior of the Domain Object. The subclass is also generated, but only once, it will never be overwritten by the generator.

The gap class is not generated initially. When you need a gap class you specify that in the DSL with gap keyword.

Testing is Simple

2009-08-08T06:30:00.000+02:00

In previous articles we created a simple helloworld application without any tests. This article shows how to do integration testing of services with Sculptor.

Alternative video format (mpg)

We turn on generation of junit test and complete the failing tests from previous articles. For each Service there is a generated JUnit test class that we are encouraged to implement. It uses Spring transactional test fixtures and DbUnit.

Spring beans are injected in the test with ordinary @Autowired annotations.

public class PlanetServiceTest extends AbstractDbUnitJpaTests
    implements PlanetServiceTestBase {
  private PlanetService planetService;

  @Autowired
  public void setPlanetService(PlanetService planetService) {
    this.planetService = planetService;
  }

  @Test
  public void testFindById() throws Exception {
    Planet found = planetService.findById(getServiceContext(), 1L);
    assertEquals("Earth", found.getName());
  }

  @Test
  public void testFindAll() throws Exception {
    List<Planet> found = planetService.findAll(getServiceContext());
    assertEquals(2, found.size());
  }

  @Test
  public void testSave() throws Exception {
    int countBefore = countRowsInTable("PLANET");
    Planet planet = new Planet("Pluto");
    planet.setPopulation(0L);
    planetService.save(getServiceContext(), planet);
    assertEquals(countBefore + 1, countRowsInTable("PLANET"));
  }

  @Test
  public void testDelete() throws Exception {
    int countBefore = countRowsInTable("PLANET");
    Planet planet = planetService.findById(getServiceContext(), 2L);
    planetService.delete(getServiceContext(), planet);
    assertEquals(countBefore - 1, countRowsInTable("PLANET"));
  }
}

The initial test data is defined in DbUnit xml file. The database is refreshed for each test method.

<?xml version="1.0" encoding="UTF-8"?>

<dataset>
  <PLANET ID="1" NAME="Earth" POPULATION="7000000000" VERSION="0"/>
  <PLANET ID="2" NAME="Jupiter" POPULATION="0" VERSION="0"/>
  <MOON/>
</dataset>

Above tests only covers the normal cases so far and we should of course do more tests for exceptional cases and validation boundaries, such as negative population.

The tests illustrated here are kind of integration tests and you should do ordinary unit tests for domain objects and other classes of importance.

The World is Changing

2009-07-28T06:00:00.000+02:00

In the previous article the simple helloworld application consisted of one single domain object, the Planet. In this article I will add some more features to the picture, including an association to the Moons of the Planet.

Alternative video format (mpg)

It is good to have a natural business key, which is used for equals and hashCode. The name of the Planet is a candidate. Let us add some properties for the population and diameter of the Planet also. Hibernate validator is supported and validations can be defined directly in the model, e.g. min="0".

We add the Moon Entity and its association to Planet.

Build and start Jetty. The changes are immediately reflected in the generated CRUD GUI.

The DSL and the code generation drives the development and is not a one time shot. The application can be developed incrementally with an efficient round trip loop.

Jump Start

2009-07-22T08:15:00.000+02:00

This article illustrates the first basic steps of how to use Sculptor. We will create a simple hello world application. It will only have one simple domain object and some CRUD operations.

Alternative video format (mpg)

We start by firing off some maven archetype commands to create the business and presentation tier projects. The sculptor archetypes creates maven pom files with needed dependencies. We generate eclipse projects from those maven projects using the ordinary maven-eclipse-plugin. These projects are imported into Eclipse. You find the commands for this in the wiki.

We open the design model that is the input to the code generator. It is a textual DSL that defines the application structure. We define the Planet Entity in a Module.

We also need some CRUD operations to be able to do something with the Planet in the to be generated GUI. It is easiest to add those with the scaffold keyword. Note that the DSL editor has support for code completion, error highlight and outline view (including ctrl+O).

Now it is time to generate code, but first we will turn off one feature. Sculptor promotes writing junit tests, but for this demo we turn off that by adding a property to sculptor-generator.properties.

generate.test=false

We build the application by executing mvn install from the -parent project. The code generation is part of the maven build cycle.

We start the web application server by running mvn jetty:run from the -web project. Note that this is also a maven command, and no specific installation of an application server and database is required. Hsqldb inmemory database is used.

The generated CRUD GUI is available at http://localhost:8080/helloworld-web/

That was everything needed to create a minimal application. It illustrates how easy it is to get started. Within 15 minutes you can go from scratch to a running application, including build scripts, Eclipse projects, domain model, JPA persistence, services, Web Flow application and much more. Thereafter you can continue by evolving the design, add manual code and regenerate.

This was an extremely simple example. Sculptor can be used for much more advanced applications. The CRUD GUI is useful for administrative sections of the application or to serve as a scaffolding for your manually crafted user interface. The major strength of Sculptor is in the business tier when developing typical enterprise or web applications that benefit from a rich and persistent domain model.

What is Sculptor?

2009-06-27T09:16:00.000+02:00

Sculptor is an open source productivity tool that applies the concepts from Domain-Driven Design and Domain Specific Languages.

You express your design intent in a textual DSL, from which Sculptor generates high quality Java code and configuration. It is not a one time shot. The application can be developed incrementally with an efficient round trip loop.

Sculptor is useful when developing typical enterprise or web applications that benefit from a rich and persistent domain model. Sculptor also provides a sophisticated CRUD GUI for administrative sections of the application or to serve as a scaffolding for your manually created pages.

The generated code is based on well-known frameworks, such as JPA, Hibernate, Spring Framework, Spring Web Flow, JSF, RCP, and Java EE.

Product characteristics:

Easy to learn, intuitive syntax of the textual DSL, editor with error highlight, code completion, and outline

Quick development round trip, it is not a one time generation

Existing IDE tools, such as refactoring, code assist and debugger will continue to be of service to you

High quality of generated code, based on well known frameworks, best practices, and a lot of experience

Great extensibility and customization options

Easy to remove the tool, no runtime magic

Visit the Fornax Sculptor web site to learn more and try it.