Sculptor 3.0 has a new home

Starting with the release of version 3.0 Sculptor has its new home at http://sculptorgenerator.org/.

The development is moved to GitHub https://github.com/sculptor and the new blog is hosted there as well.

So take a look at Sculptors new home!

Sculptor 2.1.0 Released

One very useful new feature is the generated finder methods. It is now possible to specify the query in the model file and let sculptor generate the code for the repository operations. In some cases the query can be derived from the method signature only.

This will generate a query returning all persons with the specified first name.

Entity Person {
  String firstName
  String lastName
  Date birthDate

  PersonRepository {
    findBy(String firstName);
  }
}

More advanced conditions can also be specified

  PersonRepository {
    findBornBefore(Date bd) condition="birthDate < :bd" orderBy="firstName";
    findByName(String name) condition="lastName i= :name or firstName i= :name";
  }

Sculptor generates findByCondition expressions and will report at generation time if there are any errors in the queries.


Another nice addition is the generated builder classes that provides a fluent interface to build domain objects in a manner that can be easier to work with and read.

    Book book = book()
        .createdBy("me")
        .createdDate(now)
        .title("Ender's Game")
        .isbn("Some-ISBN")
        .addMediaCharacter(mediaCharacter()
            .name("Ender")
            .build())
        .build();

Read more about the release.

Sculptor 2.0 is out

This is the 11th major release of Sculptor.

Three new and noteworthy features:

1. The new REST support in Sculptor makes it very easy to expose restful services by using conventions and easy delegation to Services to reduce boilerplate coding. Spring MVC is the underlaying REST framework.

2. Mixin composition is something Java developers are missing, but with Sculptor it becomes available.

3. The services can now easily be used with Spring Remoting. This is convenient for the RCP client, but can be used for other clients also.

In addition to these and several other new features we have also improved the technical quality.

We have upgraded support for latest versions of most of the underlaying tools and frameworks. Most important is Xtext 1.0, Eclipse Helios, Maven 3, Spring 3.0.

The logging API has been changed from Commons Logging with Log4j to SLF4J with Logback. SLF4J allow many backends and also have special bridges for many (legacy) logging frameworks. SLF4J with Logback is considered to be best logging framework available.

Mixin Composition

I think one of the best features in Scala is traits. Using traits it is possible to compose small pieces of behavior and state in an elegant way. I miss traits when I use Java. To mitigate that problem I have implemented support for traits in Sculptor. This article illustrates how this tool can be used for designing good, rich, domain models with traits.

Traits provide a mixin composition mechanism that is missing in Java. Similar to interfaces in Java, traits are used to define object types by specifying the signature of the supported methods. Unlike interfaces, traits can be partially implemented; i.e. it is possible to define implementations for some methods. Similar to abstract classes, but you don't have to wast your single inheritance opportunity.

How many times have you stared at java code like this:
if (p1.compareTo(p2) <= 0)

Why not spell it out, to make the code more readable:
if (p1.lessThanOrEquals(p2))

We seldom do that, because implementing 4 methods, lessThan, lessThanOrEquals, greaterThan and greaterThanOrEquals, in each and every class that needs to be compared is too much work. We stick to the cryptical compareTo.

What if I say you only have to implement them once and then you can easily mixin the methods in each and every class that needs to be compared or sorted in some way.

I hope I don't have to elaborate on how bad idea it is to try to use inheritance (base class) for these kind of reusable pieces of code.

In Sculptor's textual DSL model traits are defined like this:

        Trait Ordered {
            def boolean greaterThan(@Ordered other);
            def boolean greaterThanOrEquals(@Ordered other);
            def boolean lessThan(@Ordered other);
            def boolean lessThanOrEquals(@Ordered other);
            def abstract protected int compare(@Ordered other);
        }
        
        Entity Product with Ordered {
            String name
        }

After code generation this means that you can implement the 4 comparison methods once, in the Ordered trait, and they will be available in Product and other domain objects that are defined 'with Ordered'. The compare method must still be implemented in Product, because it is there you know what to compare with. The internal, generated, implementation is based on delegation, i.e. no magic.

Let us define another trait, which illustrates that traits also can hold state.

        Trait PriceTag {
            - protected Money normalPrice
            protected double discount
            def Money price;
            def Money price(String currency);
        }

        Entity Product with PriceTag {
            String name
        }

        BasicType Money with Ordered {
            BigDecimal amount
            String currency
        }

This means that normalPrice and discount will be mixed in to Product. You implement the price methods in the PriceTag trait. Product and all other domain objects that are defined 'with PriceTag' will have those fields and methods.

Wouldn't it be nice to be able to compare product by price? Let us do that by combining the two traits. First add the compare method in PriceTag, so that you only have to implement it at one place.

        Trait PriceTag {
            - protected Money normalPrice
            protected double discount
            def Money price;
            def Money price(String currency);
            def protected int compare(Ordered other);
        }

Then mixin both traits into Product

        Entity Product with Ordered with PriceTag {
            String name
        }

That's it. We have designed products with a rich price and compare interface.
Note that compareTo is no longer implemented in Product, only in PriceTag.

Try this new feature in latest Sculptor 2.0.0-SNAPSHOT.

EDA in Massive Web Sites

This is a comment to @h3nk3's latest blog post on Concurrency, Parallelism and Actors.

I totally agree that there is no single solution for everything. Regarding massive web sites I would like to add a few things to your conclusions, which makes asynchronous event driven solutions important for web sites also.

You should try to do as little as possible in the request thread, i.e. minimize latency. Threads in them selves are a scarce resource. You can do the essential validation and then hand off the rest of the work to a task queue (e.g. Flickr). You can precompute everything and cache it so that serving the requests becomes trivial (e.g. Reddit). Both these techniques can benefit from using event driven solutions in the backend.

We see great improvements in the area of truly asynchronous web solutions, which means that you don't have to deliver a synchronous response to every request. The result can be pushed to the clients asynchronously. Then it becomes natural to use event driven solutions in the server. The reasons for using Actors it is not only scalability and concurrency, it is also much about using a simpler concurrency model than threads and locks.

Of course it is also important to use HTTP as designed, since HTTP caching is a key ingredient in highly scalable web sites.

Event Sourcing with Sculptor - Snapshots

Yesterday I described how events can be used as storage mechanism. Instead of storing current state we store each change as a Domain Event. Current state is reconstructed by loading and replaying all historical events. For Entities with a long life cycle it can be too many events. This can be solved with an optimization that is based on periodically storing a rolling snapshot of current state.

Let us look at the code in the Sculptor port of the Simple CQRS Example to understand what this means in practice.

When loading InventoryItem we start by applying latest snapshot, if any, and thereafter replaying the events after the snapshot. This is the code in the Repository:

    @Override
    public InventoryItem findByKey(String itemId)
            throws InventoryItemNotFoundException {
        InventoryItem result = super.findByKey(itemId);

        loadFromHistory(result);

        return result;
    }

    private void loadFromHistory(InventoryItem entity) {
        InventoryItemSnapshot snapshot = getInventoryItemSnapshotRepository()
                .getLatestSnapshot(entity.getItemId());
        entity.applySnapshot(snapshot);
        long snapshotVersion = snapshot == null ? 0 : snapshot.getVersion();

        List<InventoryItemEvent> history = getInventoryItemEventRepository()
                .findAllAfter(entity.getItemId(), snapshotVersion);
        entity.loadFromHistory(history);
    }

That is how the snapshots are used when loading InventoryItem. Let us see how they are saved. We define the Snapshot Value Object for InventoryItem like this

        ValueObject InventoryItemSnapshot {
            String itemId index
            boolean activated
            Long version
            
            Repository InventoryItemSnapshotRepository {
                @InventoryItemSnapshot getLatestSnapshot(String itemId);
                protected findByCondition(PagingParameter pagingParameter);
                save;
            }
        }

Here we store the state as explicit attributes, which is simple with Sculptor, since we got the persistence (to MongoDB or JPA) for free, but it can also be stored as a blob (ecoded as xml, protobuf, or whatever you prefer). In this example the only state is the activated flag, but it can be much more in a real application.

The storage of the snapshot is triggered by a subscriber on the ordinary Domain Event flow. Simply defined as this in the model:

        Service InventoryItemSnapshotter {
            subscribe to inventoryItemTopic
            inject @InventoryItemRepository
            inject @InventoryItemSnapshotRepository
        }

The implementation calculates how many events has passed since previous snapshot by comparing version numbers. When the delta exceeds a threshold (e.g. 100 events) a snapshot is created and saved.

    public void receive(Event event) {
        if (!(event instanceof InventoryItemEvent)) {
            return;
        }

        InventoryItemEvent inventoryItemEvent = (InventoryItemEvent) event;
        String itemId = inventoryItemEvent.getItemId();

        InventoryItemSnapshot snapshot = getInventoryItemSnapshotRepository()
                .getLatestSnapshot(itemId);
        long snapshotVersion = snapshot == null ? 1 : snapshot.getVersion();
        long eventVersion = inventoryItemEvent.getAggregateVersion() == null ? 1
                : inventoryItemEvent.getAggregateVersion();
        if (eventVersion - snapshotVersion >= VERSION_DELTA) {
            takeSnapshot(itemId);
        }
    }

    private void takeSnapshot(String itemId) {
        InventoryItem item;
        try {
            item = getInventoryItemRepository().findByKey(itemId);
        } catch (InventoryItemNotFoundException e) {
            log.warn("takeSnapshot failed: " + e.getMessage());
            return;
        }

        InventoryItemSnapshot snapshot = item.createSnapshot();
        getInventoryItemSnapshotRepository().save(snapshot);
    }

By using snapshots we can dramatically improve performance for loading Entities with many historical changes. However, you can always start development without snapshotting and add it later, as a performance enhancement.

Also, note that snapshots and event are immutable and therefore we have great opportunities for using caching for improving performance.

The complete source code for this example is available here: http://github.com/patriknw/sculptor-simplecqrs/

Event Sourcing with Sculptor

A while a go Greg Young published the Super Simple CQRS Example. There are also some good documents of how to use events as storage mechanism at the CQRS Info Site.

I have implemented the same example with Java and Sculptor. In this post I will describe how the Event Sourcing is implemented. Even though my implementation is facilitated by using Sculptor and MongoDB the design can easily be done with other techniques, such as JPA or plain JDBC, and without Sculptor.

A domain model typically holds current state of the world. Event Sourcing makes it possible to see how we got to this state. Essentially it means that we have to capture all changes to an application state as a sequence of events. These events can be used to reconstruct current and past states.

The sample application is a simplified inventory of items. In line with CQRS it has a strict separation of commands and queries. Changes to the InventoryItem domain object are published as Domain Events. The InventoryItem can be modified with a few commands that I have represented as operations in a Service.

    Service InventoryFacade {
        inject @InventoryItemRepository
        createInventoryItem(String itemId, String name);
        deactivateInventoryItem(String itemId);
        renameInventoryItem(String itemId, String newName);
        checkInItemsToInventory(String itemId, int count);
        removeItemsFromInventory(String itemId, int count);
    }

All these commands are handled in the same way, except createInventoryItem, which is slightly different. The Service looks up the Entity and calls corresponding method on the domain object.

    public void deactivateInventoryItem(String itemId) {
        InventoryItem item = tryGetItem(itemId);
        item.deactivate();
        getInventoryItemRepository().save(item);
    }

The Entity performs validation and creates Domain Event for state changes.

    public void deactivate() {
        if (!isActivated())
            throw new IllegalStateException("already deactivated");
        applyChange(new InventoryItemDeactivated(new Date(), getItemId()));
    }

In this case InventoryItem has a flag (state) indicating that the item is activated or not. Note that this state is not changed directly, but it is changed later as a result of the InventoryItemDeactivated event. All changes are done with Domain Events, which is important, because we don't save current state, we only save the Domain Events. The activated flag is not stored explicitly.

The DomainEvent is applied and added to a list of changes, which later will be stored and published.

    private void applyChange(InventoryItemEvent event) {
        applyChange(event, true);
    }
    
    private void applyChange(InventoryItemEvent event, boolean isNew) {
        DynamicMethodDispatcher.dispatch(this, event, "apply");
        if (isNew) {
            changes.add(event);
        } else {
            setVersion(event.getAggregateVersion());
        }
    }

    public void apply(InventoryItemDeactivated event) {
        setActivated(false);
    }

When saving, the InventoryItem instance is saved, but only as a key and version. The version is used for detecting concurrent modifications (optimistic locking). Additionally, the changes, the Domain Events are stored. The version number of InventoryItem is stored in Each Domain Event. An additional sequence number is also used to ensure the correct order of the events.

We need to override the default save method in the Repository to handle these sequence numbers and also save the events.

    @Override
    public InventoryItem save(InventoryItem entity) {
        InventoryItem saved = super.save(entity);

        List<InventoryItemEvent> changes = entity.getUncommittedChanges();
        changes = applyVersionToChanges(changes, saved.getVersion());
        for (InventoryItemEvent each : changes) {
            getInventoryItemEventRepository().save(each);
        }
        entity.markChangesAsCommitted();

        return saved;
    }

    private List<InventoryItemEvent> applyVersionToChanges(
            List<InventoryItemEvent> changes, long version) {
        List<InventoryItemEvent> result = new ArrayList<InventoryItemEvent>();
        long sequence = version * 1000;
        for (InventoryItemEvent each : changes) {
            result.add(each.withAggregateVersion(version).withChangeSequence(
                    sequence));
            sequence++;
        }
        return result;
    }

When saving the Domain Events they are also published to a topic, which the read side subscribes on. That is handled with the publish/subscribe mechanism in Sculptor. In the model we simply need to specifiy publish on the save method.

        abstract DomainEvent InventoryItemEvent {
            persistent
            String itemId index
            Long aggregateVersion nullable
            Long changeSequence nullable
            
            Repository InventoryItemEventRepository { 
                save publish to inventoryItemTopic;
                List<@InventoryItemEvent> findAllForItem(String itemId);
                protected findByCondition;
            }
        }

        DomainEvent InventoryItemDeactivated extends @InventoryItemEvent {
        }

In the read side we add subscribers to this topic.

    Service InventoryListView {
        subscribe to inventoryItemTopic
        inject @InventoryItemListRepository
    }
    
    Service InventoryItemDetailView {
        subscribe to inventoryItemTopic
        inject @InventoryItemDetailsRepository
    }

Alright, then we are almost done. One more thing though, when retrieving a InventoryItem we must replay all events to recreate current state. We do that by overriding the default findByKey method in the Repository.

    @Override
    public InventoryItem findByKey(String itemId)
            throws InventoryItemNotFoundException {
        InventoryItem result = super.findByKey(itemId);

        loadFromHistory(result);

        return result;
    }

    private void loadFromHistory(InventoryItem entity) {
        List<InventoryItemEvent> history = getInventoryItemEventRepository()
                .findAllForItem(entity.getItemId());
        entity.loadFromHistory(history);
    }

To retrieve all events we use a simple query in the InventoryItemEventRepository.

    public List<InventoryItemEvent> findAllForItem(String itemId) {
        List<ConditionalCriteria> criteria = criteriaFor(
                InventoryItemEvent.class).withProperty(itemId()).eq(itemId)
                .orderBy(changeSequence()).build();
        return findByCondition(criteria);
    }

The loaded events are applied to the InventoryItem Domain Object.

    public void loadFromHistory(List<InventoryItemEvent> history) {
        for (InventoryItemEvent each : history) {
            applyChange(each, false);
        }
    }

    private void applyChange(InventoryItemEvent event, boolean isNew) {
        DynamicMethodDispatcher.dispatch(this, event, "apply");
        if (isNew) {
            changes.add(event);
        } else {
            setVersion(event.getAggregateVersion());
        }
    }

    public void apply(InventoryItemCreated event) {
        setActivated(true);
    }

    public void apply(InventoryItemDeactivated event) {
        setActivated(false);
    }

    public void apply(Object other) {
        // ignore
    }

In this example we have used a naive approach when loading the InventoryItem by replaying all historical events. For Entities with a long life cycle it can be too many events. Then we can use a snapshot technique, which I will describe in a separate blog post.

The complete source code for this example can be found here: http://github.com/patriknw/sculptor-simplecqrs/tree/event_sourcing_without_snapshots