Friday, October 29, 2010

Event Sourcing with Sculptor - Snapshots

Yesterday I described how events can be used as storage mechanism. Instead of storing current state we store each change as a Domain Event. Current state is reconstructed by loading and replaying all historical events. For Entities with a long life cycle it can be too many events. This can be solved with an optimization that is based on periodically storing a rolling snapshot of current state.

Let us look at the code in the Sculptor port of the Simple CQRS Example to understand what this means in practice.

When loading InventoryItem we start by applying latest snapshot, if any, and thereafter replaying the events after the snapshot. This is the code in the Repository:


@Override
public InventoryItem findByKey(String itemId)
throws InventoryItemNotFoundException {
InventoryItem result = super.findByKey(itemId);

loadFromHistory(result);

return result;
}

private void loadFromHistory(InventoryItem entity) {
InventoryItemSnapshot snapshot = getInventoryItemSnapshotRepository()
.getLatestSnapshot(entity.getItemId());
entity.applySnapshot(snapshot);
long snapshotVersion = snapshot == null ? 0 : snapshot.getVersion();

List<InventoryItemEvent> history = getInventoryItemEventRepository()
.findAllAfter(entity.getItemId(), snapshotVersion);
entity.loadFromHistory(history);
}



That is how the snapshots are used when loading InventoryItem. Let us see how they are saved. We define the Snapshot Value Object for InventoryItem like this


ValueObject InventoryItemSnapshot {
String itemId index
boolean activated
Long version

Repository InventoryItemSnapshotRepository {
@InventoryItemSnapshot getLatestSnapshot(String itemId);
protected findByCondition(PagingParameter pagingParameter);
save;
}
}


Here we store the state as explicit attributes, which is simple with Sculptor, since we got the persistence (to MongoDB or JPA) for free, but it can also be stored as a blob (ecoded as xml, protobuf, or whatever you prefer). In this example the only state is the activated flag, but it can be much more in a real application.

The storage of the snapshot is triggered by a subscriber on the ordinary Domain Event flow. Simply defined as this in the model:

Service InventoryItemSnapshotter {
subscribe to inventoryItemTopic
inject @InventoryItemRepository
inject @InventoryItemSnapshotRepository
}


The implementation calculates how many events has passed since previous snapshot by comparing version numbers. When the delta exceeds a threshold (e.g. 100 events) a snapshot is created and saved.

public void receive(Event event) {
if (!(event instanceof InventoryItemEvent)) {
return;
}

InventoryItemEvent inventoryItemEvent = (InventoryItemEvent) event;
String itemId = inventoryItemEvent.getItemId();

InventoryItemSnapshot snapshot = getInventoryItemSnapshotRepository()
.getLatestSnapshot(itemId);
long snapshotVersion = snapshot == null ? 1 : snapshot.getVersion();
long eventVersion = inventoryItemEvent.getAggregateVersion() == null ? 1
: inventoryItemEvent.getAggregateVersion();
if (eventVersion - snapshotVersion >= VERSION_DELTA) {
takeSnapshot(itemId);
}
}

private void takeSnapshot(String itemId) {
InventoryItem item;
try {
item = getInventoryItemRepository().findByKey(itemId);
} catch (InventoryItemNotFoundException e) {
log.warn("takeSnapshot failed: " + e.getMessage());
return;
}

InventoryItemSnapshot snapshot = item.createSnapshot();
getInventoryItemSnapshotRepository().save(snapshot);
}


By using snapshots we can dramatically improve performance for loading Entities with many historical changes. However, you can always start development without snapshotting and add it later, as a performance enhancement.

Also, note that snapshots and event are immutable and therefore we have great opportunities for using caching for improving performance.

The complete source code for this example is available here: http://github.com/patriknw/sculptor-simplecqrs/

Thursday, October 28, 2010

Event Sourcing with Sculptor

A while a go Greg Young published the Super Simple CQRS Example. There are also some good documents of how to use events as storage mechanism at the CQRS Info Site.

I have implemented the same example with Java and Sculptor. In this post I will describe how the Event Sourcing is implemented. Even though my implementation is facilitated by using Sculptor and MongoDB the design can easily be done with other techniques, such as JPA or plain JDBC, and without Sculptor.

A domain model typically holds current state of the world. Event Sourcing makes it possible to see how we got to this state. Essentially it means that we have to capture all changes to an application state as a sequence of events. These events can be used to reconstruct current and past states.

The sample application is a simplified inventory of items. In line with CQRS it has a strict separation of commands and queries. Changes to the InventoryItem domain object are published as Domain Events. The InventoryItem can be modified with a few commands that I have represented as operations in a Service.


Service InventoryFacade {
inject @InventoryItemRepository
createInventoryItem(String itemId, String name);
deactivateInventoryItem(String itemId);
renameInventoryItem(String itemId, String newName);
checkInItemsToInventory(String itemId, int count);
removeItemsFromInventory(String itemId, int count);
}


All these commands are handled in the same way, except createInventoryItem, which is slightly different. The Service looks up the Entity and calls corresponding method on the domain object.

public void deactivateInventoryItem(String itemId) {
InventoryItem item = tryGetItem(itemId);
item.deactivate();
getInventoryItemRepository().save(item);
}


The Entity performs validation and creates Domain Event for state changes.

public void deactivate() {
if (!isActivated())
throw new IllegalStateException("already deactivated");
applyChange(new InventoryItemDeactivated(new Date(), getItemId()));
}


In this case InventoryItem has a flag (state) indicating that the item is activated or not. Note that this state is not changed directly, but it is changed later as a result of the InventoryItemDeactivated event. All changes are done with Domain Events, which is important, because we don't save current state, we only save the Domain Events. The activated flag is not stored explicitly.

The DomainEvent is applied and added to a list of changes, which later will be stored and published.


private void applyChange(InventoryItemEvent event) {
applyChange(event, true);
}

private void applyChange(InventoryItemEvent event, boolean isNew) {
DynamicMethodDispatcher.dispatch(this, event, "apply");
if (isNew) {
changes.add(event);
} else {
setVersion(event.getAggregateVersion());
}
}

public void apply(InventoryItemDeactivated event) {
setActivated(false);
}


When saving, the InventoryItem instance is saved, but only as a key and version. The version is used for detecting concurrent modifications (optimistic locking). Additionally, the changes, the Domain Events are stored. The version number of InventoryItem is stored in Each Domain Event. An additional sequence number is also used to ensure the correct order of the events.

We need to override the default save method in the Repository to handle these sequence numbers and also save the events.


@Override
public InventoryItem save(InventoryItem entity) {
InventoryItem saved = super.save(entity);

List<InventoryItemEvent> changes = entity.getUncommittedChanges();
changes = applyVersionToChanges(changes, saved.getVersion());
for (InventoryItemEvent each : changes) {
getInventoryItemEventRepository().save(each);
}
entity.markChangesAsCommitted();

return saved;
}

private List<InventoryItemEvent> applyVersionToChanges(
List<InventoryItemEvent> changes, long version) {
List<InventoryItemEvent> result = new ArrayList<InventoryItemEvent>();
long sequence = version * 1000;
for (InventoryItemEvent each : changes) {
result.add(each.withAggregateVersion(version).withChangeSequence(
sequence));
sequence++;
}
return result;
}


When saving the Domain Events they are also published to a topic, which the read side subscribes on. That is handled with the publish/subscribe mechanism in Sculptor. In the model we simply need to specifiy publish on the save method.


abstract DomainEvent InventoryItemEvent {
persistent
String itemId index
Long aggregateVersion nullable
Long changeSequence nullable

Repository InventoryItemEventRepository {
save publish to inventoryItemTopic;
List<@InventoryItemEvent> findAllForItem(String itemId);
protected findByCondition;
}
}

DomainEvent InventoryItemDeactivated extends @InventoryItemEvent {
}


In the read side we add subscribers to this topic.

Service InventoryListView {
subscribe to inventoryItemTopic
inject @InventoryItemListRepository
}

Service InventoryItemDetailView {
subscribe to inventoryItemTopic
inject @InventoryItemDetailsRepository
}


Alright, then we are almost done. One more thing though, when retrieving a InventoryItem we must replay all events to recreate current state. We do that by overriding the default findByKey method in the Repository.


@Override
public InventoryItem findByKey(String itemId)
throws InventoryItemNotFoundException {
InventoryItem result = super.findByKey(itemId);

loadFromHistory(result);

return result;
}

private void loadFromHistory(InventoryItem entity) {
List<InventoryItemEvent> history = getInventoryItemEventRepository()
.findAllForItem(entity.getItemId());
entity.loadFromHistory(history);
}


To retrieve all events we use a simple query in the InventoryItemEventRepository.

public List<InventoryItemEvent> findAllForItem(String itemId) {
List<ConditionalCriteria> criteria = criteriaFor(
InventoryItemEvent.class).withProperty(itemId()).eq(itemId)
.orderBy(changeSequence()).build();
return findByCondition(criteria);
}


The loaded events are applied to the InventoryItem Domain Object.


public void loadFromHistory(List<InventoryItemEvent> history) {
for (InventoryItemEvent each : history) {
applyChange(each, false);
}
}

private void applyChange(InventoryItemEvent event, boolean isNew) {
DynamicMethodDispatcher.dispatch(this, event, "apply");
if (isNew) {
changes.add(event);
} else {
setVersion(event.getAggregateVersion());
}
}

public void apply(InventoryItemCreated event) {
setActivated(true);
}

public void apply(InventoryItemDeactivated event) {
setActivated(false);
}

public void apply(Object other) {
// ignore
}


In this example we have used a naive approach when loading the InventoryItem by replaying all historical events. For Entities with a long life cycle it can be too many events. Then we can use a snapshot technique, which I will describe in a separate blog post.

The complete source code for this example can be found here: http://github.com/patriknw/sculptor-simplecqrs/tree/event_sourcing_without_snapshots

Thursday, October 21, 2010

Switching an existing app to MongoDB

Ron has switched project from using Hibernate/MySQL to MongoDB and wrote notes on the changes and gotchas.
Read it: Switching an existing app to MongoDB

Saturday, September 18, 2010

Photo from San Francisco



Karsten saw this in San Francisco :-)

Thursday, September 9, 2010

EDA CQRS Betting Sample

For todays Xtext/Sculptor seminar we have prepared a new example that illustrates some of the new event-driven features in Sculptor. It highlights a good case for using an architecture in line with Command and Query Responsibility Segregation (CQRS) pattern.

Consider an online betting system. It receives a massive amount of bets. A simple model for that could be:

ValueObject Bet {
String betOfferId
String customerId
Double amount

Repository BetRepository {
save;
}
}

Service BettingEngine {
inject @BetRepository

placeBet(@Bet bet);
}


The BettingEngine process the Bet and stores the information.

Questions pop up:

  • How many bets have been done by customer A?

  • Which customers place the higest bets, on average?

  • What are the top 10 high stakes?



It would be rather easy to develop support for those kind of queries in the master betting engine domain, but problems will soon bubble up.


  • Poor performance - the structure of the domain model is not optimized for all types of queries.

  • Not scalable - single centralized database will become a bottleneck.

  • Hard to change - too much functionality in one monolithic system.



Command-Query Responsibility Segregation (CQRS) comes to the rescue. Simplified it is about separating commands (that change the data) from the queries (that read the data). Separate subsystems take care of answering queries (reporting) and the domain for processing and storing updates can stay focused. The result of the commands are published to query subsystems, each optimized for its purpose.

Back to the betting sample. We are aiming for a design as illustrated in next drawing. Command side on the left (green) and Query side on the right (blue).



We publish domain events when bets are placed. There are several ways to do that in Sculptor, but one easy way is to mark a Service operation with the publish keyword in the model.

DomainEvent BetPlaced {
- Bet bet
}

Service BettingPublisher {
publishEvent(@BetPlaced betEvent) publish to jms:topic:bet;
}


Then, in the hand-written java code simply invoke this method. In this case a one-liner in BettingEngine.

That's all for the command side, now let us take a look at the query side. We define a new module or even better a completely new business component.


Module customer {
Consumer BettingConsumer {
inject @CustomerStatisticsRepository
subscribe to jms:topic:bet
}

Service BettingQueryService {
getHighBetters => CustomerStatisticsRepository.findHighAverageCustomers;
}

Entity CustomerStatistics {
gap
String customerId key
int numberOfBets
double averageAmount index

Repository CustomerStatisticsRepository {
findByKey;
save;
List<@CustomerStatistics> findHighAverageCustomers(double limit);
protected findByCondition;
}
}
}


We receive the events published from the betting engine by defining the subscribe keyword in the BettingConsumer.

In the query side we use domain objects that are optimized for the views and reports that are needed. We denormalize the data to minimize the number of joins needed when retrieving the data. Data is calculated ahead of time. In this case we calculate the average amount for the CustomerStatistics.

Since we are using publish/subscribe via a message bus, it might take a while until the query side is updated with the latest data, but that is not a problem. Most systems can be eventually consistent on the query side.

The full source code for the example is available here: http://github.com/patriknw/sculptor-betting

In the example we have chosen MongoDB as persistence store, Camel together with ActiveMQ as event bus. It is a matter of simple configuration to use something else, such as Oracle with JPA/Hibernate and Spring Integration for the event bus.

I was pretty impressed myself when I counted the number of lines of code that was required to implement this example. 70 lines! 40 in the model and 30 lines of hand written java code. User interface not counted.

Sculptor Presentation

Today we had a Sculptor presentation as part of the seminar 'Developing Domain-Specific Languages with Xtext'. The slides from this presentation is available at slideshare: http://www.slideshare.net/patriknw/sculptor

Thursday, September 2, 2010

Free Seminar: Developing DSLs with Xtext

Next Thursday, September 09, in Stockholm, we will talk about Sculptor in the context of how to develop textual Domain-Specific Languages with Xtext. In this talk Sven Efftinge, original creator of Xtext, will present Xtext. I'm looking forward to listen to Sven's presentation.

More information and registration here: http://jwsdsl09sep.eventbrite.com/

Thursday, August 26, 2010

EDA Akka as EventBus

Some time since last entry in the EDA-sequence, but here we are again. Today we are going to do something really interesting. A friend and colleague of us, Jonas Bonér, is the creator of a super interesting framework called Akka.
Akka is an event driven platform for constructing highly scalable and fault tolerant applications. It is built with Scala, but also have a rich API for java. It follows the Actor Model and together with Software Transactional Memory (STM), it raises the abstraction level and provides an easy to use tool for building highly concurrent applications.
So, today we are going to take advantage of the java API in Akka to do our own EventBus implementation.

First, update your pom with the stuff needed for Akka (repo's and dependency):
<repository>
<id>Akka</id>
<name>Akka Maven2 Repository</name>
<url>http://www.scalablesolutions.se/akka/repository/ </url>
</repository>

<repository>
<id>Multiverse</id>
<name>Multiverse Maven2 Repository</name>
<url>http://multiverse.googlecode.com/svn/maven-repository/releases/</url>
</repository>

<repository>
<id>GuiceyFruit</id>
<name>GuiceyFruit Maven2 Repository</name>
<url>http://guiceyfruit.googlecode.com/svn/repo/releases/ </url>
</repository>

<repository>
<id>JBoss</id>
<name>JBoss Maven2 Repository</name>
<url>https://repository.jboss.org/nexus/content/groups/public/ </url>
</repository>

...

<dependency>
<groupId>se.scalablesolutions.akka</groupId>
<artifactId>akka-core_2.8.0</artifactId>
<version>0.10</version>
</dependency>


Second, create your implementation of the bus, AkkaEventBus.java:


package org.foo;

import org.fornax.cartridges.sculptor.framework.event.Event;
import org.fornax.cartridges.sculptor.framework.event.EventBus;
import org.fornax.cartridges.sculptor.framework.event.EventSubscriber;

import se.scalablesolutions.akka.actor.ActorRef;
import se.scalablesolutions.akka.actor.ActorRegistry;
import se.scalablesolutions.akka.actor.UntypedActor;
import se.scalablesolutions.akka.actor.UntypedActorFactory;

public class AkkaEventBus implements EventBus {

public boolean subscribe(final String topic, final EventSubscriber subscriber) {
UntypedActor.actorOf(new UntypedActorFactory() {
public UntypedActor create() {
return new ActorListener(topic, subscriber);
}
}).start();

return true;
}

public boolean unsubscribe(String topic, EventSubscriber subscriber) {
// TODO : implement mapping between arbitrary subscriber and actor in registry
return true;
}

public boolean publish(String topic, Event event) {
ActorRef[] actorsForTopic = ActorRegistry.actorsFor(topic);
for (int i = 0; i < actorsForTopic.length; i++) {
actorsForTopic[i].sendOneWay(event);
}
return true;
}

@SuppressWarnings("unchecked")
private class ActorListener extends UntypedActor {
final String topic;
final EventSubscriber subscriber;

ActorListener(String topic, EventSubscriber subscriber) {
this.topic = topic;
this.subscriber = subscriber;
this.getContext().setId(topic);
}

@Override
public void onReceive(Object message) throws Exception {
this.subscriber.receive((Event) message);
}

}
}

As you can see, I lack the unsubscribe implementation and I left out equals and hashCode overrides, but that I leave to you.
Third, add it as our bus implementation through spring config:


<bean id="akkaEventBus" class="org.foo.AkkaEventBus"/>
<alias name="akkaEventBus" alias="eventBus"/>


What we have done now is built an highly scalable and concurrent EventBus that dispatches event asynchronously.
Pretty easy, right? :-)

It doesn't run over the network, but Akka has some nice modules for that as well, so that is our task for next time.

Wednesday, August 11, 2010

EDA events over system boundaries - with camel

Last time we examined how we could make our events travel over system boundaries with the help of JMS and Spring-integration.
Today we will do exactly the same but we will use Apache Camel as event engine instead. The task is to enable application domain events over system boundaries through JMS. If you don't remember our model, here it is again:
Application Universe {
basePackage=org.helloworld

Module milkyway {
Service PlanetService {
@BigLandingSuccess landOnPlanet(String planetName, String astronautName)
publish to milkywayChannel;
}

DomainEvent BigLandingSuccess {
String planetName
String astronautName
}
}
Module houston {
Service GroundControlService {
subscribe to milkywayChannel
bringOutTheChampagne(int noOfBottles);
}
}
}
To use Apache Camel as event engine, you need to specify it in the sculptor-generator.properties file:
integration.product=camel
And add the needed dependencies to the projects pom-file:

<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jms</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-camel</artifactId>
<version>5.3.2</version>
</dependency>
<!-- xbean is required for ActiveMQ broker configuration in the spring xml file -->
<dependency>
<groupId>org.apache.xbean</groupId>
<artifactId>xbean-spring</artifactId>
<version>3.7</version>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.1</version>
</dependency>
Regenerate and you will have a new camel.xml file in src/main/resources. This file is only generated once, and you can use it to add configuration for Camel.
To do the same as we did with spring-integration, i.e. put a domain event on a jms topic, we just have to add route rule to the camel.xml file:

<camel:route>
<camel:from uri="direct:shippingChannel"/>
<camel:to uri="jms:topic:shippingEvent"/>
</camel:route>

And that's it. Now we are publishing our domain event to an ActiveMQ topic. Yes, a bit strange that it works, but it gets clearer if we look in camel.xml and see whats was already there:

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:camel="http://camel.apache.org/schema/spring" xmlns:broker="http://activemq.apache.org/schema/core" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean id="camelEventBusImpl" class="org.fornax.cartridges.sculptor.framework.event.CamelEventBusImpl"/>
<alias name="camelEventBusImpl" alias="eventBus"/>

<camel:camelContext id="camel">
<camel:package>org.sculptor.shipping</camel:package>
<camel:template id="producerTemplate"/>
</camel:camelContext>
<!--
Camel ActiveMQ to use the ActiveMQ broker
-->
<bean id="jms" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL" value="tcp://localhost:61616"/>
</bean>
</beans>

We have a bean that wires the jms api's to the actual ActiveMQ instance together. So you have to have an ActiveMQ instance up and running listening on port 61616.

All for today, next time...we'll see what the subject is...

Monday, August 2, 2010

EDA events over system boundaries

In the last post we explained why we have the 'event bus' notion. And earlier we have seen how to use it to publish and subscribe, both through the dsl in the model, or through plain java code.
Today we thought we should look how you can make domain events part of the public api of your business component and publish them to the rest of the world.
To do this we will use the event bus implementation that is based on spring integration. We will use its JMS outbound channel adapter to publish the 'BigLandingSuccess' domain event to a jms topic that the rest of the world can listen to.
If you don't remember, here is the model:

Application Universe {
basePackage=org.helloworld

Module milkyway {
Service PlanetService {
@BigLandingSuccess landOnPlanet(String planetName, String astronautName)
publish to milkywayChannel;
}

DomainEvent BigLandingSuccess {
String planetName
String astronautName
}
}
Module houston {
Service GroundControlService {
subscribe to milkywayChannel
bringOutTheChampagne(int noOfBottles);
}
}
}

But first, how do we replace the default event bus implementation with the spring-integration based?
In sculptor-generator.properties, add the property:
integration.product=spring-integration

And in the project pom file, add the dependency:
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>org.springframework.integration</artifactId>
<version>1.0.4.RELEASE</version>
</dependency>
Regenerate, and you are home. So, now all event routing will use spring-integration as engine. So without any other changes, it works exactly as before.
Now you have to decide how your public api should look like, and in our case we decides that the 'BigLandingSuccess' domain event should be made available for other applications to listen to.
And now we take advantage of the power of spring-integration. When we regenerated with spring-integration enabled, we ended up with the file:
src/main/resources/spring-integration.xml
This is the place where all spring-integration definitions lands. It is generated once, so you are now free to edit it. Open the file and add:
<jms:outbound-channel-adapter id="publicMilkywayChannel"  destination="publicMilkywayTopic" channel="milkywayChannel"/>
You will need a spring jms bean named 'publicMilkywayTopic' and that bean needs to map to an actual message broker instance, for example ActiveMQ. But the spring bean and message broker setup is out scope for this blog entry, so that one we leave to you.
The only caveat now is that we are exposing our java objects in the message. This can be cured with a transformation step before the jms adapter. So add a transformer to your spring-integration configuration:
<object-to-string-transformer input-channel="milkywayChannel" output-channel="milkywayMessagesAsStringChannel"/>
And change the jms outbound channel to wire up with the new channel:
<jms:outbound-channel-adapter id="publicMilkywayChannel"  destination="publicMilkywayTopic" channel="milkywayMessagesAsStringChannel"/>
Spring-integration has a lot of transformers you can use. The above just transforms a java object to a string through its toString method. That is of course not always what you want, but it works for this example.

So that was all that had to be done to make a domain event part of your public api. Of course, you need to have proper documentation to give others a fair chance of finding your event.

If we take one step back and consider what we have done.
First, we declared the 'BigLandingSuccess' domain event and publishes it internally.
Second, we add an adapter that listens to the channel and in its turn publishes it on a jms topic, i.e. makes it public.
Third, we added a transformation step before doing a public publish to remove the dependency to our classes.

Now, this is nice, isn't it? We can have a lot of domain events internally in our application and by that take advantage of all the nice attributes of EDA. And we can by choice make a domain event public with 'just' configuration changes.

Quite long post, but we got pretty much done. Next time we will look how to do the same with Apache Camel

Sunday, August 1, 2010

EDA why the event bus in sculptor?

We have come to our seventh entry about Event Driven Architecture. We are in the topic of how Sculptor supports EDA. In previous posts we have covered how to publish and subscribe, both through the dsl in the model, and through plain java code.
Today we will talk a bit more on the thin layer we call 'the event bus'.

As stated before, there are a lot of different approaches to EDA. You can use/implement it locally or apply it too the entire enterprise. But also, in its simplest form its about the observer pattern, regardless if we talk about big or small implementations. This leads us to the main motivations of our event bus abstraction.
We want to keep it simple, but at the same time still have the power of doing event handling big and small. In the Enterprise, or locally. And doing this with the same programming interfaces, i.e. keeping it simple.
So, based on the above we have the event bus api. And with the risk of repeating my self, it is a very simple one. Methods for publishing, subscribing, and un-subscribing.

It also ships with a default implementation named (you got it) simple event bus. This one is really easy (hey, I found another word instead of simple) to use and fully functional on its own. Though, if you need to integrate with another system you are going to need another implementation.
And that is one of our other motivation for the event bus, it should be easy to swap implementations.
Beside the default, we currently have two implementations based on Spring-integration and Apache Camel. Both of these are what's called "lightweight integration frameworks". Supporting these two frameworks brings a lot of power to the solution when it comes to integration.
And that brings us to the next topic of this series. Some examples of how to bring your events to life over system boundaries, i.e. integration stuff.

Friday, July 30, 2010

EDA pub/sub with plain java

In a couple of posts we have seen how to publish and subscribe to events. I showed how to do that through the dsl in the model. But you have of course also the possibility to do pub/sub by plain java code as well. The key thing is to have a handle to the event bus.
The easiest way to explain it is to show an example, so, a simple service that publish an event as part of it's business logic:


@Service
public class GroundControlService {

@Autowired
@Qualifier("eventBus")
private EventBus eventBus;

@Override
public void receive(Event event) {
if (event instanceof BigLandingSuccess) {
eventBus.publish("earthChannel", new Celebration("We did it!"));
this.bringOutTheChampagne(ALL);
eventBus.publish("supplierChannel", new MissingResource("Champagne"));
}
}
}
And for the other end, the subscriber:

@Service
public class SupplierService {

@Autowired
@Qualifier("eventBus")
private EventBus eventBus;

public void init() {
eventBus.subscribe("supplierChannel", new EventSubscriber() {
@Override
public void receive(Event event) {
if (event instanceof MissingResource) {
raceForContract(event);
}
}
});
}
}

Not so hard.
Now, here and there I've used the term 'event bus', and you have also seen it in java code above. Next time we will look closer into the bus. Why is there an abstraction and what can you do with it?

Thursday, July 29, 2010

EDA a simple example with Sculptor continued

Lets continue with our simple example from the last post. We declared a domain event in the model and we marked a service to publish that event to a channel.
Now, what about getting notified when such an event is published?

Continuing with our model from last time, its just a matter of editing it. Lets create another module with a service that is interested of the event:

Application Universe {
basePackage=org.helloworld

Module milkyway {
Service PlanetService {
@BigLandingSuccess landOnPlanet(String planetName, String astronautName)
publish to milkywayChannel;
}

DomainEvent BigLandingSuccess {
String planetName
String astronautName
}
}
Module houston {
Service GroundControlService {
subscribe to milkywayChannel
bringOutTheChampagne(int noOfBottles);
}
}
}

The result is that the GroundControl service will implement the EventSubscriber interface and be marked with @Subscribe annotation. That means that the GroundControl service will automatically be added as subscriber to milkywayChannel. It will be notified, receive method called, when events are published to that channel.
You will get a stub of the receive method of the EventSubscriber interface.


@Override
public void receive(Event event) {
// TODO Auto-generated method stub
throw new UnsupportedOperationException("receive not implemented");
}

So that you will need to implement with your logic:


@Override
public void receive(Event event) {
if (event instanceof BigLandingSuccess) {
this.bringOutTheChampagne(999999);
}
}
Ok, so now we covered some very simple notations for publishing and subscribing to events. Next time I'll show how to do the same but in plain java code.

Monday, July 26, 2010

EDA a simple example with Sculptor

Now we have gone through EDA in general, and briefly covered how we support it in Sculptor. It's time to see how it can be used.
We start with a simple example where we will show how to declare a domain event and how that event is published.
Let us use our old, well known, hello world example.

Application Universe {
basePackage=org.helloworld

Module milkyway {
Service PlanetService {
void landOnPlanet(String planetName, String astronautName);
}

}
}
That is our model. So, what might the rest of the 'universe' be interested of here? Well, if NASA sent out a space ship with an astronaut on it who's mission was to land on some far far away planet, wouldn't they be interested in the event of a landing? I think so. So, when a astronaut lands on a planet, we would like the service to publish an event of this happening.
Lets start with re-defining the model with the event:
Application Universe {
basePackage=org.helloworld

Module milkyway {
Service PlanetService {
void landOnPlanet(String planetName, String astronautName);
}

DomainEvent BigLandingSuccess {
String planetName
String astronautName
}
}
}

DomainEvents may contain attributes and references in the same way as ValueObjects and Entities. DomainEvents are always immutable and not persistent.

Events are about something happening at a point in time, so it's natural for events to contain time information. Sculptor automatically adds two timestamps, occurred and recorded. The time the event occurred in the world and the time the event was noticed.

Ok, so now we have the event defined, now we want it to be published. The easiest way of doing this is to mark the service in the dsl (you can of course publish events programmatically, but more about that in a later blog entry) . So, once again, lets re-define our model:

Application Universe {
basePackage=org.helloworld

Module milkyway {
Service PlanetService {
@BigLandingSuccess landOnPlanet(String planetName, String astronautName)
publish to milkywayChannel;
}

DomainEvent BigLandingSuccess {
String planetName
String astronautName
}
}
}
The operation must return a DomainEvent or take a DomainEvent as parameter. That event is published to the defined channel when the operation has been invoked.

As an alternative the DomainEvent instance can be created from the return value or parameters. The DomainEvent must have a matching constructor.

The result of the above declarative way of publishing events is a generated annotation @Publish on the method. It will trigger the Spring AOP advice PublishAdvice that is part of Sculptor framework.

Recap: We have declared an event in our model, and further marked our service to publish this event when it happens. As an API, I think this is very neat. Everyone who is interested can see that when an astronaut lands on a planet, he or she can be notified.
And that is what we will talk about next time:

How can I be notified when things happens?

Monday, July 19, 2010

EDA sculptor support

In the previous post we gave an overview of Event Driven Architecture. It is a very big area, and you can use it to a lot of things.
It is a very good complement to DDD, and is a corner stone when building scalable systems.
When constructing systems it is a very nice match to accomplish loosely coupled modules and bounded contexts, i.e. business components. And much more.

We strive for simplicity, but at the same time not restricting us.

So, how can you use with Sculptor?
In the 1.9.0 release, we have focused on support for Publish/Subscribe, Comman-Query Responisbility Segregation (CQRS) and Event Sourcing.

The most useful part is of course the pub/sub support. Event sourcing is an architectural style that has its niche. CQRS is also an architectural style that there has been some publicity around lately. CQRS uses pub/sub and can with advantage be constructed to use Event Sourcing.

To support the above, there are three central parts in our implementation:
  • An event bus abstraction
  • DomainEvent
  • CommandEvent
"The bus"
The event bus is an extremely simple API, with three different implementations (in 1.9.0), "Simple", Spring Integration and Apache Camel. The idea is that the central parts, i.e. pub/sub, should be easy to use. The only thing you have to work with is an event bus where you publish and subscribes to and from events. And if you need some non functional behavior (asynchronism, over the wire, etc) for your events, you plug in an event bus that can handle this requirements.
And as stated above, the easiest way to accomplish that with the 1.9.0 release is to use Spring Integration or Apache Camel. But you can also choose to implement your own event bus.
You can publish and subscribe to and from the bus either declarative through the DSL, or programatically through the event bus API.

DomainEvent vs CommandEvent
CommandEvent is an instruction for something to happen. The system processes a CommandEvent and takes appropriate actions.

DomainEvent states fact - that something has happen. This fact is published to the rest of the world and the publisher just lets it go with no further interest in what happens to the event, i.e. who receives it and what they do.

If you compare it to an application API, CommandEvent could be the input API, while the DomainEvent is the output API for the application. I.e. the CommandEvent is what you can make the application perform, while the DomainEvent is what the application reports has happen.

As you probably figured out, CommandEvent is for implementing CQRS and Event Sourcing.

Finally, let us take quick look at how the notion for DomainEvents look like in the DSL.
DomainEvent ShipHasArrived {
- ShipId ship
- UnLocode port
}

DomainEvent ShipHasDepartured {
- ShipId ship
- UnLocode port
}
As you can see, you define a domain event in the same way as for entities or value objects.

And to declare pub/sub:
Service TrackingService {
@ShipHasArrived recordArrival(DateTime occurred, @Ship ship, @Port port)
publish to shippingChannel;
}
Service Statistics {
subscribe to shippingChannel
int getShipsInPort(@UnLocode port);
reset;
}


That was all for today, next time will it be more concrete with examples of how to use it.


Thursday, July 15, 2010

EDA overview

As stated in the last post, we would like to talk a bit (very shortly) about EDA in a broader sense to give an overview of it and to give you a sense for what we read into the term EDA. This is quite important as a background when we go further and explain our interpretation and implementation of it.

Event Driven Architecture is a very broad area. But in a short sentence its about:
Applications and systems that produce, consume and reacts on events.

And even if there is a lot of attention on EDA as means of implementing integration between applications and systems, EDA can be applied within applications and even in parts of applications.

The benefits of an EDA is:
  • loosely coupled systems (or internals of a system)
  • high performance (fire and forget)
  • high scalability
Of course this comes with a trade-off. An extra abstraction is added, i.e. it becomes more complex.

In it's simplest form, EDA is about the Observer Pattern. Something happens somewhere, and 0 to n parties is interested in that happening. From this simple pattern, all interpretations, implementations and usages of EDA are spawn.

So, before we go into how we have implemented support for EDA in Sculptor, we will give you some examples of what shapes we think EDA can take. Big and small...here we go:

GUI's
Swing uses the Observer pattern. Many different GUI frameworks uses an event driven approach. Java Server Faces works with events, event handlers and event actions. Etc...

DomainEvent
A Domain Event is registration of something that has happen. It might be of interest to 0 to n consumers. The thing is; The producer doesn't care, it just tell the world that this thing has happen and happily goes on with it's life.

PubSub
Publisher and Subscribers is a central part of all event driven integrations. It is an implementation of the Observer pattern. It often comes with persistent events. Topics in the Java/JMS tech domain is an implementation of it.

SOA2.0
Now when SOA has been around for some years the next thing is SOA2.0. In the 2.0 version of SOA events plays a central role as a complement to services.
An event-driven system typically consists of event emitters (or agents) and event consumers (or sinks). Sinks have the responsibility of applying a reaction as soon as an event is presented. The reaction might or might not be completely provided by the sink itself. For instance, the sink might just have the responsibility to filter, transform and forward the event to another component or it might provide a self contained reaction to such event. The first category of sinks can be based upon traditional components such as message oriented middleware while the second category of sinks (self contained online reaction) might require a more appropriate transactional executive framework, for example an ESB.

Event servers
Event servers are servers that are specialized in processing events. They often provide filter/query capabilities for events. Weblogic Event Server is an example of this kind of product.

Event sourcing
All changes to an application is recorded as a series of events. An applications current state can be queried. But not only that, an application state can be rolled forward or backwards as you wish, giving you a lot of power. Martin Fowler has written all about it here. Also, Patrik has written a couple of blog entries of how he has played with it and now also added support for it in Sculptor, see here and here.

CQRS
Command and Query Responsibility Segregation is a very cool area. I will not try to explain it in detail, it is very well done here.
Simplified it is about separating commands (that change the data) from the queries (that read the data). Separate subsystems take care of answering queries (reporting) and the domain for processing and storing updates can stay focused. The result of the commands are published to query subsystems, each optimized for its purpose.

Ok, that was probably a very short and incomplete list of EDA related topics, but it gave you a taste for it and a ground for further reading. We will use it as a foundation when we talk about how we think about and implements EDA.
If you think we have missed an important topic, tool, concept, etc, around EDA, please add a comment about it.

Next up, how we support EDA in Sculptor.

Monday, July 12, 2010

EDA intro

Both Patrik and I have always been believers of Event Driven Architecture. EDA has been around for many years, but recently the attention towards it has increased. The reasons for that is probably many, but I guess one of the most important is that EDA will help you build more concurrent and scalable system.
Therefore, as we strive to support popular technologies, we take the opportunity to implement support for EDA in the newly released 1.9.0 version of Sculptor.

Patrik has written a couple of interesting posts (here and here) about event driven architecture in a special implementation model, event sourcing.
Event sourcing is a powerful tool in certain circumstances. Though, not always a perfect match.
Also, many people reads SOA2.0 or ESB when you say EDA. This is for sure true in many cases, but just a small set of the truth.
EDA is much more than event sourcing, SOA2.0 or ESB implementations.
EDA plays a big role in application architecture and design (or even module design) as well.

Our approach is that you as a developer should be able to choose how and where you implement EDA.

That is what we will talk about in a sequence of posts.
EDA in general.
EDA, how we see it.
EDA, how we support it.

So, stay tuned for more about EDA.

Sunday, July 11, 2010

What's Next, after 1.9.0

Next release will mostly be a technical upgrade. Xtext and Xpand version 1.0 were released together with Eclipse Helios. We will upgrade to that.

Spring 3.0 and JPA 2.0 are also important upgrades.

Oliver will contribute with more cool stuff. He has some almost finished things that he will share with us (hopefully) soon.

The original design of Sculptor has worked fine and we will not change it, but 3 years of additions of different combinations of target implementations is starting to hurt some templates. Therefore we will remove some old (probably unused) target implementations. Those have been deprecated in 1.9.0 and in case you are using any of it you should start migration to some of the more modern alternatives:

  • EJB/Spring combination => migrate to EJB3 or Spring

  • EJB 2 => migrate to EJB3 or Spring

  • Hibernate mapping with XML => migrate to JPA/Hibernate with annotations

  • Spring definitions in XML => migrate to Spring with annotations

  • Hibernate without JPA => migrate to JPA/Hibernate


This makes a good reason to use version number 2.0.0 for next release. Personally I get bad vibrations when Open Source project go to 2.0. That often means total redesign and no backwards compatibility. That will not be the case for Sculptor 2.0. It will be a cleanup, and some pieces are not supported any more, but most of it will stay intact and be compatible (with some migration steps, as usual).

Exploring new features in 1.9.0

This post is dedicated for those of you who are already using Sculptor and would like to have a quick overview of some selected features in the release. I will not bring up bug fixes and all improvements. Please refer to the complete list in issue tracker for that.

I have written a separate teaser about the MongoDB and EDA features of the new release.

Generated Documentation

In the model you can write documentation as a quoted string in front of almost every element (attribute, reference, entity, operation, ...). From the model Sculptor generates HTML documentation of all domain objects including their attributes and associations. Documentation of Services are also included. The generated result is located in src/generated/resources/DomainModelDoc.html

Improved Graphviz Visualization

Several diagrams with different focus and level of detail are generated. They are also included in the generated documentation. Samples here. There is a new maven plugin fornax-graphviz-m2-plugin that generates images (.png) from the .dot files.

Syntax Diagrams

We have mainly focused on sample based documentation. Some users have asked for more formal syntax description of the DSL. We have created railroad syntax diagrams.

Aggregate Syntax

Aggregates can now be defined with the new belongsTo keyword. It is more informative than the previous !aggregateRoot.


Entity Cargo {
- TrackingId trackingId key
- Location origin required
- Location destination required
- Itinerary itinerary nullable inverse opposite cargo
}

ValueObject Itinerary {
belongsTo Cargo
not optimisticLocking
not immutable
- Cargo cargo nullable opposite itinerary
- List legs inverse
}

"An itinerary consists of one or more legs."
ValueObject Leg {
belongsTo Cargo
- CarrierMovement carrierMovement;
- Location from;
- Location ^to;
}


Nullable in key

It is now allowed to have some nullable fields as part a composite natural key.


ValueObject HandlingEvent {
- Type type key
- CarrierMovement carrierMovement nullable key
- Location location key
DateTime completionTime key
DateTime registrationTime
- Cargo cargo key opposite events
}


ToStringStyle

We are using commons-lang for toString of domain objects. It has a style parameter, which is now possible to define in sculptor-generator.properties. You can choose between several styles. I like this format:

toStringStyle=SHORT_PREFIX_STYLE


You can also override this for individual DomainObject with hint:

ValueObject Foo {
hint="toStringStyle=MULTI_LINE_STYLE"
String aaa
String bbb
}


databaseJoinTable

It is possible to define many-to-many join table with databaseJoinTable and its columns with databaseColumn at both sides of the bidirectional association:

- Set<@Media> existsInMedia opposite mediaCharacters
databaseJoinTable="MED_CHR" databaseColumn="CHR"


BTW opposite doesn't have to be defined last any more.

Those keywords are also useful for unidirectional to-many associations. Additionally databaseJoinColumn is used, since there is no opposite side to define the column on.

- Set<@Person> playedBy databaseJoinTable="CHR_PERS"
databaseColumn="PERS" databaseJoinColumn="CHR"


Clob/Blob

Clob and Blob are now supported with default java types String and byte[]. They are mapped with JPA @Lob.

GUI with Not Persistent Value Objects

The structure of the persistent domain model might not always match the presentation, for example you might want a dialog for editing several domain objects in one single screen. Then you can create a non-persistent ValueObject with a corresponding Service. The application service handles the transformation between the presentation object and the persistent domain objects. More documentation and sample is available here.

Sculptor 1.9.0 - Support for MongoDB and Event-Driven Architecture

The Sculptor development team has a track record delivering around 3 releases per year. We have delivered 10 releases in total. That means that the core pieces are rock solid. I have personally used it successfully together with 15 other developers on daily basis for about a year now. It simply works very well.

We care about bugfixing and making small improvements. At the same time we are excited about learning new technology and using emergent design. This release contains two new features in the area of scalability. Persistence backed with MongoDB and support for Event-Driven Architecture.

MongoDB bridges the gap between key-value stores (which are fast and highly scalable) and traditional RDBMS systems (which provide rich queries and deep functionality). I think this makes MongoDB very interesting for applications that need high-performance and/or scalability, but also prefer using a rich persistent domain model with complex associations. The schema less structure is attractive from a developer productivity perspective, which is one of the two goals with Sculptor (quality is the other).

Sculptor generates data mapper classes that converts domain objects to/from MongoDB data structures, DBObjects. This makes it easy to use a domain model à la DDD with automatic mapping to MongoDB data structures.

Sculptor provides generic repository operations for use with MongoDB. This includes operations such as save, delete, findById, findByKey, findByCondition, and some more. You get CRUD operations, and GUI, for free.

Queries can be expressed with a slick fluent api that support code completion and refactoring.

Rich support for associations. Aggregates are stored as a embedded documents. Other associations are stored with referring ids. In the domain objects there are generated getters that lazily fetch associated objects from the ids. This means that you don't have to work with the ids yourself, you can follow associations as usual.

Read more about how to use MongoDB with Sculptor here.

Event-Driven Architecture (EDA) is a good complement to Domain-Driven Design. We think EDA is an important ingredient for building scalable systems. It is also an enabler for designing loosely coupled modules and bounded contexts.

For this Sculptor makes it possible to define Domain Events in the model in similar way as Entities and Value Objects. Sculptor also provide a mechanism to publish and subscribe through a simple event bus. This is done either in a declarative way in the model, or programatically with a simple API.

Implementations of the event bus that integrates with Apache Camel and Spring Integration are available out-of-the-box.

In addition to publish/subscribe Sculptor also has support for CQRS and EventSourcing.

CQRS is about separating commands (that change the data) from the queries (that read the data). Separate subsystems take care of answering queries (reporting) and the domain for processing and storing updates can stay focused.

Event Sourcing makes it possible to see how we got to the current state and query how the state looked liked in the past. Essentially it means that we have to capture all changes to an application state as a sequence of events. Sculptor provides a default implementation of EventSourcing, which can be extended and customized to fit specific needs.

Read more about how to use the new event features of Sculptor here. We will blog more about this topic later, so stay tuned.

Friday, June 4, 2010

New Home Page

New Sculptor home page is live: http://sculptor.fornax-platform.org/



I have also updated the original TSS article: Improving Developer Productivity with Sculptor.

Tuesday, June 1, 2010

Event Sourcing Snapshots

I got a relevant question on yesterdays post. How does the snapshot mechanism work?

The easiest way to reconstruct a historical state is to start with a clean database, maybe populated with static reference data, and then apply all events in order up to the point you are interested in. This will take forever if your system has been running for a while, it contains a lot of events.

The trick to make it work is to periodically take a snapshot of the complete database. When doing replay it can start with the preceding snapshot and only replay events after the snapshot time up to the time you are interested in.

Taking a snapshot can be implemented in various ways. I have tried an approach that aims at being able to take a snapshot without stopping/locking normal operation, i.e. inflow of events and processing of them can continue.

The steps to create snapshots are illustrated by the following figure. It also illustrates a query to replay events to a specific point in time.



How do I copy database? For MongoDB there is a simple copydb command, which can be executed from the application. Exactly how to do this with other databases is vendor specific, but I guess it is possible somehow. At least in command line admin mode. Since I always copy from a read-only (snapshot) database it is should be a consistency safe operation.

These operations will of course not be super fast and temporary databases for historical queries must probably be prepared in advanced to have decent response times. If you need to do a lot of historical queries you need something more, such as a separate system responsible for answering such queries. It may be a data warehouse, a denormalized database, or whatever tailored for the purpose. This separate system is fed from the same original event stream. The design is called CQRS.

Monday, May 31, 2010

Prototyping Event Sourcing

In Domain Language Newsletter March 2010 we could read Eric Evans writeup on Domain Events.

Over the last few years it has become clear that it is very useful to add a new pattern to the DDD "Building Blocks" (Entities, Value Objects, Services, etc.) -- Domain Events. This pattern has been around for a long time. Martin Fowler wrote a good description.


Andreas and I have lately been prototyping on some mechanism to better support Event-Driven Architecture in Sculptor. In this article I will describe how to do Event Sourcing.

A domain model typically holds current state of the world. Event Sourcing makes it possible to see how we got to this state and query how the state looked liked in the past. Essentially it means that we have to capture all changes to an application state as a sequence of events. These events can be used to reconstruct current and past states.

In the prototype we have used Martin Fowler's shipping system example. The full source code for the sample is available in Subversion. I have developed Event Sourcing with existing Sculptor mechanisms. Next step will be to move the general pieces into the tool so that it can be easily used. I would like to share the ideas right away in this blog.
The domain model for the shipping system is very simple. Ships that carry cargo and move between ports. It looks like this in Sculptor DSL:
Entity Ship {
gap
- ShipId shipId key
String name
- Port port nullable
- Set<@Cargo> cargos

Repository ShipRepository {
gap
inject @CargoRepository
save;
findByKey;
}
}

Entity Port {
- UnLocode unlocode key;
String city
- Country country

Repository PortRepository {
save;
findByKey;
}
}

Entity Cargo {
gap
String cargoId key
boolean hasBeenInCanada

Repository CargoRepository {
save;
findByKey;
}
}

BasicType ShipId {
String identifier key
}

"United nations location code."
BasicType UnLocode {
String identifier key
}

enum Country {
US,
CANADA
}

When a ship arrives at port it is registered in the TrackingService, recordArrival operation. Instead of taking that information and directly save it to current domain objects, the TrackingService creates a DomainEvent and pass it in to the processor for further execution.
public void recordArrival(DateTime occured, Ship ship, Port port) {
DateTime now = new DateTime();
ArrivalEvent event = new ArrivalEvent(occured, now, ship.getShipId(), port.getUnlocode());
getDomainEventProcessor().process(event);
}





The DomainEventProcessor stores the event, which is important for audit and possibility to replay events to reconstruct past states. The DomainEventProcessor is generic, knows nothing about the shipping domain. It dispatches the event to an EventHandler that knows how to process the shipping specific events. As a start I use simple Spring dependency injection to select EventHandler, but we will probably use a slightly more sophisticated mechanism later. The important thing is that the generic DomainEventProcessor doesn't know how to process events, so it delegates to application specific event handlers.

A useful utility for dispatching events to separate methods is available in commons beanutils. The ShippingEventHandler looks like this (error handling removed):

public void handleEvent(DomainEvent event) {
dispatch(event);
}

/**
* Runtime dispatch to handle method with correct event parameter type
*/
protected void dispatch(DomainEvent event) {
MethodUtils.invokeMethod(this, "handle", new Object[] { event });
}

public void handle(ArrivalEvent event) {
Ship ship = getShipRepository().findByKey(event.getShip());
Port port = getPortRepository().findByKey(event.getPort());
ship.arrival(port);
getShipRepository().save(ship);
}

public void handle(DepartureEvent event) {
Ship ship = getShipRepository().findByKey(event.getShip());
Port port = getPortRepository().findByKey(event.getPort());
ship.departure(port);
getShipRepository().save(ship);
}
The events are also defined in Sculptor DSL. So far I have used ordinary ValueObjects for the events but we will add a special syntax for defining events like this:
DomainEvent ArrivalEvent {
- ShipId ship
- UnLocode port
}

DomainEvent DepartureEvent {
- ShipId ship
- UnLocode port
}
What have we done? We have made the design more complicated by an intermediate event step, but we have also a foundation for all the exciting things that can be done with Event Sourcing.

We can query the state of a ship for a specific point in time, and it is not only the state of the ship, the complete domain model can be used as usual at a specific point in time.

DateTime to = travelStart.plusDays(15);
ReplaySpecification replaySpec = new ReplaySpecification().withTo(to).withTarget(tmpDb);
domainEventProcessor.replay(replaySpec);
DbManager.setThreadInstance(tmpDb);
Ship ship = referenceDataService.getShip(shipId);
Port port = ship.getPort();
Set<Cargo> cargos = ship.getCargos();
When doing this prototype we have used MongoDB, which is excellent for these kind of features because:
  • it is schema-less, so any event type can be stored without predefined schema
  • it has low latency, so replay of events are fast
  • it is easy to make new database instances and copy database, which make replay and snapshot mechanisms simple
Note that in the above example we replay the events into a temporary database (tmpDb), a Parallel Model. This is a full featured MongoDB instance and therefore we have all the ordinary query capabilities, i.e. exactly the same domain model including repositories can be used with the temporary database as with the ordinary database.

The generic DomainEventProcessor provides several methods to operate on the events. For example to speed up replay of large data volumes it is possible to create snapshots. Latest preceding snapshot is used as base when replaying, i.e. only the events after the snapshot time need to be processed.

The event processor module is defined like this in the prototype. The idea is that it will be provided automatically by Sculptor. Of course with customization possibilities to support variations.
Module event {
Service DomainEventProcessor {
inject @DomainEventRepository
inject @SnapshotRepository

process(@DomainEvent event);
getAllEvents => DomainEventRepository.findAll;
replayAll;
replayUpTo(DateTime timePoint);
replay(@ReplaySpecification spec);
save(List<@DomainEvent> events);
String createSnapshot(DateTime timePoint);
}

abstract ValueObject DomainEvent {
gap
DateTime occured index;
DateTime recorded;

Repository DomainEventRepository {
save;
save(List<@DomainEvent> entities);
findAll(PagingParameter pagingParameter);
findBetween(DateTime from, DateTime to, PagingParameter pagingParameter);
protected findByCondition(PagingParameter pagingParameter);
}
}

ValueObject ReplaySpecification {
not persistent
DateTime from nullable
DateTime to nullable
DB target nullable
}

ValueObject SnapshotInfo {
DateTime timePoint index
String name

Repository SnapshotRepository {
@SnapshotInfo findLatest(DateTime timePoint);
String snapshotName(DateTime timePoint);
@SnapshotInfo createSnapshot(DateTime timePoint);
copyDb(String fromDbName, String toDbName) => CopyDbAccessObject;
protected findByCondition;
protected save;
}
}
}
We are also prototyping how to add other event mechanisms, such as publish/subscribe and integration with various products, such as Spring Integration, Apache Camel and Akka. More about that later.

Tuesday, May 4, 2010

Generating Syntax/Railroad diagrams from Xtext

Problem
Some months ago we started to speak with Patrik about syntax (railroad) diagrams for Sculptor. Syntax of Sculptor DSL is very rich now and we have no formal reference documentation. In Advanced and Developer doc many details of language are described but pure syntax description would be very helpful.

Solution
I started looking for some tools. I found following generator for syntax diagrams:

Little bit old but output looks good. Now we need only BNF definition for our language. I looked at Xtext definition and start to manually change it. For example this Xtext:
DslValueObject :
(doc=STRING)?
(abstract?="abstract")? "ValueObject" name=ID ("extends" (("@"extends=[DslValueObject]) | (extendsName=DslJavaIdentifier)))? "{"
("package" "=" package=DslJavaIdentifier )?
(((notOptimisticLocking?=NOT "optimisticLocking") | ("optimisticLocking")) |
((notImmutable?=NOT "immutable") | ("immutable")) |
((cache?="cache") | (NOT "cache")) |
((gapClass?="gap") | (noGapClass?="nogap")) |
(scaffold?="scaffold") |
("hint" "=" hint=STRING) |
("databaseTable" "=" databaseTable=STRING) |
("discriminatorValue" "=" discriminatorValue=STRING) |
("discriminatorColumn" "=" discriminatorColumn=STRING) |
("discriminatorType" "=" discriminatorType=DslDiscriminatorType) |
("discriminatorLength" "=" discriminatorLength=STRING) |
("inheritanceType" "=" inheritanceType=DslInheritanceType) |
("validate" "=" validate=STRING) |
((notPersistent?=NOT "persistent") | ("persistent")))*
((attributes+=DslAttribute) |
(references+=DslReference))*
(repository=DslRepository)?
"}";


have to be transformed to this BNF notation:
DslValueObject :
STRING?
"abstract"? "ValueObject" ID ("extends" (("@"DslValueObject) | DslJavaIdentifier))? "{"
("package" "=" DslJavaIdentifier )?
((NOT? "optimisticLocking") |
(NOT? "immutable") |
(NOT? "cache") |
("gap" | "nogap") |
"scaffold" |
("hint" "=" STRING) |
("databaseTable" "=" STRING) |
("discriminatorValue" "=" STRING) |
("discriminatorColumn" "=" STRING) |
("discriminatorType" "=" DslDiscriminatorType) |
("discriminatorLength" "=" STRING) |
("inheritanceType" "=" DslInheritanceType) |
("validate" "=" STRING) |
(NOT? "persistent"))*
(DslAttribute |
DslReference)*
DslRepository?
"}";
As you can see, it is not that different. In first round I did it manually but I found that it will be possible automatize with short sed script. Store following to convertToBNF.sed:

s/[A-Za-z]\+.=//g
s/\[//g
s/\]//g
s/""/" "/g
s/(\([A-Za-z"]\+\))/\1/g
s/^enum *//
s/^terminal *//
/^grammar /d
/^generate /d

You can apply it to Xtext file by:
sed -f convertToBNF.sed Sculptordsl.xtext > Sculptordsl.bnf

Now you can copy and paste content of Sculptordsl.bnf to text area on previous mentioned site:

For making it nice we will use bigger font and than resize to 50% with anti-aliasing. Set font size to 22 and image size to double of expected result image. For me with our example it's width 2000 and height 2000. For big grammars more images are generated on page. Just save whole page to some directory. All new browsers always store html file and all other resources like images, styles, javascripts to separate directory. Go to this resource directory and run following command (this need ImageMagic convert tool):
for i in *.png; do convert $i -trim -resize 52% ${i%.png}-S.png; done

Now you have small version of original syntax diagrams with nice anti-aliasing. Here is my result:


Conclusion
As you can see, generating nice looking syntax diagrams from Xtext grammar isn't that difficult. In next days we will make page with all Sculptor DSL syntax diagrams. Question is: "Can I apply it for my Xtext grammar too?". Sed tool is working with pure text. It's possible to confuse sed with strange declaration but for normal grammars it should work. It will be better to have special tool which will parse Xtext notation, maybe some transformation and simplification for nicer results and generate BNF notation directly, but this is bigger job and sed work nice with Sculptor Xtext. Here is full version of sed script which I'm using now for transforming Xtext grammar to BNF notation. You can see also some transformations at end of sed script which do beautification of BNF:

s/[A-Za-z]\+.=//g
s/\[//g
s/\]//g
s/""/" "/g
s/(\([A-Za-z"]\+\))/\1/g
s/^enum *//
s/^terminal *//
/^grammar /d
/^generate /d
$i \
ID :\
LETTER (LETTER | NUMBER)*\
\
STRING :\
"""" CHAR+ """";\
\
LETTER :\
"A-Z" | "a-z";\
\
CHAR :\
LETTER | NUMBER | " ~!@#$%&*()_+-={}[]:|\';/.,;<>?";\
\
INTEGER :\
NUMBER+;\
\
NUMBER :\
'0-9';\

# Syntactic sugars
# X | (Y X) => Y? X
s/\([A-Za-z"]\+\) *| *(\([A-Za-z"]\+\) *\1)/\2? \1/g

# X | (X Y) => X Y?
s/\([A-Za-z"]\+\) *| *(\1 *\([A-Za-z"]\+\))/\1 \2?/g

# (X Y) | Y => X? Y
s/(\([A-Za-z"]\+\) *\([A-Za-z"]\+\)) *| *\2/\1? \2/g

# (X Y) | X => X Y?
s/(\([A-Za-z"]\+\) *\([A-Za-z"]\+\)) *| *\1/\1 \2?/g