Spring Integration based Dispatcher-Worker with Worker Queues

In the back-office world the central concept in most of the systems is one of a Trade. A Trade has many events (e.g. Inception, Amend, Novation, Termination). Generally events from different trades can be processed in parallel because they have no interdependencies, however, events from the same trade cannot be processed in parallel due to the fact that they modify the same internal instance of a Trade.

A useful pattern for this kind of scenario is dispatcher-worker with worker queues. Each worker has a job queue which it processes in a sequential fashion. Each job queue only contains events for a single trade. This allows parallel processing across trades while maintaining sequential processing on events for a single trade.

Image

I’ve developed simple version of this concept using Spring Integration. The first step is to create a Router that routes inbound trade events into channels that are specific to a trade. If the channel doesn’t exist then the Router will create a new one and register it with the Spring framework.

@Router
public String dispatch(CustomMessage inputMessage) {
  String channelName = inputMessage.getId() + CHANNEL_SUFFIX;

  synchronized (channelName.intern()) {
    if (activeChannels.get(channelName) == null) {
      QueueChannel activeChannel = createNewChannel(channelName);
      PollingConsumer activeConsumer = createConsumerWithWorker(inputMessage, activeChannel);
      activeConsumer.start();
    }
  }

  return channelName;
}

Creation of a channel is the only place where synchronisation is required. We only synchronise on the channel name which corresponds to the trade id. Hence contention is minimal. I also attach a Polling Consumer to the channel at the point that the channel is created. Creation of a channel and registering it to Spring framework is quite straight forward as shown in the snippet below:

private QueueChannel createNewChannel(String channelName) {
  QueueChannel activeChannel = new QueueChannel();
  activeChannel.setBeanName(channelName);
  activeChannels.put(channelName, activeChannel);
  applicationContext.getBeanFactory().registerSingleton(channelName, activeChannel);
  return activeChannel;
}

Although I attach a Polling Consumer to each channel. We don’t have to have a thread per channel. We can use a Task Executor to run the polling consumers which will allow much better control over the number of concurrent threads in the system using a thread pool:

private void startConsumingFromChannel(final String consumerName, final PollingConsumer activeConsumer) {
  activeConsumer.setBeanName(consumerName);
  activeConsumer.setAutoStartup(true);
  activeConsumer.setBeanFactory(applicationContext.getBeanFactory());
  activeConsumer.setTaskExecutor(consumerExecutorPool);
  applicationContext.getBeanFactory().registerSingleton(consumerName, activeConsumer);
}

Finally (not yet implemented) you can run a Reaper Thread that can remove channels and consumers that have not seen activity for a specified threshold. You can also back the inbound channel with a Message Store to ensure that the system can come backup in a consistent state on failure.

The source code is at Github.

Advertisements

Musings on Zero Downtime Deployment

I’ve been thinking about Zero Downtime Deployment for the past few weeks. I even raised it as a discussion topic in our LSCC Roundtable. Here are the key points discussed. Obviously the feasibility/suitability completely depends on the application and platform architecture. Also this is not an attempt to provide a solution but more to record the thoughts and ideas:

  • Switch over to backup upgrade the primary and then switch back, and then update the backup site.
  • Have a HA configuration, take hosts out upgrade then switch gradually.
  • Allow for parallel runs, so you can switch over to a parallel when updating.
  • Develop application to run on previous and new version of the database. Hot deploy the app and “hot” migrate the database.
  • Use dynamic features in your platform to allow for hot deployment.
  • Use semi structured datastore so data migrations are minimal.
  • Provide targeted releases, i.e. if a particular area of the application changes than only that area is updated and not the whole application.
  • If eventual consistency across partitions is acceptable then deploy in partitions and update partitions gradually, i.e. like Facebook, Google etc.

Saxon XQuery With Multiple Documents

XQuery CloudSaxon is a wonderful API for XML processing. It provides complete support for XPath, XQuery and XSLT. Although I’m always baffled with it’s lack of adoption compared to Xalan and Xerces. Having said that the online documentation can definitely do with some improvement.  The following is a quick example of of how you may execute an XQuery that takes multiple XML documents as input.

@Test
public void runXQueryWithMultipleInputDocuments() throws SaxonApiException {
    Processor processor = new Processor(false);

    DocumentBuilder documentBuilder = processor.newDocumentBuilder();
    XdmNode document = documentBuilder.build(
            new StreamSource(new StringReader("<my><document>content</document></my>")));
    XdmNode document2 = documentBuilder.build(
            new StreamSource(new StringReader("<my><document>content2</document></my>")));

    XQueryCompiler xQueryCompiler = processor.newXQueryCompiler();
    XQueryExecutable xQueryExecutable = xQueryCompiler.compile(
            "declare variable $mydoc external; " +
            "declare variable $mydoc2 external; " +
            "<newdoc><doc1>{$mydoc/my/document/text()}</doc1>" +
            "<doc2>{$mydoc2/my/document/text()}</doc2></newdoc>");

    XQueryEvaluator xQueryEvaluator = xQueryExecutable.load();

    QName name = new QName("mydoc");
    xQueryEvaluator.setExternalVariable(name, document);
    QName name2 = new QName("mydoc2");
    xQueryEvaluator.setExternalVariable(name2, document2);

    System.out.println(xQueryEvaluator.evaluate());
}

This result is an output of:

  <newdoc>
   <doc1>content</doc1>
   <doc2>content2</doc2>
</newdoc>

The difference between UI Designer and UI Developer

In a previous post I talked about developers distinguishing themselves as specialists in a particular part of the application e.g. server side, gui, database etc. This kind of specialisation is counter productive to creating good software. However, there are roles where the specialisation is important.

One such specialisation is the role of the GUI Designer. To be more precise the roles of an Art Director and a User Experience Consultant. The User Experience Consultant works with the customer/business analyst to understand the UI requirements and creates user journeys / screen mocks to define how a user will interact with the system. An Art Director takes these mocks and creates graphical representation of these, adhering to the customer branding. It’s the job of the Art Director to present design choices, helping the customer create a suitable and effective look and feel for the UI. The UI Developer then takes the Design Guidelines from the Art Director and create a faithful representation in the UI technology of choice.

These roles – especially that of the Art Director – are highly specialised and are disciplines in their own right. A developer should not be expected to fullfil them. Yet most people in the Software Engineering industry have not even heard of these. Consequently the most important part of the UI Application – i.e. the User Interface – are put together as an afterthought.

The legacy of misplaced Testing

Recently I’ve been exposed to a number of projects that have been going on for a few years. As you’d expect they are at a stage where the cost of change is phenomenal. The codebase is large, convoluted and very difficult to understand. However, there are a lot of tests both at the system and unit level. A sigh of relief – if the code has a lot of tests then at least that would make refactoring easier. The reality is very different – the tests are even more complicated then the code and significantly add to the cost of change.

Not only we have convoluted unit tests we have a very large set of system tests that exercise lots of variations on the main flows. These are very expensive tests because they require a lot of test data to be setup and go through the full flow in order to verify a very small part in that flow. You can see that as the code base became more convoluted and dependencies became difficult to mock, the developers started  relying more and more on the system tests which proved comparatively easier to setup by copying the setup data from previous tests. If you have hundreds of these and no one knows exactly how many of the main flows/scenario are exercised then what good are these system tests?

How do you turn this tide? One approach is to start moving these variations into unit tests in order to reduce the system test suite into a very small set of targeted tests that are testing the main business scenarios. Granted you’ll have to spend effort on untangling the dependencies. You can adopt an automated testing framework (e.g. JBehave, Cucumber etc.) so the system and unit tests are defined in the language of the business. The units under test become more coarse grained to satisfy these scenarios. This may allow swathes of complicated unit tests to be replaced – don’t be afraid to remove them. You must be horrified – remove tests! – well if they are not improving our confidence in refactoring then one can argue that they are no longer fit for purpose.

I am in no way saying that we stop doing TDD/BDD. Lack of TDD/BDD is most likely the reason we get ourselves into this situation in the first place. This is more an approach to lessen the pain of misplaced and substandard tests – to allow us to gradually turn the tide on the cost of change.

Design aligned to the Problem Vs Design aligned to the Technology

Lets develop a web app from start: Well I know Java pretty well so it’s going to be Java and then the de facto design is Spring + Hibernate with Spring MVC. I know I’ll push the boat out get some AJAX in the mix. Wait I’m no “GUI developer” so let’s go for GWT it’ll generate my Javascript.

Lets develop an enterprise integration app: Well an upstream system needs to send me something – bring out my favourite messaging provider. A bit of Message Driven Bean or Spring Messaging support, some Hibernate for the persistence and Bob’s your uncle.

How often do we see this kind of design? You can have a very similar conversation for C# .Net. This kind of thinking process is further encouraged by so called architects imposing their favourite solutions regardless of the problem. Feels to me that the enterprise application development industry has armed it self with a few hammers and then preceded to whack everything in the head pretending it’s a nail. I’m certainly guilty of this thinking process.

There are many arguments for this approach – uniformity of software assets, availability of skills, developer mobility to name but a few. However, it can also be said that this kind of thinking process stifles innovation. In an industry that is built on innovation, that is a bit of a disadvantage.

A lot of us have effectively constrained ourselves in silos to the point where we are only GUIs developers, server side developers, “don’t touch the database” developers, “allergic to CSS and Javascript” developers so on and so on.

Our application designs are constrained by the prescribed technologies and frameworks often resulting in complicated and unmanageable solutions. This cannot be avoided due to the available skills and technologies. However we need to broaden the choices available to us so that our application designs result in more suitable solutions. There is always a compromise between uniformity and flexibility. Currently we lean heavily towards uniformity sacrificing the flexibility of choice during the design effort. A more balanced approach requires a breed of developer who is at home with learning different technologies. This requires strong fundamentals that underpin the current technology landscape allowing them to switch between technologies and learn new ones with relative ease. It is essential for a senior developer to be a polyglot so that during the design effort he/she is not dominated by one particular technology or framework. The emphasis then shifts towards understanding paradigms and approaches rather than specific implementations. Having said that one cannot implement a solution without seeking expert knowledge.

In conclusion a software engineering effort requires breadth and depth of skills during application design/development and it is our responsibility as professional software developers to ensure that we always strive for the best solution to every problem.

Naming your Unit Tests

Often I see Unit Tests with the test methods that have the same name as the method under test prefixed with the word “test” e.g. testSubmitApplication. This provides no extra information on which “flow” of the mothod is being tested. Other test method names provide a bit more information by suffixing the nature of the test e.g. testSubmitApplicationWithInvalidCriteria. It better but not much better. A number of IDEs actually allow the developer to generate test method names based on the class under test which in my opinion defeats the object.

Unit test methods should be named in such as way that they provide a clear description of the test. In my opinion the prefix “test” is redundent and should never appear in your test method names unless it is part of the domain vocabulary. For example AppicationSubmittedWithInvalidCriteriaMustRaiseException* is more informative then testSubmitApplication. Providing a more descriptive name also serves to keep a clear focus on the flow under test and leads the devloper to create a test method per flow.

*Please Note that the example method name is a simplification. I would consider the term “InvalidCriteria” a bit too high-level for a real unit test. It should be more specific such as AppicationSubmittedWithNoSurnameMustRaiseException.