Project Documentation

ImageOur project has a number of places where we keep documentation. To name but a few we have a Wiki, Sharepoint Site, Jira etc. When I want to know something that is specific to my project the first place I look is my e-mail archive or shout a question to my team and if it’s a general question about a framework or library then I go for the Google.

We had an issue today with the Oracle Pooled Datasource XML Type handling. Our XML retrieval was coming back with an extra space in the middle of the document. Naturally this was a general question and we hit Google. A few people had some similar issues and had some not so great workarounds mentioned. I then happened to be talking to one of my colleagues and mentioned the problem to him. He told me that they faced this problem a while back and developed a much nicer work around. I would’ve gone with the ugly workaround from the web if I hadn’t spoken to him.

So why the long story; well if the web is the first place we hit for general questions then why don’t we document our general questions on the web so that we may have a better chance of finding it rather than buried deep in someone’s mail trail or an internal wiki or even worse in someone’s head.

The same goes for project specific documentation – often the most useful insights are buried in mail trails – that’s why I search my mail archive before I look anywhere else. It’s not the most effective place to keep some of the most valuable project documentation. The problem is some of these valuable mail trails start with some ad-hoc point-to-point conversation.

I once worked on a project where they had an internal version of Stack Overflow and all discussion, questions, insights, messages were conducted using this application creating a wealth of searchable project documentation. However this does not fit the bill completely because you have to make a conscious decision that the point-to-point conversation you are about to have will end up in a valuable mail trail.

Only if there was a button to “promote” a mail-trail into a searchable forum? Maybe there is – I’m hitting Google …


Spring Integration based Dispatcher-Worker with Worker Queues

In the back-office world the central concept in most of the systems is one of a Trade. A Trade has many events (e.g. Inception, Amend, Novation, Termination). Generally events from different trades can be processed in parallel because they have no interdependencies, however, events from the same trade cannot be processed in parallel due to the fact that they modify the same internal instance of a Trade.

A useful pattern for this kind of scenario is dispatcher-worker with worker queues. Each worker has a job queue which it processes in a sequential fashion. Each job queue only contains events for a single trade. This allows parallel processing across trades while maintaining sequential processing on events for a single trade.


I’ve developed simple version of this concept using Spring Integration. The first step is to create a Router that routes inbound trade events into channels that are specific to a trade. If the channel doesn’t exist then the Router will create a new one and register it with the Spring framework.

public String dispatch(CustomMessage inputMessage) {
  String channelName = inputMessage.getId() + CHANNEL_SUFFIX;

  synchronized (channelName.intern()) {
    if (activeChannels.get(channelName) == null) {
      QueueChannel activeChannel = createNewChannel(channelName);
      PollingConsumer activeConsumer = createConsumerWithWorker(inputMessage, activeChannel);

  return channelName;

Creation of a channel is the only place where synchronisation is required. We only synchronise on the channel name which corresponds to the trade id. Hence contention is minimal. I also attach a Polling Consumer to the channel at the point that the channel is created. Creation of a channel and registering it to Spring framework is quite straight forward as shown in the snippet below:

private QueueChannel createNewChannel(String channelName) {
  QueueChannel activeChannel = new QueueChannel();
  activeChannels.put(channelName, activeChannel);
  applicationContext.getBeanFactory().registerSingleton(channelName, activeChannel);
  return activeChannel;

Although I attach a Polling Consumer to each channel. We don’t have to have a thread per channel. We can use a Task Executor to run the polling consumers which will allow much better control over the number of concurrent threads in the system using a thread pool:

private void startConsumingFromChannel(final String consumerName, final PollingConsumer activeConsumer) {
  applicationContext.getBeanFactory().registerSingleton(consumerName, activeConsumer);

Finally (not yet implemented) you can run a Reaper Thread that can remove channels and consumers that have not seen activity for a specified threshold. You can also back the inbound channel with a Message Store to ensure that the system can come backup in a consistent state on failure.

The source code is at Github.

Speed Reading

It's not often you read a book that changes your perceptions. Speed Reading by Susan Norman/Jan Cisek is one of them. I've always loved reading books but had felt that I read very slow. According to the book reading has many types and you employ a type that suits the purpose for your reading. The problem was not the fact that I read slow but that I only had one style of reading irrespective of what I was reading and why.

The “slow” type where you read every individual word is one where the purpose is “reading for pleasure”. Other styles such as Skimming (looking for key words), Scanning (Looking for very specific information) are more useful for quickly gleaning info from a book.

There are many techniques listed in the book that can make reading a more productive experience. Examples include using your peripheral vision more effectively, preparing your mind before reading, employing your sub conscience mind etc.

Some of the techniques may be quite difficult to master but it will at least change how you view reading. You may even discover that you already use some of these techniques naturally.


The Spirit to Improve

This week in our weekly design committee, on of the developer demonstrated how to use the Saxon Injectors to create mock XQuery functions for unit testing.

It’s been one of the biggest problem with our use of XQuery and the developer solved it in his spare time. It was a much needed reminder that the spirit to always improve the software and the development practices is at the core of a good and effective developer and team.

These kind of efforts always raise the bar that much higher in terms of the effectiveness and quality of what we produce and are life blood of the development effort. It is also a key indicator of a healthy project.

This kind work also serves to motivate us in the face of project delivery pressures and becomes even more important when we feel demotivated by repetitive work and the rush to meet the next deadline.

Encouraging Collaboration via Design Committee

Agile process promotes the view that system design emerges and evolves throughout it’s life. It is a result of continuos discussions and decisions by the teams.

On large projects with multiple teams, the ideal of involving everyone in design discussion becomes very cumbersome. Even if you can organise such sessions, the smallest discussion takes ages to conclude.

The idea of a design committee has been proposed by some team members in particular [Sandro Mancuso]. The committee is made up of representatives from each team who convene on a weekly basis. The format of the meeting is similar to an Open Space. However the ideas/discussions are posted and voted on a day prior to the meeting. This allows the proposer(s) to come prepared for the meeting. Each member is allowed a single vote before every meeting and they may vote on the same topic over several weeks.

The team representatives rotate over time, and are chosen by the team. It is the responsibility of these representatives to act as spokesmen for their team and to feedback the discussions/decision held in the meetings.

The committee may also arrange ad hoc sessions which an be called up by any member of the committee.

At the start of the meeting a facilitator is elected who is also responsible for recording the outcomes for that meeting.

Encouraging Collaboration via Review Branches and Pull Requests

I have always advocated the use of a small well balanced team for projects rather than many less experienced ones. However this is not always possible.

Some time projects require a very fast ramp up with a much larger set of teams where achieving a good balance of talent/skills is not possible and the experience is a commodity. Several large projects I have been involved with in the past have had to make this compromise.

Collaboration is the key. Especially if the system under development is new and going through constant evolution. Although one can argue that healthy systems must always be in a constant state of evolution.

In these cases a good level of collaboration within the team is difficult but achievable. Collaboration across teams is near enough impossible unless we setup explicit processes that encourages such behaviour.

The use of Pull Requests and Review Branches is one such process. Review Branches are essentially very short lived branches, often only lasting a few hours and never longer than a day. The main purpose of the branch is to develop in the smallest increment that is reasonable for the purposes of a review. The development of a single feature would typically result in a number of these branches. The mechanism for inviting others to review a branch and merge into master is known as a Pull Request.

The principle behind the Review Branch is that the codebase is small enough to allow for an effective review by someone in 15 minutes max. In the first instance all members of the teams other than your own are invited to do the review. Once one or two people have agreed to do the review they will then sit with the developer to go through the changes and merge them to master after a successful review. In case of geographical separated teams desktop sharing technologies and conference facilities should be used. If no one volunteers within 5 or 10 minutes then you can invite your own team member to do the review.

The primary reason is to share the knowledge across teams via the review process. Quality assurance is a side effect of this process.

We have found that it often helps if you ask for a “review buddy” who will pick all your pull/review requests for a particular feature. This will help speed up the process and stop you from going over the same ground with different reviewers.

Also team members should report in their daily standup the number of pull requests they have reviewed during the previous day and in scrums of scrums the team should report the number of pull requests each team review. They should also report the number of pull requests that were left unanswered within the agreed time limit. This will allow for rebalancing if certain individuals or teams are taking most of the reviews and others are reluctant to volunteer. Needless to say the reviews should be planned in as part of your capacity planning.

Musings on Zero Downtime Deployment

I’ve been thinking about Zero Downtime Deployment for the past few weeks. I even raised it as a discussion topic in our LSCC Roundtable. Here are the key points discussed. Obviously the feasibility/suitability completely depends on the application and platform architecture. Also this is not an attempt to provide a solution but more to record the thoughts and ideas:

  • Switch over to backup upgrade the primary and then switch back, and then update the backup site.
  • Have a HA configuration, take hosts out upgrade then switch gradually.
  • Allow for parallel runs, so you can switch over to a parallel when updating.
  • Develop application to run on previous and new version of the database. Hot deploy the app and “hot” migrate the database.
  • Use dynamic features in your platform to allow for hot deployment.
  • Use semi structured datastore so data migrations are minimal.
  • Provide targeted releases, i.e. if a particular area of the application changes than only that area is updated and not the whole application.
  • If eventual consistency across partitions is acceptable then deploy in partitions and update partitions gradually, i.e. like Facebook, Google etc.