Distributed command handling with Axon, JGroups and Docker

In a recent project I used Axon Framework together with JGroups, to create a clustered, or distributed command bus.

In that project we had some concurrency issues. One of those issues was that two events were applied on a single aggregate with an identical sequence number. The JGroupsConnector in Axon uses a consistent hashing algorithm to route commands. This ensures that commands with the same routing key will be sent to the same member, regardless of the sending member of that message. Within the project we used the aggregate identifier as the routing key, ensuring that commands for a single aggregate are processed in a single JVM, thus preventing duplicate sequence numbers.

To demonstrate such a setup, I’ve created a simple demo application, based on the latest version of Axon Framework (3.0-M3). Using Docker Compose, the application is launched twice (in two containers). The containers should then form a JGroups cluster, and handle a number of commands. Go check it out on GitHub!

The hierarchy of Continuous Deployment

In this post I’m presenting a twist on Maslow’s hierarchy of needs, the hierarchy of Continuous Deployment. This version is based on the steps that are required to successfully implement Continuous Deployment (a step up from Continuous Delivery).

Maslow’s hierarchy of needs (also known as Maslow’s Pyramid) is a well-known theory in psychology, describing human needs in a particular ranking, with the most basic of needs at the bottom. It has since been updated slightly to deal with 21st century needs.

Maslow's hierarchy of needs

The hierarchy of Continuous Deployment

The needs are ranked in order of increasing scope, difficulty and abstraction.

Code

Trust. It all starts with trust. A development team must trust its code, and trust it well enough to continuously improve and adapt it. TDD, refactoring, continuous integration, a solid dose of code coverage (did I hear anyone say 100%?) and tons of quality metrics lead to a healthy, mature, predictable, and above all, trustworthy code base. Also, pair programming.

Functionality

Story done, feature delivered, high fives all around. Was it the right feature? Wait, what did we ship? Involve the three amigos and come up with scenarios, then test your implementation against those scenarios. BDD, DDD, UBL.

Architecture

A stable, fast build pipeline (preferably, as code) producing consistent artifacts. Respect the build. Verify that service A and service B are still on a first-name basis and talk the same language. Check that the code does not kill the database using some obscure SQL dialect which is no longer supported. Throw 10x the load at the system. Then 100x.

Infrastructure

Time spent firefighting a flaky system is time not spent delivering value. Ensure easily replaced, stable servers with plenty of horsepower. Load balancers left and right. Self-healing, autoscaling clouds for the win.

Continuous Deployment

Automate all the things. Push to master. No human intervention, no manual gates, no downtime during deployments. Fast iterations, happy business. More high fives. Or beer. Or high fives with beer.

Hierarchy of Continuous Deployment

Agree? Disagree? Let me know what you think!

In future blog posts I’ll explain some of the things required to climb all the way to the top of this hierarchy.

Phing development update

A few years ago I pledged to publish regular updates on Phing’s development. I’m a little ashamed that I haven’t done that for nearly three years. Needless to say, a lot has happened, so it’s about time to post an update!

Recent releases

The latest version, 2.14.0, was released in March. Ever since the move of the code to GitHub, a steady flow of pull requests and contributions has resulted in a lot of new functionality, improved documentation and numerous squashed bugs.

Here’s a selection of interesting features that have recently been added:

  • Support for PHP 7 & HHVM.
  • A flurry of new tasks: stopwatch, switch, throw, retry and many more.
  • Better support for composer and numerous dependencies (such as PHPUnit).
  • Additional selectors and conditions.
  • Remember the old HTML user guide? That has been replaced by (significantly improved!) Docbook5 documentation.
  • Additional tests & test coverage.

Of course there’s much more and all the details can be found in the changelog.

Coming up

The next minor version, 2.15.0, should be released in a few weeks, the tickets assigned to that version can be found here.

Beyond Phing itself, there are a few planned changes to the website and the build process:

  • Migrating the ticket tracker (Trac) to GitHub issues.
  • Shipping a “full” .phar file with each release (containing recent versions of the most popular dependencies).
  • Testing the generated .phar file in the travis-ci builds.
  • Adding a documentation search.
  • Refreshing the website.

Phing 3.0

A serious refactor of Phing has been on my own wishlist for quite some time. Some work towards that goal has been committed to the 3.0 branch and I try to keep this branch more or less synchronized with master (albeit with some delay). The branch is entirely functional and requires PHP 5.6+ or above.

Some of the work that’s still in progress or on the roadmap: integrating SPL, changing the way booleans are handled, a proper task autoloader, cleaning up the code, improving the test harness, etc. If you have suggestions, then please let me know! Time available to spend on a pull request is even better 🙂

CD: Continuous Delivery or Continuous Deployment?

I’m writing this post to contribute my take on the acronym CD, and the distinction (or perhaps, confusion) between the phrases Continuous Delivery and Continuous Deployment. These phrases are used interchangeably, and sometimes incorrectly, in various books, blogs and talks. And while these software engineering approaches (or disciplines) share a lot, there is one key difference.

Continuous Everything

Let’s first discuss two important cornerstones of CD: Continuous Integration (CI) and automation.

Continuous Integration has been around for a while. A phrase that was first coined in the nineties, it was further refined and explained by Beck and Fowler. CI refers to the practice of integrating changes to a code base regularly, preferably multiple times per day. Each integration is verified by running some sort of build, which should be self-testing, automated, fast and repeatable. This enables quick detection of errors and conflicting changes, resulting in a more stable (and faster) development process.

Continuous Delivery expands on CI and adds a process to automatically deploy a build to an acceptance environment. There, the team can access and manually test the product. When the build is considered ready for general consumption, a developer can promote it to the production environment by a simple button click.

Now, one of the goals of CD (regardless of what the D means) is to achieve shorter development cycles, i.e. reduce the time for new features or product ideas to be put in production and validated. Having a manual gate somewhere in the release process means that the cycle time won’t be as short as it could be, if everything were automated. Enter Continuous Deployment: automating every single step of the way, no manual promotions. To me, this is true CD. Luckily, Wikipedia agrees with me.

Continuous Delivery vs. Continuous Deployment

In future blog posts I’ll discuss some of the requirements to effectively implement Continuous Deployment, and how to easily set up a proper pipeline in minutes, as code, with Jenkins 2. We’ve come a long way since CruiseControl

Final thoughts

I’ve been thinking about the phrase “acceptance environment”. In true CD, there are no acceptance tests, at least not of the traditional, manual kind. So, with nothing to “accept”, the phrase seems to have lost some of it’s value. From an automated test point of view however, the environment itself is still useful. Ideally equivalent to production, where we can run performance and load tests, and verify deployments before progressing to the production stage.

Pre-production? Functionally Equivalent To Production (FETP) ? Let me know if you can think of something better!

 

The road to continuous deployment: a case study

Below are the slides of my talk “The road to continuous deployment: a case study”, as presented at the Dutch PHP Conference in June 2016. I’m planning to dedicate a series of blogposts to this particular topic later this year.

It’s a situation many of us are familiar with: a large legacy application, limited or no tests, slow & manual release process, low velocity, no confidence…. Oh, and management wants new features, fast.

But how to proceed? Using examples and lessons learned from a real-world case, I’ll show you how to strangle the legacy application with a modern service architecture and build a continuous deployment pipeline to deliver value from the first sprint. On the way, we take a look at testing strategies and various (possibly controversial!) tips and best practices.