Roughly a month ago I posted the first in, what should become, a series of development updates concerning Phing. This second blog post details the most important updates during the past four weeks as Phing is moving steadily towards another release (2.4.10 is due out next week).

User Guide starts move to DocBook format

The biggest commit last month was without a doubt the merge of Johan Persson’s docbook5 branch, which adds a DocBook 5 version of the entire Phing User Guide,  accompanied by its own rendering infrastructure. This feature has been on my personal wishlist for a long time, and it’s finally around the corner! The new docs will probably not make it into 2.4.10, but, at some point in the coming weeks, the ‘live’ user guide (on the Phing website) will be switched to the new format. The 2.4.11 packages will also contain the new docs.

Liquibase docs

After succesfully merging the Liquibase tasks last month, Stephan Hochdörfer completed the addition by submitting a pull request containing the necessary documentation!

PhpDocumentor

You’ve probably heard about this, the popular documentation tools phpDocumentor and DocBlox have merged! Phing supports the new phpDocumentor releases by adding a phpdoc2 task.

PHP 5.4 compatibility

Phing itself seems to work fine with the recently released PHP 5.4, however, the unit tests experienced some problems – these have been corrected to make sure the (unstable) snapshot builds are pushed out regularly.

PHAR package

Starting with the upcoming 2.4.10 release, each new version of Phing will also be available as a PHAR package. Initially, this will just contain Phing itself (similar to the PEAR package). After that, the package will be expanded with a few popular dependencies (depending on feedback).

Various (bug)fixes, the 2.4.10 release

Last but not least, here’s a list of tickets that have been closed since the last update.

Last year, Chris Shiflett started the Ideas for March movement, an effort to revitalize the interesting conversations that happen(ed) on blogs. This year, several people are refreshing that effort.

This is my pledge to write (blog) more about what I see and do, the projects that I work on, the tools that I use daily, the people I meet, the events I visit – the things that inspire me. I will be blogging about all things Phing, hopefully post a blog or two about Gearman, SOAP, ZF, security and numerous other subjects that have been stuck in my grey matter, but never managed to make it into a blog post.

Plus, and I’m very excited about this, my good friend Joeri is busy working on a redesign of this site, which will hopefully make all that new content more appealing and easy to read!

The community site joind.in is the definitive community event tracker, and it’s entirely open source! Today I learned that the site has been nominated for the 2012 Impact Awards. As a conference visitor and speaker, I visit and use this site regularly (you can find my speaker profile here) and am really excited for this nomination!

Please vote here!

Starting today, I’ll post regular updates on the development of Phing (which is, as you might know, a PHP build tool based on / inspired by Apache Ant).

For the past three years I’ve been the lead developer on this open source project, and thus responsible for deciding the direction of the project, integrating the various contributions, publishing releases, etc. Should you be interested, Lorna Jane Mitchell published an excellent post detailing some of the challenges faced by open source project leads.

The last few months have been especially interesting, as I’ve moved the Phing source code to GitHub. This has greatly improved the number of contributions and in general speeded up the development of Phing.

Let’s discuss some of the interesting commits that were made during the past two months!

Try/Catch task

A feature that has been requested before (and has been on my personal wishlist) is the ability to run task(s) whenever a task fails (very similar to PHP’s try/catch statement). During my talk at the PHPUK2012 conference, one of the attendees asked when this feature could be expected. So, while waiting for my flight home I decided to get busy and add the trycatch task.

<trycatch property="prop.testTryCatchFinally">
	<try>
		<fail>Tada!</fail>
	</try>

	<catch>
		<echo>In &lt;catch&gt;.</echo>
	</catch>

	<finally>
		<echo>In &lt;finally&gt;.</echo>
	</finally>
</trycatch>

Parallel task

After a very interesting talk by Jeroen Keppens (@jkeppens) during this year’s PHP Benelux conference, Mike van Riel (@mvriel), the author of DocBlox, and me set out to enable some form of parallel processing in both Phing and Docblox. Mike authored a generic library, which I used in the new parallel task. This task is still very experimental (and it only works on Linux due to some OS limitations), but it’s definitely a nice example of how the community works and inspires!

<parallel threadCount="100">
	<echo>Job 1</echo>
	<echo>Job 2</echo>
	<echo>Job 3</echo>
	<echo>Job 4</echo>
	<echo>Job 5</echo>
	<echo>Job 6</echo>
	<echo>Job 7</echo>
	<echo>Job 8</echo>
	<echo>Job 9</echo>
	<echo>Job 10</echo>
	<echo>Job 11</echo>
	<echo>Job 12</echo>
	<echo>Job 13</echo>
	<echo>Job 14</echo>
	<echo>Job 15</echo>
	<echo>Job 16</echo>
</parallel>

Liquibase

Together with Stephan Hochdörfer (@shochdoerfer) I’ve been working on integrating his Liquibase (a Java database refactoring / change management application) tasks into the Phing core. The liquibase tasks will complement the dbdeploy task and assist the user with database changes / rollbacks.

ApiGen

ApiGen is a Nette framework based documentation generation tool for PHP 5.3. Phing now supports this tool through the new apigen task, contributed by Jaroslav Hanslík.

RNG schema updates

For a while now, Phing ships with a RNG schema to validate your build files. However, this schema was incomplete and did not contain some of the more recent tasks and changes. Johan Persson has done some invaluable work to get the RNG schema up to date.

Various (bug)fixes, the 2.4.10 release

Steady progress has been made towards the 2.4.10 version (which is scheduled to be released early April). Below is an overview of the tickets that have been closed so far:

Besides those commits, a good amount of other (small) fixes and contributions will make it into 2.4.10. A complete list of commits can be found here. Keep those pull requests coming!

Below are the slides of my Phing talk I gave at the PHP UK Conference 2012.

Also, some of the examples discussed during the talk can be found here.

Below are the slides of the Phing talk I gave during the PHPNW11 conference.

Finally got around to doing the ZCE (http://www.zend.com/services/certification/) exam yesterday, and I passed! Click the logo below to see my certification details.

Below are the slides of a presentation on Phing I gave at this year’s Dutch PHP Conference.

Jenkins CI (the new name of Hudson) is a very popular continuous integration system. It can be used to monitor the execution of various jobs, including but not limited to compilation, packaging, testing and deploying of software. Also, it is very easy to configure and comes with a great set of (3rd party) plugins.
I use Jenkins in a number of ways: to monitor and prepare & test new releases of Phing, monitor various internal processes (such as backup logs), and build and deploy various other projects that I work on.

In this post I will expand on some of the techniques discussed in an earlier IBM developerWorks article, to (automatically) build and deploy Java J2EE applications to a WebSphere server. The code fragments listed below are contained in a downloadable archive that you’ll be able to download at the end of this post.

Requirements

To get started, you’ll need to have installed:

• Jenkins CI with the following plugins (can be installed via “Manage Jenkins” -> “Manage Plugins”):
  • Copy Artifact
  • Blame Subversion
  • Parameterized Trigger
  • RAD Builder
• Ant Contrib
• IBM Rational Application Developer
• A test/staging installation of the WebSphere Application Server

This post assumes you have some knowledge of Ant, and are able to install Jenkins and IBM RAD.

Job configuration

For this particular case we will configure two Jenkins CI jobs: one job will build number of artifacts (in this case, .ear files) from source code contained in a version control repository, and another job will deploy the generated artifacts to a WebSphere server. This deployment job will be triggered when the build job completes (successfully).

Jenkins CI Dashboard

Build job

Create a new “free-style” job, and configure it as you normally would. Make sure you check out the source code to the src directory, within the job workspace.

Build Job - SVN config

Then, click “Add build step”, and select the IBM RAD plugin. The field “build file” should contain the path to the build file we will use (Builder\build.xml in the archive). The field “RAD workspace” points to a directory (within the job’s workspace) where a RAD workspace will be created, in this case (see the build file below) we use the path “rad-workspace”. The other settings can be left at their default values.

Build Job - RAD builder

Build file

The Jenkis RAD builder plugins creates a fresh workspace, similar to the workspace that is used inside RAD (or Eclipse). To prepare this workspace with the right configuration settings, we use the task workspacePreferenceFile. The input for this task is a simple preferences file, either text format (key=value pairs, see sample), or the Eclipse .epf format.

compiler.compliance=1.5
compiler.source=1.5
classpath.SOMELIBRARY=D:\Development\somelibrary.jar

The task workspacePreferenceFile is then called in the setup-workspace target.

<target name="setup-workspace"
	description="Sets the preferences for the current workspace">

	<!-- Debug information -->
	<echo level="verbose" message="rad.preferences.filename=${rad.preferences.filename}"/>

	<!-- Set the workspace preferences -->
	<workspacePreferenceFile
		PreferenceFileName="${rad.preferences.filename}"
		useEclipsePrefs="false"
		overwrite="true"/>
	<echo level="verbose" message="workspacePreferenceFile done"/>
</target>

Next, the code that has previously been checked out by Jenkins will need to be copied to this new workspace. The properties copy.from.path and copy.excludes (optional, comma-separated list of excluded patterns) are set in the IBM RAD builder configuration (build job).

<target name="copy-projects"
description="Copies the content of a folder to the current workspace">

	<!-- Debug information -->
	<echo level="verbose" message="copy.from.path=${copy.from.path}"/>
	<echo level="verbose" message="workspace=${workspace}"/>

	<copy
		todir="${workspace}"
		includeEmptyDirs="true">
		<fileset dir="${copy.from.path}" excludes="${copy.excludes}">
			<include name="**/**"/>
		</fileset>
	</copy>
	<echo level="verbose" message="copy done"/>
</target>

Now that the workspace is configured and contains the projects we’d like to build, it’s time to make RAD aware of the contents by actively importing each project – this is done by calling the projectImport task. The list of projects is generated by scanning the workspace for directories that contain a .project file.

<target name="import-projects"
	description="Imports a set of projects into the current workspace">

	<!-- Retrieve list of projects (folders containing a .project file) -->
	<dirset id="projects.list" dir="${workspace}">
		<include name="*"/>
		<present targetdir="${workspace}">
			<mapper type="glob" from="*" to="*/.project" />
		</present>
	</dirset>
	<pathconvert property="projects.name" refid="projects.list" pathsep=",">
		<map from="${workspace}" to=""/>
	</pathconvert>

	<!-- Debug information -->
	<echo level="verbose" message="projects.name=${projects.name}"/>

	<!-- Import the projects -->
	<foreach
		list="${projects.name}"
		target="import-project"
		param="project.name"/>
</target>

<target name="import-project">
	<!-- Debug information -->
	<echo level="verbose" message="project.name=${project.name}"/>
	<echo level="verbose" message="workspace=${workspace}"/>

	<projectImport
		projectName="${project.name}"
		projectLocation="${workspace}/${project.name}"/>
	<echo level="verbose" message="projectImport ${project.name} done"/>
</target>

The most important part of the build file is the target build-workspace, which calls the task workspaceBuild to perform a full build. By default, this task will fail the build if any (compiler) errors are encountered – this is what we want.

<target name="build-workspace"
	depends="setup-workspace,copy-projects,import-projects"
	description="Builds the current workspace">

	<!-- Fully build the workspace -->
	<workspaceBuild
		BuildType="Full"/>
	<echo level="verbose" message="workspaceBuild done"/>
</target>

Hopefully, there are no errors, and we are in a situation where all the projects have been built successfully. Time to generate some artifacts!
The target export-ear first updates the (generated) manifest file with a few Jenkins parameters, such as build number, SVN revision, job name, and the current date. This data is a useful (extra) aid to identify the version / origin of deployed code (please note that you can also use the fingerprinting functionality for this, see below).

We then call the earExport task to create a .ear file, identical to choosing “Export” -> “EAR file” within RAD.

<target name="export-ear"
	description="Exports the EAR defined by the ear.project.name/ear.filename properties">

	<property name="ear.filename" value="${workspace}${ear.project.name}-${env.BUILD_NUMBER}-${env.BUILD_ID}.ear"/>

	<!-- Update the manifest with Jenkins build info -->
	<echo>Updating manifest</echo>
	<tstamp>
		<format property="TODAY" pattern="yyyy-MM-dd HH:mm:ss"/>
	</tstamp>
	<manifest
		file="${workspace}${ear.project.name}/META-INF/MANIFEST.MF"
		mode="update">
		<attribute name="Built-By" value="Jenkins CI"/>
		<attribute name="Implementation-Version" value="#${env.BUILD_NUMBER} - r${env.SVN_REVISION} - ${env.BUILD_ID}"/>
		<attribute name="Implementation-Title" value="${env.JOB_NAME}"/>
		<attribute name="Built-Date" value="${TODAY}"/>
	</manifest>

	<!-- Debug information -->
	<echo level="verbose" message="ear.filename=${ear.filename}"/>
	<echo level="verbose" message="ear.project.name=${ear.project.name}"/>

	<!-- Export the EAR project as an EAR file -->
	<earExport
		EARProjectName="${ear.project.name}"
		EARExportFile="${ear.filename}"
		ExportSource="false"
		IncludeProjectMetaFiles="false"
		Overwrite="true"/>
	<echo level="verbose" message="earExport ${ear.filename} done"/>
</target>

When the RAD builder finishes succesfully, the build part of the job is completed and a number of artifacts (.ear files) will have been generated.

Build job - post-build actions

In the post-build actions we make sure the generated artifacts are scooped up and archived. This makes sure that artifacts are kept even if the original build was (re)moved. Additionally, we enable the recording of fingerprints on each artifact. In essence, this will calculate and store a hash value (MD5 or similar) based on the contents of each file. Should we need to identify a particular artifact at some point in the future, we can simply upload that file to Jenkins, let it calculate a hash value, and match that hash value against its internal fingerprint database. If there’s a match, Jenkins will tell us the job name, build number, date, and any other useful information.

Finally, we call the deploy job using the parameterized trigger plugin. In this case, we do not override any of the default parameters (see below). Should you want to, click “Add parameter”, then “Predefined parameters”. Enter the parameters (key=value pairs) in the text area.

Deploy job

As stated before, the deployment job copies generated artifacts from the build job, and installs the artifacts on a (test/staging) WebSphere server. To achieve this, the job calls the wsadmin tool and executes a single JACL script.

An important part of this job are the predefined parameters, telling the JACL script which SOAP connection to use, and which node / cell / server name / virtual host to install the application to. In this case, each of these parameters has a default value – pointing to a default (local) testing server.

Deploy Job - Build parameters

The build phase of the job consists of three separate build steps:

• Remove any artifacts that were left by previous builds
• Copy the artifacts generated by the last successful run of the build job
• Execute ws_ant (Ant with WebSphere functionality/classes preloaded), which in turn uses wsadmin to run a JACL script.

Deploy Job - Build steps

The JACL script has two modes of operation. First, it stops and uninstalls the previous version of the application we are trying to install. Errors that occur during this first part are ignored.

set appManager [$AdminControl queryNames cell=$cell,node=$node,type=ApplicationManager,process=$server,*]

catch { $AdminControl invoke $appManager stopApplication $appname } result

$AdminConfig save

$AdminApp uninstall $appname

$AdminConfig save

In the second part of the script, the application is installed on the specified node/cell/server/virtual host. Then, after giving the application server some time to process the installed artifact, the script starts the application. If this completes without errors the application is ready to use!

$AdminApp install "$workspace/$earfile" "-node $node -cell $cell -server $server -verbose -defaultbinding.virtual.host $vhost -usedefaultbindings"

$AdminConfig save

set ready false
set retries 0

while {$retries < 20} {
	incr retries
	set ready [$AdminApp isAppReady $appname]
	puts "AdminApp isAppReady: $ready ($retries)"

	if {$ready} { break }

	sleepDelay 5
}

set appManager [$AdminControl queryNames node=$node,cell=$cell,type=ApplicationManager,process=$server,*]

$AdminControl invoke $appManager startApplication $appname

$AdminConfig save

Conclusion / thoughts

In this post you’ve seen how to use Jenkins CI to build (through IBM RAD) and deploy (through IBM wsadmin) a J2EE application to a WebSphere server. I hope these exampless can serve as a starting point for your forays into the exciting world of Jenkins CI.

Comments and suggestions are very welcome!

Downloads

• WebSphere Demo Files

MODx is a very popular PHP/MySQL content management system – no surprise when you look at its flexibility, light weight and ease of use. However, as any MODx developer will attest, it is not without its quirks.

One of those problems is the lack of (automated) support for maintaining separate development and staging or production environments, and migrations between those environments. MODx saves content, templates and code snippets in its database, without versioning. Even though the Revolution branch (MODx has recently launched the first release candidate) aims to solve this by introducing the Transport Package concept, many developers will still need to support installations that run on MODx Evolution.

The code featured in this blog post is a mix of a Phing task that is executed on the development machine, and a small script that is uploaded to the staging or production machine. The task uses the MODx manager log to detect the changes made since a particular date. It then tries to match those changes to the database on the staging/production machines, and collates the changes to a SQL file.

In my case, I usually can’t access a production database from my development machine – this is where the updserver script comes in. It returns mysqldump-style output to the Phing task, and should be uploaded to your staging / production machine and installed in the same directory MODx resides in.

The syntax of the task call in the build file is as follows:

<?xml version="1.0"?>
<project name="MODx update" default="update">
        <target name="update">
            <taskdef name="modxchanged" classname="ModxChangedTask"/>
            <modxchanged
                hostname="localhost" username="root"
                password="test123" database="modx123"
                remoteurl="http://localhost/modx/" remotekey="test123"
                prefix="" timestamp="${timestamp}"/>
        </target>
</project>

In this example, MODx is installed on localhost/modx/. The remote key is the same as the remote database password.

Running the task should result in an SQL file that can be executed on the staging/production machine, and output similar to the following:

MODx update > update:

Searching changes from 1970-01-01:

===== Templates =======================================

NONE

===== Documents =======================================

[ 10] Test                          2010-03-24 19:58:41 [NEW]

===== Chunks ==========================================

NONE

===== Snippets ========================================

NONE

BUILD FINISHED

The code in this post is very much a work-in-progress, so there are a few caveats / unimplemented features:

• Does not correctly handle conflicting/shared id’s
• Does not process deleted items
• Checking (and manually adjusting) the generated SQL file is always a good idea

Any comments or questions are highly appreciated! You can also e-mail me directly at info AT touchdownconsulting.nl!

DOWNLOAD (modx-migration-286.tgz)