Non-Java Binary Dependencies in Maven

Suppose you have a Java Server application, and some of the runtime binaries in that application are external to your application. Generated image files, compiled Silverlight components in your pages, or resource files which are managed by an external team.

Much like the jar files used by your application, these external binaries can be seen as dependencies, with versions. This blogpost assumes your project is built with Maven 2, because the real world isn’t always a greenfield project.

Because Maven is designed around jar file dependencies, and a lot of it’s internal decisions are based on file extensions, it looks like this problem can not be tackled with Maven. But there is a way to do this. It will decouple your sub-projects and make version and dependency management much better.

Uploading the binaries files in a remote repository
The hardest part of this problem is getting the binary uploaded to your maven repository. It seems that there is no other way than disquising it as a jar file, which only leads to confusion. But it turns out that you can, actually upload a .exe file to the repository. To do so, place a pom.xml file in the root of the project from which you want to upload the binary file. You do not have to mavenize the whole project, we only use Maven to do the upload to the repository.

Find the file you want to deploy. It can be placed anywhere but I prefer to use the target directory so it looks maven-esque. Make sure you choose the “pom” packaging type so maven does not generate a jar for you.

<project>
  <groupId>com.rolfje.example</groupId>
  <artifactId>executablestuff</artifactId>
  <packaging>pom</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>Some executable component</name>
  <build>
  <plugins>
    <plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <version>1.8</version>
    <executions>
      <execution>
      <id>attach-artifacts</id>
      <phase>package</phase>
      <goals>
      <goal>attach-artifact</goal>
      </goals>
      <configuration>
      <artifacts>
        <artifact>
        <file>${basedir}/target/myexecutable.exe</file>
        <type>exe</type>
        </artifact>
      </artifacts>
      </configuration>
      </execution>
    </executions>
    </plugin>
  </plugins>
  </build>
</project>

When you run “maven install”, your exe file will be pushed to your local repository (more on using a remote repository further down).

Downloading the binaries files from the remote repository
In the project that will depend on your binary file, you want to copy it from your repository into your target directory at the compile stage. You can do this by adding the following plugin configuration to your pom:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-dependency-plugin</artifactId>
  <executions>
    <execution>
      <id>copy-dependency</id>
      <phase>compile</phase>
      <goals>
        <goal>copy</goal>
      </goals>
      <configuration>
        <artifactItems>
          <artifactItem>
            <groupId>com.rolfje.example</groupId>
            <artifactId>executablestuff</artifactId>
            <type>exe</type>
            <overWrite>true</overWrite>
            <outputDirectory>target</outputDirectory>
          </artifactItem>
        </artifactItems>
      </configuration>
    </execution>
  </executions>
</plugin>

Multiple artifacts in a single dependency
When the non-java project produces multiple files with the same extension, you will notice that the trick I just described will not work because you can only specify one file with one extension. This is because maven uses the extension to locate the exact file.

In order to deploy multiple files, it is best you bundle them in a zip and unpack them at the other end. In the non-java project, add the assembly plugin:

<plugin>
  <artifactId>maven-assembly-plugin</artifactId>
  <version>2.2-beta-2</version>
  <executions>
  <execution>
    <id>generate-assembly</id>
    <phase>package</phase>
    <goals>
    <goal>single</goal>
    </goals>
  </execution>
  </executions>
  <configuration>
  <descriptors>
    <descriptor>src/maven/descriptor.xml</descriptor>
  </descriptors>
  <finalName>${artifactId}</finalName>
  <outputDirectory>target</outputDirectory>
  </configuration>
</plugin>

We use so the zipfile will always have the same name in the target directory, which will make build-helper-maven-plugin configuration easier. It also makes it easier for non-maven scripts to find the file. Upon deploying to the repository, maven will fix the filename so don’t worry about that too much.

The contents of src/maven/descriptor.xml can look like this:

<?xml version="1.0" encoding="UTF-8"?>

<assembly xmlns="http://maven.apache.org/POM/4.0.0"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
  http://maven.apache.org/xsd/assembly-1.0.0.xsd"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <formats>
  <format>zip</format>
  </formats>

  <includeBaseDirectory>false</includeBaseDirectory>

  <fileSets>
  <fileSet>
    <directory>target/binaries</directory>
    <outputDirectory></outputDirectory>
    <includes>
    <include>*.exe</include>
    <include>*.dll</include>
    </includes>
  </fileSet>
  </fileSets>
</assembly>

In the java project, you don’t need to change much. Instead of “copy”, you tell the dependency plugin to unzip the file into a directory:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-dependency-plugin</artifactId>
  <executions>
  <execution>
    <id>copy-dependency</id>
    <phase>compile</phase>
    <goals>
    <goal>unpack</goal>
    </goals>
    <configuration>
    <artifactItems>
      <artifactItem>
      <groupId>com.rolfje.example</groupId>
      <artifactId>executablestuff</artifactId>
      <type>zip</type>
      <overWrite>true</overWrite>
      <outputDirectory>target/unpacked</outputDirectory>
      </artifactItem>
    </artifactItems>
    </configuration>
  </execution>
  </executions>
</plugin>

Deploying to a remote repository
If the two builds run on two different machines, you may need an external repository. For larger projects, I’d recommend an external, locally managed repository not just for disctribution, but also for speed. Your team will have almost instant access to libraries, speeding up your (initial) builds. Installing a local jFrog Artifactory is a good choice.

To make deployment to this external repository work, you need to add the following to your non-java pom file:

<distributionManagement>
  <!-- final releases will be deployed to this repository -->
  <repository>
  <id>artifactory.releases</id>
  <name>Artifactory releases</name>
  <url>http://myartifactory.example.com/artifactory/libs-release-local</url>
  </repository>

  <!-- snapshot releases will be deployed to this repository -->
  <snapshotRepository>
  <id>artifactory.snapshots</id>
  <name>Artifactory snapshots</name>
  <url>http://myartifactory.example.com/artifactory/libs-snapshots-local</url>
  </snapshotRepository>
</distributionManagement>

If you get an error like:

Error deploying artifact: Failed to transfer file: ://myartifactory.example.com/artifactory/libs-snapshots-local/com.rolfje.example/1.0-SNAPSHOT/executablestuff-1.0-20130604.103146-1.pom. Return code is: 401

Your artifactory is probably not configured to do anonymous deployments. Make sure you can access the artifactory with a userid and password, and add those to the ~/.m2/settings.xml of the machine which will deploy the binary artifacts.

I advise to use a special “deployment user” to do this, so you can share this configuration between builds and not depend on a user changig his password. The correct way to store a password for the Artifactory is explained here.  Using the DESede encrypted password on your Artifactory profile page, the maven settings.xml file of the buildserver user can look like this:

<settings>
  <server>
  <id>artifactory.releases</id>
  <username>your-repository-username</username>
  <password>\{DESede\}kIniw826kaluA1OPa865A==</password> 
  </server>
  <server>
  <id>artifactory.snapshots</id>
  <username>your-repository-username</username>
  <password>\{DESede\}kIniw826kaluA1OPa865A==</password>
  </server>
</settings>

And then, of course, you need your java project to know where to download the dependencies so add this to the pom.xml file in your binary project:

<repositories>
  <repository>
  <id>central</id>
  <url>http://myartifactory.example.com/repo</url>
  <releases>
    <enabled>true</enabled>
    <checksumPolicy>fail</checksumPolicy>
  </releases>
  <snapshots>
    <enabled>true</enabled>
    <checksumPolicy>fail</checksumPolicy>
  </snapshots>
  </repository>
</repositories>

If you’ve reached the end of this blogpost without problems, you should now be able to build your binaries, have them uploaded with correct versions to your external repository, and your Maven/Java build server will download the correct versions of these dependencies at build time. Switching back and forth between tags on your Java project will automatically fetch the correct versions of the binaries, the same way as it does for your jar dependencies.

Happy coding!