Non-Java Binary Dependencies in Maven

binary-pillowSuppose you have a Java Server application, and some of the runtime binaries in that application are external to your application. Generated image files, compiled Silverlight components in your pages, or resource files which are managed by an external team.

Much like the jar files used by your application, these external binaries can be seen as dependencies, with versions. This blogpost assumes your project is built with Maven 2, because the real world isn’t always a greenfield project.

Because Maven is designed around jar file dependencies, and a lot of it’s internal decisions are based on file extensions, it looks like this problem can not be tackled with Maven. But there is a way to do this. It will decouple your sub-projects and make version and dependency management much better.

Uploading the binaries files in a remote repository
The hardest part of this problem is getting the binary uploaded to your maven repository. It seems that there is no other way than disquising it as a jar file, which only leads to confusion. But it turns out that you can, actually upload a .exe file to the repository. To do so, place a pom.xml file in the root of the project from which you want to upload the binary file. You do not have to mavenize the whole project, we only use Maven to do the upload to the repository.

Find the file you want to deploy. It can be placed anywhere but I prefer to use the target directory so it looks maven-esque. Make sure you choose the “pom” packaging type so maven does not generate a jar for you.

<project>
  <groupId>com.rolfje.example</groupId>
  <artifactId>executablestuff</artifactId>
  <packaging>pom</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>Some executable component</name>
  <build>
  <plugins>
    <plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <version>1.8</version>
    <executions>
      <execution>
      <id>attach-artifacts</id>
      <phase>package</phase>
      <goals>
      <goal>attach-artifact</goal>
      </goals>
      <configuration>
      <artifacts>
        <artifact>
        <file>${basedir}/target/myexecutable.exe</file>
        <type>exe</type>
        </artifact>
      </artifacts>
      </configuration>
      </execution>
    </executions>
    </plugin>
  </plugins>
  </build>
</project>

When you run “maven install”, your exe file will be pushed to your local repository (more on using a remote repository further down).

Downloading the binaries files from the remote repository
In the project that will depend on your binary file, you want to copy it from your repository into your target directory at the compile stage. You can do this by adding the following plugin configuration to your pom:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-dependency-plugin</artifactId>
  <executions>
    <execution>
      <id>copy-dependency</id>
      <phase>compile</phase>
      <goals>
        <goal>copy</goal>
      </goals>
      <configuration>
        <artifactItems>
          <artifactItem>
            <groupId>com.rolfje.example</groupId>
            <artifactId>executablestuff</artifactId>
            <type>exe</type>
            <overWrite>true</overWrite>
            <outputDirectory>target</outputDirectory>
          </artifactItem>
        </artifactItems>
      </configuration>
    </execution>
  </executions>
</plugin>

Multiple artifacts in a single dependency
When the non-java project produces multiple files with the same extension, you will notice that the trick I just described will not work because you can only specify one file with one extension. This is because maven uses the extension to locate the exact file.

In order to deploy multiple files, it is best you bundle them in a zip and unpack them at the other end. In the non-java project, add the assembly plugin:

<plugin>
  <artifactId>maven-assembly-plugin</artifactId>
  <version>2.2-beta-2</version>
  <executions>
  <execution>
    <id>generate-assembly</id>
    <phase>package</phase>
    <goals>
    <goal>single</goal>
    </goals>
  </execution>
  </executions>
  <configuration>
  <descriptors>
    <descriptor>src/maven/descriptor.xml</descriptor>
  </descriptors>
  <finalName>${artifactId}</finalName>
  <outputDirectory>target</outputDirectory>
  </configuration>
</plugin>

We use so the zipfile will always have the same name in the target directory, which will make build-helper-maven-plugin configuration easier. It also makes it easier for non-maven scripts to find the file. Upon deploying to the repository, maven will fix the filename so don’t worry about that too much.

The contents of src/maven/descriptor.xml can look like this:

<?xml version="1.0" encoding="UTF-8"?>

<assembly xmlns="http://maven.apache.org/POM/4.0.0"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
  http://maven.apache.org/xsd/assembly-1.0.0.xsd"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <formats>
  <format>zip</format>
  </formats>

  <includeBaseDirectory>false</includeBaseDirectory>

  <fileSets>
  <fileSet>
    <directory>target/binaries</directory>
    <outputDirectory></outputDirectory>
    <includes>
    <include>*.exe</include>
    <include>*.dll</include>
    </includes>
  </fileSet>
  </fileSets>
</assembly>

In the java project, you don’t need to change much. Instead of “copy”, you tell the dependency plugin to unzip the file into a directory:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-dependency-plugin</artifactId>
  <executions>
  <execution>
    <id>copy-dependency</id>
    <phase>compile</phase>
    <goals>
    <goal>unpack</goal>
    </goals>
    <configuration>
    <artifactItems>
      <artifactItem>
      <groupId>com.rolfje.example</groupId>
      <artifactId>executablestuff</artifactId>
      <type>zip</type>
      <overWrite>true</overWrite>
      <outputDirectory>target/unpacked</outputDirectory>
      </artifactItem>
    </artifactItems>
    </configuration>
  </execution>
  </executions>
</plugin>

Deploying to a remote repository
If the two builds run on two different machines, you may need an external repository. For larger projects, I’d recommend an external, locally managed repository not just for disctribution, but also for speed. Your team will have almost instant access to libraries, speeding up your (initial) builds. Installing a local jFrog Artifactory is a good choice.

To make deployment to this external repository work, you need to add the following to your non-java pom file:

<distributionManagement>
  <!-- final releases will be deployed to this repository -->
  <repository>
  <id>artifactory.releases</id>
  <name>Artifactory releases</name>
  <url>http://myartifactory.example.com/artifactory/libs-release-local</url>
  </repository>

  <!-- snapshot releases will be deployed to this repository -->
  <snapshotRepository>
  <id>artifactory.snapshots</id>
  <name>Artifactory snapshots</name>
  <url>http://myartifactory.example.com/artifactory/libs-snapshots-local</url>
  </snapshotRepository>
</distributionManagement>

If you get an error like:

Error deploying artifact: Failed to transfer file: ://myartifactory.example.com/artifactory/libs-snapshots-local/com.rolfje.example/1.0-SNAPSHOT/executablestuff-1.0-20130604.103146-1.pom. Return code is: 401

Your artifactory is probably not configured to do anonymous deployments. Make sure you can access the artifactory with a userid and password, and add those to the ~/.m2/settings.xml of the machine which will deploy the binary artifacts.

I advise to use a special “deployment user” to do this, so you can share this configuration between builds and not depend on a user changig his password. The correct way to store a password for the Artifactory is explained here.  Using the DESede encrypted password on your Artifactory profile page, the maven settings.xml file of the buildserver user can look like this:

<settings>
  <server>
  <id>artifactory.releases</id>
  <username>your-repository-username</username>
  <password>\{DESede\}kIniw826kaluA1OPa865A==</password> 
  </server>
  <server>
  <id>artifactory.snapshots</id>
  <username>your-repository-username</username>
  <password>\{DESede\}kIniw826kaluA1OPa865A==</password>
  </server>
</settings>

And then, of course, you need your java project to know where to download the dependencies so add this to the pom.xml file in your binary project:

<repositories>
  <repository>
  <id>central</id>
  <url>http://myartifactory.example.com/repo</url>
  <releases>
    <enabled>true</enabled>
    <checksumPolicy>fail</checksumPolicy>
  </releases>
  <snapshots>
    <enabled>true</enabled>
    <checksumPolicy>fail</checksumPolicy>
  </snapshots>
  </repository>
</repositories>

If you’ve reached the end of this blogpost without problems, you should now be able to build your binaries, have them uploaded with correct versions to your external repository, and your Maven/Java build server will download the correct versions of these dependencies at build time. Switching back and forth between tags on your Java project will automatically fetch the correct versions of the binaries, the same way as it does for your jar dependencies.

Happy coding!

Advertisements

7 Responses to Non-Java Binary Dependencies in Maven

  1. mikko says:

    Thanks for the good article. Is there any way how to tell what binary to download for different operating systems? I would like to download binary A for Windows and binary B for linux.

    BR,

    Mikko

  2. Rashid says:

    Thanks for the article. I’m trying to push a single zip file coming out of a non-maven build to a specific location in Artifactory’s repo structure. But I’m having some issue in the upload. I’m using your pom pretty much verbatim with the distributionManagement block with Artifactory repo info. I’m not seeing any error when I execute ‘mvn install’. But no upload is happening. I’m wondering if we need more stuff like using maven-deploy-plugin. Does the attach-artifacts goal of build-helper-maven-plugin push artifact to remote repo? The documentation on it is extremely bad.

  3. Rashid says:

    Thanks for the quick response. I used maven-deploy-plugin and upload is working nicely!!

  4. Francisco says:

    Great article. However, I’m a bit dissapointed with the solution “Downloading the binaries files from the remote repository”, because the dependency is declared at the plugin configuration, therefore, maven dependency features (like transitive dependencies) are not leveraged. Is there a way to achieve this?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s