Way to Go: Combining Go and Gradle Plugins

My team and I have recently started working on a new project where we heavily leverage the Kubernetes API. Go was chosen as our primary technology. It seemed like a good (natural) choice due to the nature of the project. It’s a smaller plugin used as an extension to an existing product. The Kubernetes Go client library was a perfect fit for our needs.

Go has been around for a longer time. It was first released in 2012. Although it’s nice to develop using it, it introduces some challenges when combined with not-so-golang-based build tools.

Yet, Go still doesn’t have a dominant tool for build automation which leaves some pitfalls when trying to build a project with it.

And then also… product failure

There were a couple of requirements for building the project because of the integration part:

1. To enable the pluggability of our repository with the current product, a jar file needs to be generated. In our case, there is no Java code in the jar file. Only simple XML resources are required by the main product to recognize the plugin.
2. The Docker image must be built with a Go executable as an entry point. Due to specific project requirements, the main product will interact with the plugin by providing some input, spinning up a container where the Go executable is executed, doing some operations with Kubernetes API, and then returning the output to the main product.

The outcome of the project build should be a jar file that makes the plugin “pluggable” and a docker image, with the Go executable as an entrypoint, pushed to some image registry.

Everything in the product and plugins is built and automated with the Gradle build tool. So naturally, Gradle was also our choice. Building, testing, generating images, and publishing them should be automated with the help of Gradle.

Generating images, deploying, and publishing them are all standard Gradle stuff. But how can the compilation of the Go source code be automated using Gradle?

What are the options?

In the examples below, three different ways will show how to combine Gradle and Go:

• using the existing GoGradle plugin,
• using Make, and
• using Mage.

What will be shown is how to automate the compilation of Go code and the building of Docker images using Gradle as the build tool.

The logic of the Go code won’t do much. It will connect to the existing Kubernetes cluster and print all namespaces inside the cluster using the Kubernetes Go client library. Also, Go modules will be used for dependency management since it’s the standard way of managing Go dependencies.

Modules, code, Dockerfile, and image generation will be identical in all projects, the only difference will be the way Go code is compiled.

Let’s introduce identical stuff first.

Modules

At the root of each subproject, a Go module file is created. Each Go module is defined by a go.mod file that describes its properties, including its dependencies on other modules and versions of Go.

module github.com/mkeretic/hello-go-gradle
go 1.18
require (
	k8s.io/apimachinery v0.25.3
	k8s.io/client-go v0.25.3
)

In the go.mod file the name of the module, the Go version, and direct dependencies to some libraries that will be used are defined.

Code

At the root of each subproject, a main.go file is created, where the business logic lives.

func main() {
	clientSet, err := k8s.GetClientSet()
	if err != nil {
			k8s.HandleError(err)
	}
	namespaces, err := k8s.ListNamespaces(clientSet)
	if err != nil {
			k8s.HandleError(err)
	}
	fmt.Println("Namespaces: ", namespaces)
}

The code connects to the Kubernetes cluster, fetches and prints all namespaces. The whole code can be found on GitHub, but it is now less important.

Dockerfile

Dockerfile will be used to generate the images.

FROM alpine:3.14
COPY hello-go-gradle /
ENTRYPOINT ["/hello-go-gradle"]

Starting from the base alpine image, the Go executable is copied and set to execute at the entry point.

build.gradle

At the root of the project the main build.gradle file is created. This will define how all subprojects build and push docker images. This is also identical for all subprojects.

subprojects { sp ->
  apply plugin: "java"
  apply plugin: "com.bmuschko.docker-remote-api"
  targetCompatibility = JavaVersion.VERSION_11
  task copyPluginBinary(type: Copy) {
    dependsOn('goBuild')
    from "${project.buildDir}/plugin/linux-amd64"
    into "${project.buildDir}/docker/"
  }
  task copyDockerFile(type: Copy) {
    from "${projectDir}/Dockerfile"
    into "${project.buildDir}/docker/"
  }
  task buildImage(type: DockerBuildImage) {
    dependsOn(copyPluginBinary, copyDockerFile)
    images.add("${registryOrg}/${project.name}:${project.version}")
  }
  task pushImage(type: DockerPushImage) {
    dependsOn(buildImage)
    images = 
["${registryUrl}/${registryOrg}/${project.name}:${project.version}"]
  }
  build.dependsOn 'buildImage'
}

First, the Go executable and Dockerfile are copied (the generation of Go executable will be shown in the sections below) and put into the directory from which the image will be built. Then the image is built and pushed to the local registry using gradle-docker-plugin and its DockerBuildImage and DockerPushImage tasks.

Compiling Go

The only thing left to do in the project is to show how to compile the Go code to put it inside the image. Three different subprojects will be created. Each shows a different way to automate Go code compilation using GoGradle, Make, and Mage. One thing to remember is that it should be possible to compile Go code for different platforms.

GoGradle

The first thought that comes to mind when using Gradle is to take an already existing Gradle plugin and make it do the heavy lifting.

GoGradle is the most popular one with more than 700 stars on GitHub, with the second most popular plugin not even reaching 50 stars. So the obvious choice is GoGradle.

A hello-go-gradle subproject is created, along with a build.gradle file, which is placed at the root of the subproject.

plugins {
  id "com.github.blindpirate.gogradle" version "$goGradlePluginVersion"
}
golang {
  // go import path of project to be built, NOT local file system path!
  packagePath = 'github.com/mkeretic/hello-go-gradle' 
}
goBuild {
  targetPlatform = [
      'darwin-amd64', 'linux-amd64', 'windows-amd64'
  ]
  // The ${} placeholder will be rendered in cross-compile
  go "build -o 
${project.buildDir}/plugin/\${GOOS}-\${GOARCH}/\${PROJECT_NAME}\${GOEXE}"
}

In build.gradle, the GoGradle plugin is imported at the top and then its tasks are used to compile Go code. GoGradle plugin provides two Gradle tasks that can be used, golang and goBuild. Golang task is used to define the Go import path of the project to be built. GoBuild task is used to define for which platforms the Go executables should be built and where they should be outputted.

With everything set up correctly, the hello-go-gradle subproject can be built with ./gradlew :hello-go-gradle:build command.

Unfortunately, the build fails

Task resolveBuildDependencies is part of the GoGradle plugin and it is executed before goBuild task. Its job is to resolve app dependency packages and their transitive dependencies. Taking a deeper look into the GoGradle project on GitHub, it can be seen that the GoGradle project has been archived for a few years. Many people report having dependency hell issues when using newer Go packaged with GoGradle.

In the hello-go-gradle subproject case, that is the Kubernetes Go client package. GoGradle project was left unmaintained and without proper support when Go modules were introduced as the default way to manage Go libraries.

Even though GoGradle might work with older packages, and the resolve BuildDependencies task will be able to resolve all build dependencies, it is not a good long-term solution for project builds.

Currently, none of the Go Gradle plugins is suitable. There must be a different solution without using any of the Gradle plugins.

What is Make used for?

Why not just use the good old Make and Makefile? Make is an incredibly useful automation tool for running and building not just Go applications, but for most programming languages. It is a very popular choice for automating tasks in the Go community and seems to do the job well.

A new subproject called hello-make is created and is at its root. The simplest possible makefile is placed, containing a simple task to run the “go build” command.

Step-by-step guide

build:
	go build

If the “make build” command is run now, the “hello-make” Go executable will be generated in the same folder.

This works well for a simple build but should be expanded so it can be compiled for different platforms and with a custom output directory.

The desired result is to combine Make with Gradle by defining a task to build Go code in Makefile, and executing that task using Gradle Exec tasks.

build:
	env GOOS=${GOOS} GOARCH=${GOARCH} CGO_ENABLED=0 go build -o $
      {PROJECT_BUILD_DIR}/${GOOS}-${GOARCH}/${PROJECT_NAME}

For the build task, the possibility to take different platforms as input is added. Also, the custom output directory is provided in which the generated Go executable should be placed.

Then, in the root of the hello-make subproject, a build.gradle file is created in which the tasks to compile the Go code are defined.

task goBuild {
  def targetPlatform = [
      'darwin-amd64','linux-amd64', 'windows-amd64'
  ]
  doLast {
    targetPlatform.each { platform ->
      def (os, arch) = extractOsAndArch(platform)
      exec {
        workingDir project.projectDir
        commandLine "make", "build", "GOOS=${os}", "GOARCH=${arch}", 
"PROJECT_NAME=${project.name}", 
"PROJECT_BUILD_DIR=${project.buildDir}/plugin"
      }
    }
  }
}

Same as in GoGradle plugin, as the first step, the target platforms are defined for which the Go executables should be generated. Then, an Exec task is created for each platform. In Gradle, Exec task is a task that executes a command line process. In the hello-go-make subproject that is the “make build” command for all defined platforms.

With everything set up correctly, the project should now be buildable with ./gradlew :hello-make:build command.


The build is green! Go code is successfully compiled. An image with the Go executable as an entry point is successfully generated and pushed to the local registry. The container can now run test of the generated image.

Everything has been executed successfully and all namespaces are printed, which is the desired result.

Using Make for compiling Go code is a good solution and works nicely in most cases, but there is another option to explore, a more Go native approach.

Mage

The third tool that can be used is Mage. Mage is a make/rake-like build tool using Go. Developers write plain-old Go functions, and Mage automatically uses them as Makefile-like runnable targets. Mage can be installed following the official documentation.

After Mage is installed, a new subproject called hello-mage can be created with a magefile.go placed at the root of the subproject… In this file, the build tasks will be defined.

//go:build mage
package main
import (
	"fmt"
	"github.com/magefile/mage/sh"
)
func Build(os string, arch string, projectBuildDir string, projectName string) error {
	goos := fmt.Sprintf("GOOS=%s", os)
	goarch := fmt.Sprintf("GOARCH=%s", arch)
	outputLocation := fmt.Sprintf("%s/%s-%s/%s", projectBuildDir, os, arch, projectName)
	fmt.Println(outputLocation)
	return sh.Run("env", goos, goarch, "CGO_ENABLED=0", "go", "build", "-o", outputLocation)
}

What is the Mage file?

A mage file is any regular Go file marked with a “mage” build target and located in the main package. The Build function is defined, which takes the following and executes a Go build command in the terminal (similar to the makefile build task):

• target operating system
• target architecture
• a project build directory,
• and project name parameters as input

Then, in build.gradle, the goBuild task can be defined.

task goBuild {
  def targetPlatform = [
      'darwin-amd64','linux-amd64', 'windows-amd64'
  ]
  doLast {
    targetPlatform.each { platform ->
      def (os, arch) = extractOsAndArch(platform)
      exec {
        workingDir project.projectDir
        commandLine "mage", "build", os, arch, 
"${project.buildDir}/plugin", project.name
      }
    }
  }
}

It is also similar to the build.gradle file for Make build. The only difference is that the command “mage” is executed instead of “make” and arguments are passed to magefile.go Build function in a tad different way.

After goBuild task is defined, then the project should be buildable with the
./gradlew :hello-make:build command.

The output is identical to the one achieved with Make build, which is exactly the desired result. The Go code is successfully compiled, and the docker image with the Go executable as an entry point is generated and pushed to the local registry. The container should now be runnable.

It works as expected, and all namespaces in the cluster are printed to the terminal. Mage also works nicely for this use case and is a viable option to choose when automating builds with Go.

“Make” as our choice

Go programming language doesn’t have a dominant tool for automation which presents some challenges when trying to combine it with a popular automation tool such as Gradle.

Some options that can be used to combine Gradle and Go were iterated through. However, no existing Gradle plugin was suitable, so another solution had to be found.

Both Make and Mage are good options for automating Go tasks, and both do the job well. It is up to preference which you want to choose for your project.

My Notch team and I eventually chose Make as our build tool since we found it a more mature and widespread tool with plenty of available documentation covering any use cases.