Your Go App Can Run Docker Containers, No Fancy Orchestrators Needed

Go is the one of the key languages driving cloud computing today. (Who knows what the future holds.)

One on hand, Go has a broad set of building blocks for managing much of the activity we do on the cloud. Built-in libraries for context handling, HTTP requests, building in-memory stores and many other modern needs. On the other hand, it means any Go application shares the same DNA as much of the tooling we are using to build and manage the cloud today. Things like Kubernetes and Docker, Terraform and Caddy server.

Very sophisticated projects are built using Go. But it doesn’t take a large, sophisticated effort to create an application that is just as flexible, portable and reliable as the ones from the big companies.

Let’s take containers, for example. Containers are one of the key elements that have allowed cloud-based computing to take off, and it has helped eased the development and deployment process for so many of us. Containers are the lingua franca of Kubernetes, sure, but we don’t need a complex orchestration tool like that to manage some containers. We can do the same with an application we write in Go ourselves.

Adding Containers to A Go App

This Go client for Docker allows us to communicate with the daemon, to start containers, manage containers, process stdout and logs, and shut down containers. If you already have a CLI or web application running in Go, it’s a small task to extend it with a container process. We’ll show some code below.

Official Docker example using the SDK.

But … why would we want to do this? Well, imagine you have a program that runs fine inside a container, maybe something custom or not. One option is to rewrite the program in Go. Another is to add a container orchestrator to run that single image. A third — scary option — is to write our own container orchestrator to run the image. Eek!

A fourth option, to run that program with the least tooling possible, is the one I prefer. Now, you may say, “I already use bash or some other tool to do this. Why stop now?!"

If this is you speaking, then keep at it. You probably have a successful career in operations already!

Example Use Cases

Let’s consider a few use cases where running a containerized program in our app would be helpful, and require far less effort than rewriting the original program.

Data processing/ETL – Go is a great tool for many programming tasks, but not all. One place it cause a fair bit of confusion is transforming data structures in an out of JSON or other basic formats. For some folks used to languages with dynamic types, Go is unforgiving and opaque. Fair point. What if we used some other language to perform the transformation and pass that off to Go?

Wrap the control mechanism – There are lots of tutorials about writing a basic CLI application or web app in Go. It takes very little effort. What if we had a containerized program that would be great to run inside that application? When a user enters a specific CLI action, we build the container, run it and process the output, for example. Or the web app receives an HTTP request on a given route, where we run that image and return a success or error message.

Manage the process environment – Many applications have a set of environment parameters or secrets, as well as custom logic for connecting to a database, or processing logs and other metrics. While it’s easy for our Go app to run a container image, it may not be easy to pass all of that context into the container, without rebuilding it ourselves. In one example, our app can perform some runtime logic to generate a dynamic set of parameters, which are passed as environment variables to the image. In another example, the container’s stdout and stderror channels are processed through our app, which can then write certain data to the database, or send metrics and logs on its existing connections.

What This Does Not Do

This approach is intended to improve your productivity without a lot of added complexity. (I realize that depends on the context.) So it’s worth noting what this does NOT do, alongside what it does:

You don’t have to rewrite a program in Go or find a new way to manage the dependencies and run it. Let’s put the program and its dependencies in a Docker image and use that.

You don’t need to run Kubernetes or any other container service beyond Docker engine.

You don’t need to invent anything like a new container manager system!

Show Me The Code

This simple program creates a container from a local image, runs it, and removes it when the workload is complete.

package main
 
import (
  "context"
  "fmt"
  "log"
 
  "github.com/docker/docker/api/types"
  "github.com/docker/docker/api/types/container"
  "github.com/docker/docker/client"
  "github.com/docker/go-connections/nat"
  v1 "github.com/opencontainers/image-spec/specs-go/v1"
)
 
func main() {
  // 1. Set-up for container.
  image := "my-custom-container"
  contName := "myCustomContainer"
  ctx := context.Background()
  startDate := "2022-10-19"
 
  cli, err := client.NewClientWithOpts(client.FromEnv)
  if err != nil {
    panic(err)
  }
 
  hostBinding := nat.PortBinding{
    HostIP:   "0.0.0.0",
    HostPort: "8000",
  }
  containerPort, err := nat.NewPort("tcp", "80")
  if err != nil {
    panic("Unable to get the port")
  }
 
  portBinding := nat.PortMap{containerPort: []nat.PortBinding{hostBinding}}
  platform := v1.Platform{OS: "linux", Architecture: "amd64"}
 
  // 2. Create container.
  cont, err := cli.ContainerCreate(
    ctx,
    &container.Config{
      Image: image,
      Tty:   true,
      Env:   []string{fmt.Sprintf("START_DATE=%s", startDate)},
    },
    &container.HostConfig{
      PortBindings: portBinding,
    },
    nil,
    &platform,
    contName,
  )
  if err != nil {
    panic(err)
  }
 
  // 3. Run container.
  err = cli.ContainerStart(ctx, cont.ID, types.ContainerStartOptions{})
  if err != nil {
    panic(err)
  }
  contJSON, err := cli.ContainerInspect(ctx, cont.ID)
  if err != nil {
    panic(err)
  }
  log.Printf("Container starting: %s\n", contJSON.Name)
 
  statusCh, errCh := cli.ContainerWait(ctx, cont.ID, container.WaitConditionNotRunning)
  select {
  case err := <-errCh:
    if err != nil {
      panic(err)
    }
  case <-statusCh:
    // fmt.Println("Container is ok")
  }
 
  // 4. Attach to container logs.
  out, err := cli.ContainerLogs(ctx, cont.ID, types.ContainerLogsOptions{
    ShowStdout: true,
    ShowStderr: false,
    Follow:     true,
  })
  if err != nil {
    panic(err)
  }
 
  defer out.Close()
 
  // 5. Do some work ...
  // @todo: process stdout, wait for work outputs, ...
 
  // 6. Clean up.
  err = cli.ContainerStop(ctx, cont.ID, nil)
  if err != nil {
    panic(err)
  }
  err = cli.ContainerRemove(ctx, cont.ID, types.ContainerRemoveOptions{})
  if err != nil {
    panic(err)
  }
}

Let’s break this down.

Step 1 – Provide some set-up data, such as the image name, container name, and any specific environment variables we will pass to the container. We can also provide custom network configuration for the container to run properly. (It’s also possible to pull the image at this stage.)

Step 2 – Now we create the container, passing in the environment variables and any other custom settings. Here we pass Tty: true to help with attaching to the container’s stdout. The SDK docs have more info on available parameters.

Step 3 – Run the container, and grab the container name and any other metadata if needed. The ContainerWait method allows us to check any errors during start-up and respond appropriately.

Step 4 – The logs option allows us to read from stdout, stderr or both. See below for more details.

Step 5 – Add some custom code for working with the container, such as reading logs, or waiting for artifacts to be generated.

Step 6 – Stop the container and, if needed, delete it.

Handling container logs and stdout, stderr

Reading the log stream from the container is a bit tricky, as the metadata is embedded with the data we want to process. Here’s an excellent breakdown on the metadata format, which is contained in the first eight bytes of each log message.

The author of that breakdown has a library to parse the logs, which helps you filter stdout from stderr.

Or you can try to parse those eight bytes yourself, as shown here.

Another simple example with stdout and stderr.

Note: our example here passes Tty: true during ContainerCreate which removes those bits from the stream.

If the container is using a TTY, there is only a single stream (stdout), and data is copied directly from the container output stream, no extra multiplexing or headers.

Project Ideas

Hopefully this provides a good understanding of how running Docker images in a Go application can help make you more productive without requiring a lot of extra work or learning a new domain. If you can’t think of a current project in your work-life where this would be useful, that’s OK. Keep it in mind for the future. Here are some other ideas where it could come in handy.

Perform independent jobs – Docker images are commonly used to run a single operation or multiple, and then stop. Some examples include performing database operations, transferring data, or generate a report. Again, we can pass those connection details or datbase credentials to our container, as needed.

Produce artifacts – The container has access to the local filesystem where we can write files. The containerized program can output artifacts, such as CSV files, images, or other binaries we create.

Consume real-time outputs – Docker containers communicate via stdout and stderr channels, like many other systems we use. By tapping into those streams from the container, a Go program can filter that information and relay it to another system or process. This is similar to a typical ETL pattern, where the Dockerized program does the extract and transform portions, and Go does the load.