More Than One Way to Authenticate Cloud Resources on GCP or AWS

The more we put our resources on cloud platforms of any kind, the more we need to communicate with those resources. This can range from quick tasks using a console in the web browser, to sophisticated build processes using infrastructure as code. There is a large variety of tools available.

As much as possible, we want to use the right tool for the job. Some tools are quick and easy to spin up, making them perfect for a single-use job, for example to grab some metadata stats on a set of virtual machines. Type out some commands in a command-line interface (CLI) and you have what you need. But we’ll want something more consistent and repeatable if we need to create a set of subnets and VMs with custom access policies and rules. That’s when we reach for something like Terraform, a fully featured tool with a bit of a learning curve for sure, but also a broader set of guarantees.

The most common tool that engineers reach for, in my experience, are the CLIs for AWS and GCP platorms. They are quick to install and make a fairly modest requirement to have Python installed on your machine. The big advantage, I think, is how flexible they are for authenticating with a single account, or with one of any number of service accounts.

If you’re unfamiliar with using multiple accounts from a single machine, see:

• AWS named accounts
• Managing gcloud CLI configurations

One downside of CLI tools is that while it feels like writing code, it’s not really. A typical development environment like Code or GoLand offers some quality-of-life improvements like syntax highlighting, code autocompletion and debugging help. A terminal does not. Additionally, a valuable command that I craft in my terminal does not have the same persistence as a program written to a file somewhere. Sure many terminals allow for searching the history, but that doesn’t provide order or context for the results. And watch out if you mistakenly add a space before the command … it will never make it in the history in the first place.

Another downside is how data is returned from CLIs, or more specifically how easy it is to manipulate or parse the data that is returned. In the best cases, we receive JSON which can be piped to something like jq or sed, in order to extract the parts we want. But some times we get plain lines of text, or tab-formatted text which can be quite tricky to parse.

I believe there is another way, a better way!

Software Development Kit (SDK) for AWS and GCP

Let’s write programs, in the language of our choice, to do much of the work that we expect from a CLI. Programs can be composed in an IDE — or not — and easily extended with the addition of modules or libraries. They can be compiled and shared on multiple machines. Their behavior is often reproducible.

Counterpoint: programs can also take time to write. So this won’t replace the quick one-liner you feed to gcloud. But if you’re struggling to get a response from gcloud, send it to sed and then filter out the relevent data you want … then this is another option.

In my experience, the hard part is figuring out how to build a client request and authenticate to AWS or GCP. (You can even make a template file for that.) After that, it’s fairly easy to list S3 bucket objects, or get a list of VMs with a given tag. So let’s address how to do those two things: build a client request and authenticate.

I like to work in Go, so the examples below use that. I expect almost all of this should be similar for SDKs in other programming languages.

Client Requests

The SDK for both GCP and AWS provide separate libraries for each of the services that we want to interface with. So building a client means doing so for the GCP storage service, or for Amazon’s ec2 service. When starting a new project, you will have to run go get ... a few times to download the pieces you need.

On one hand, using various libraries and not a monolith makes it a little confusing to know which packages to download. On the other, it means we don’t have to incorporate a lot of library code that we aren’t using.

Authentication

As a developer at heart, I think this is where the process becomes interesting, because we have a fair bit of flexibility for authenticating with these different services.

The first choice is whether we want to install the CLI or not. It’s not a requirement. But if we do install the CLI and login, then our Go code will try to use the existing credentials to satisfy our requests. That can be helpful or counter-productive in the case of multiple accounts.

If we don’t install the CLI, then we can manage those credentials and build the authentication provider ourselves. Those credentials could be local variables or environment variables, the contents of a file that accompanies the program, or even a file that we embed in the Go binary using embed from the standard library. The credentials can even come from a network request, or database query.

Consider the security implications of each of these alternatives. I trust you understand your needs, so I won’t say one solution is best and another is wrong.

AWS and Authentication

When you install the AWS CLI and login the first time, it creates the ~/.aws/credentials where we should have a “default” account. In here, we can add any additional accounts, or profiles, that we need. (I typically retrieve access keys like this from the AWS console in a web browser.) Example file:

[default]
aws_access_key_id = 123
aws_secret_access_key = 45678
region = us-east-1
[s3_limited_user]
aws_access_key_id = 321
aws_secret_access_key = 87654

Default authentication

So to write some code that tries to use the “default” credentials above, we need to do nothing more than build a client using LoadDefaultConfig().

package main
 
import (
	"context"
	"fmt"
 
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)
 
func main() {
	// 1. Load the Shared AWS Configuration (~/.aws/config)
	// Note: remember to add "region" to your profile, or set it here.
	cfg, err := config.LoadDefaultConfig(context.Background())
 
	// 1a. Specify one of multiple profiles on the local machine.
	// cfg, err := config.LoadDefaultConfig(context.Background(),
	// 	config.WithSharedConfigProfile("s3_limited_user"),
	// 	config.WithRegion("us-east-2"),
	// )
	if err != nil {
		panic(err)
	}
 
	// Do some work.
	s3client := s3.NewFromConfig(cfg)
	ListBucket("my-bucket-name", s3client)
}
 
func ListBucket(bn string, cl *s3.Client) {
	cfg := &s3.ListObjectsV2Input{
		Bucket: aws.String(bn),
	}
	output, err := cl.ListObjectsV2(context.Background(), cfg)
	if err != nil {
		panic(err)
	}
 
	for _, obj := range output.Contents {
		fmt.Println(aws.ToString(obj.Key))
	}
}

Example #1 will use the “default” profile stored in that credentials file. I included a comment with example #1a, which shows how to specify another profile from that file to use instead. Note: the “region” is often required, so remember to include it in the credentials file or pass it using config.WithRegion().

Specify the credentials file

What if we don’t want to install the CLI on our machine? Or we don’t want to use the credentials stored there? Or we plan to deploy this program to a machine where none of that is available?

We can specify the location of the credentials file, like this:

package main
 
import (
	"context"
	"fmt"
 
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)
 
func main() {
	// 2. Pass credentials file. (Does not need the aws cli)
	// This is adding to the normal chain of files, so will
	// keep trying until it succeeds, i.e. ~/.aws/credentials.
	cfg, err := config.LoadDefaultConfig(context.Background(),
		config.WithSharedCredentialsFiles(
			[]string{"./aws_config_file"},
		),
		config.WithRegion("us-east-2"),
	)
	if err != nil {
		panic(err)
	}
 
	// Do some work.
	s3client := s3.NewFromConfig(cfg)
  ...
  }

One thing to keep in mind is that we are passing a slice of strings to WithSharedCredentialsFiles() — not setting a single filename. That means our program will still to resolve the credentials using the default methods (~/.aws/credentials file, etc.). But it will try our custom files first.

Manually pass credentials

The last option is to manually build the whole set of credentials ourselves. That can be through environment variables, external requests, etc. In any of those cases, we rely on the NewStaticCredentialsProvider() method.

package main
 
import (
	"context"
	"fmt"
 
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/gobuffalo/envy"
)
 
func main() {
	// 3. Build credentials from env variables.
	err := envy.Load(".env")
	if err != nil {
		panic(err)
	}
	staticCreds := credentials.NewStaticCredentialsProvider(
		envy.Get("key", ""),
		envy.Get("secret", ""),
		"",
	)
	cfg, err := config.LoadDefaultConfig(context.Background(),
		config.WithCredentialsProvider(staticCreds),
		config.WithRegion("us-east-2"),
	)
 
	if err != nil {
		panic(err)
	}
 
	// Do some work.
	s3client := s3.NewFromConfig(cfg)
  ...
}

This example assumes we have a .env file in the project, and we use the Envy library from GoBuffalo to read the variables. There are other packages that can accomplish this. Whatever you do, the static provider expects three things:

• access key ID
• secret access key
• token

In my experience, the token is not required for many typical requests.

GCP and Authentication

GCP looks very similar to AWS in many regards. Both platforms embrace the concept of having multiple accounts, user profiles or service accounts, that can be used to authenticate. It’s expected that accounts have different levels of access. And multiple accounts can be used from a single machine.

Also, if you’re using virtual machines on GCP, any program that tries to use the default NewClient method will assume the permissions granted to the machine instance. No extra work required.

The typical method for obtaining account credentials is through the GCP console in a web browser. Unlike AWS, however, the average credentials file from GCP is in the JSON format and contains a lot more data than just access-key-id and secret key. More on application default credentials.

Default authentication

After installing gcloud and logging in, we will have a number of files and folders in the ~/.config/gcloud folder. There is more happening in this folder, compared to the ~/.aws/ directory, but the purpose is the same. We have a default account for executing commands on the GCP platform, and we enjoy the option of adding other accounts as well.

package main
 
import (
	"context"
	"fmt"
	"log"
	"strings"
 
	"cloud.google.com/go/storage"
	"google.golang.org/api/iterator"
)
 
func main() {
	// 1. Default authentication using gcloud config.
	client, err := storage.NewClient(context.Background())
 
	if err != nil {
		panic(err)
	}
	// Do some work.
	ListBuckets("my-bucket-name", client)
}
 
func ListBuckets(bn string, cl *storage.Client) {
	bkt := cl.Bucket(bn)
 
	var names []string
	it := bkt.Objects(context.Background(), nil)
	for {
		attrs, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			log.Fatal(err)
		}
		names = append(names, attrs.Name)
	}
	fmt.Printf("Objects:\n%s\n", strings.Join(names, "\n"))
}

There is very little setup required. Our program looks for the credentials in the default gcloud location, and tries to execute the command.

Specify the credentials file with embed

Without the CLI installed, we will have to tell our program where to find the credentials file. (With gcloud installed, this method can also helpful to test out an account that is not enabled on the local machine.)

We can simply tell our program where to find that file on the local filesystem. Another option is to embed the file within the program itself.

Hold up! If embedding a credentials file in your program makes your spider sense tingle, that’s fair. It may be the wrong approach for your project.

package main
 
import (
	"context"
	_ "embed"
	"fmt"
	"log"
	"strings"
 
	"cloud.google.com/go/storage"
	"google.golang.org/api/iterator"
	"google.golang.org/api/option"
)
 
//go:embed credentials.json
var credsFile []byte
 
func main() {
	// 2. Read JSON file to authenticate.
	// client, err := storage.NewClient(context.Background(),
	// 	option.WithCredentialsFile("./credentials.json"),
	// )
 
	// 2a. Embed a JSON file to authenticate.
	client, err := storage.NewClient(context.Background(),
		option.WithCredentialsJSON(credsFile),
	)
 
	if err != nil {
		panic(err)
	}
	// Do some work.
  ...
}

The SDK has two methods that are relevant. One is used to point to a filename, while the other is meant to take JSON as the parameter. The embed package from Go’s standard library reads the file contents into a byte slice, so we use the second method above.

A third option is to build that JSON ourselves, using environment variables or fetching data from somewhere else. Compared to the AWS example, this is a bit more complicated in GCP and leaves me thinking it’s intended to discourage that method in favor of other authentication patterns.

Conclusion

We are just scratching the surface of powerful programs we can write using the SDKs for Go to manage and control resources on AWS or GCP. Writing programs can offer more control than the CLI tools available, without requiring a lot of new learning or time investment, like building our own Terraform.