Go and Vault (role id auth)

Intro

This section will review one possible approach to getting Go to play nicely with Vault. I’ll also include a full code reference, as well as an in-depth explanation of how all of it works. If you’re unsure of what Vault does exactly keep an eye out for the upcoming tutorial.

Pre-reqs

I will assume you have a Go application ready-to-go and either a local Vault setup or access to a running Vault service.

TL;DR

Click here to go see the full implementation

Implementation Overview

Okay, now the “in a hurry” folks are looking at errors in their IDEs, let’s go through this step-by-step

This implementation is set up as a Go module, and it contains two files: “vault.go” and “leaseUpdater.go”. The dir structure should look like this:

app/
   vault/
      vault.go
      leaseUpdater.go

The “vault.go” file includes everything needed to communicate with the Vault instance, authenticate the user and read the values from a secret.

The “leaseUpdater.go” file manages the update of the secret values, as they are rotated on a TTL basis. This is essential, as you’ll need the freshest creds while your service runs.

The power of the env

Everything is powered by env variables from the secret paths to the credential storage. Here’s what you’ll need in your env file:

# Vault instance specific
VAULT_TTL
VAULT_ADDR
VAULT_ROLE_ID
VAULT_SECRET_ID

# Secret specific
EXAMPLE_VAULT_LEASE
USER_VAULT
PASS_VAULT

What do these mean:

VAULT_TTL – Instructs the lease updater when to run. Think of this as the cycle in which new credentials are going to be requested. Should be defined in minutes
VAULT_ADDR – The address the Vault instance can be found on, with the port definition (8200 is the default)
VAULT_ROLE_ID – The role id to be used when authenticating against the Vault instance
VAULT_SECRET_ID – The role’s counterpart, completing the creds pair that’s needed to auth against Vault
EXAMPLE_VAULT_LEASE – the lease we’re about to extract data from. This can be anything really, as long as it returns a username/password combo (secrets, static creds, dynamic creds etc.)
USER_VAULT – Leave this blank. This is where the “leaseUpdater” will insert the username once fetched from the lease
PASS_VAULT – Leave this blank. This is where the “leaseUpdater” will insert the password once fetched from the lease

That’s that, nothing fancy or complex going on here. If you’re going to use the creds to connect to a database, simply instruct the connection string builder to use USER_VAULT and PASS_VAULT and you’re ready to go.

The vault package

Let’s install that now, as we’ll need the support and base logic integration it provides, so we can build on top of it later.

go get github.com/hashicorp/vault/api

Reference

The wrapper

There’re three things we need to implement here so that we can read a secret’s value:

Instantiate a vault client
Authenticate a vault user
Read from a secret

Prerequisites

The Client

To use a client within our application we need to instruct Go what a client is. In its eyes, it’ll be a struct

type Client struct {
	Token         string
	Client        *api.Client
	LeaseDuration int
}

The client has a token of type string, a client pointer, and a lease duration, which you can use to substitute the VAULT_TTL env var if you so desire.

The Token

We need to define another struct here for the token declaration:

type Token struct {
	Auth struct {
		ClientToken   string `json:"client_token"`
		LeaseDuration int    `json:"lease_duration"`
	} `json:"auth"`
}

This is a complex struct, a double-struct. The Token contains an Auth construct inside of it that will hold the ClientToken (used for identification) and the LeaseDuration. This might seem a bit weird, but it’s tailored to vault’s go package, so yeah.

The login

We need to add another struct here, fortunately it’s self-explanatory:

type appRoleLogin struct {
	RoleID   string `json:"role_id"`
	SecretID string `json:"secret_id"`
}

Instantiating a client

This is one of the two main building blocks here, so let’s write a function that will return a Vault client:

func NewVaultClient() (*Client, error) {
	vaultClient := Client{}

	client, err := api.NewClient(&api.Config{
		Address: vaultAddr,
	})

	vaultClient.Client = client

	return &vaultClient, err
}

The address injection here is done with a variable, if you want to read directly from the env, substitute it with: os.Getenv("VAULT_ADDR")

And that’s that no error handling here, that’s done in the caller function.

Authenticating a user

Before we even think about reading a secret’s values, we need to authenticate our mechanised user. Let’s write a function for that:

func (vaultClient *Client) AuthUser() (string, error) {
	// step: create the token request
	request := vaultClient.Client.NewRequest("POST", "/v1/auth/approle/login")
	login := appRoleLogin{SecretID: secretId, RoleID: roleId}

	if err := request.SetJSONBody(login); err != nil {
		return "", err
	}

	// step: make the request
	resp, err := vaultClient.Client.RawRequest(request)
	if err != nil {
		return "", err
	}

	defer resp.Body.Close()

	// step: parse and return auth
	secret, err := api.ParseSecret(resp.Body)
	if err != nil {
		return "", err
	}

	return secret.Auth.ClientToken, nil
}

So what are we doing here? We first start with declaring a request and login variables that will be used in the Vault call later on.

Note we’re using a Client pointer, this allows us to grab an instance of a Vault Client and reuse it.

Note that we’re using /v1/auth/approle/login, for more info on Vault’s authentication mechanisms click here.

Next we add the login struct to the request as a JSON body. And we send the request, whilst capturing any errors (if they occur)

If everything went well we parse the secret that was returned from the request and we now have a Client Token we can use. Congrats!

Reading a secret

This is by far the easiest function both to write, and to understand. We’re essentially doing vault read /some/secret and ingesting the values.

func ReadSecretValues(secretPath string, userDesignation string, passDesignation string) (map[string]interface{}, error) {
	data := make(map[string]interface{})
	client, err := NewVaultClient()

	if err != nil {
		return nil, err
	}

	token, err := client.AuthUser()

	if err != nil {
		return nil, err
	}

	client.Client.SetToken(token)
	secretValues, err := client.Client.Logical().Read(secretPath)

	if err != nil || secretValues == nil {
		return nil, err
	}

	data["username"] = secretValues.Data[userDesignation]
	data["password"] = secretValues.Data[passDesignation]

	return data, err
}

So what’s happening here? We grab a client. We then authenticate against Vault, using that client. And if we have a token, we ask the client to read from a secret (secret path).

If there’s data read from the secret path, we push it to a new map with a username/password designation.

Note – both the userDesignation & passDesignation have been abstracted, as some people create a secret store with user, some use username.

Half-time

Theoretically, that’s all you need to read from a secret path, the thing is that values expire and rotate in Vault. It would be cumbersome to call Vault each time you want to run a query.
This is where the lease updater file comes into play. It takes care of always keeping your values up-to-date, doing so on a set period.
So let’s look at how to implement the lease updater as well.

The lease updater

Defining the variables

Let’s define all the variables this process will consume

var (
	ttl, _ = strconv.Atoi(os.Getenv("VAULT_TTL")) // IN MINUTES

	leasePath 		= os.Getenv("EXAMPLE_VAULT_LEASE")
	leaseSecretPattern         = []string{"username", "password"}
	leaseSecretInjectionPoints = map[string]string{"username": "USER_VAULT", "password": "PASS_VAULT"}

	step   = time.Duration(ttl) * time.Minute
	ticker = time.NewTicker(step)
	done   = make(chan bool)
)

Nothing fancy here, the declaration is split into two sections. The first one relates to the secret we want to read and where to inject the values.
The second relates to the periodic looping mechanism, called a ticker in Go.

Note – if you want to run the ticker on a specific TTL, instead of a hard-coded limit in your env, substitute the first var declaration.

When reviewing the implementation, we’ll start from the bottom-up, as this is a layered approach and it’ll be easier to comprehend going backwards (stay with me)

The value injector

This function takes an arbitrary number of values and injection points and simply exports them to the OS level.

func secretValuesInjector(values map[string]interface{}, injectionPoints map[string]string) error {
	if len(values) != len(injectionPoints) {
		return errors.New("length mismatch, aborting")
	}

	for i, envKey := range injectionPoints {
		err := os.Setenv(envKey, fmt.Sprintf("%v", values[i]))

		if err != nil {
			break
		}
	}

	return nil
}

The only thing that needs to be pointed out is the values checker. If you request 3 values to be injected into 2 env vars, the function will throw an error.

The injection wrapper

This function fetches the values by reusing this function (“vault.go”) and then calls the injector.

func injectWrapper(leasePath string, secretPattern []string, injectorPoints map[string]string) error {
	if secretPattern[0] == "" || secretPattern[1] == "" {
		return errors.New("secret pattern UNDEFINED")
	}

	secretValues, err := ReadSecretValues(leasePath, secretPattern[0], secretPattern[1])

	// Fail-safes 1/2
	if err != nil {
		return err
	}

	// Fail-safes 2/2
	if secretValues == nil {
		return errors.New("no secret values in path")
	}

	// Inject the values
	injErr := secretValuesInjector(secretValues, injectorPoints)

	if injErr != nil {
		return err
	}

	return nil
}

There’s a double fail-safe here, just so we can be absolutely sure we’re not about to export some random thing to an OS variable.
The first layer checks if the read function threw an error.
The second layer checks if there’re actually any values in the requested path. Vault would happily not throw an error on empty secret values, so we need to handle that.

After the scrutiny of our fail-safes, the function calls the injector and listens for a possible error.

The abstraction wrapper

I want to be able to call the injector for an N number of leases I want to read. Thus we need to abstract the wrapper we wrote in the previous step.

func updateValues() {
	// Run the injector at least once
	err := injectWrapper(leasePath, leaseSecretPattern, leaseSecretInjectionPoints)

	if err != nil {
		return
	}
}

That’s it, if you need to read from more than one secret, we simply reuse the function.

The ticker

This is a timed continuous execution of the functions we wrote above, that will keep our values up-to-date

func UpdateSecretValues() {
	// Update the values immediately, and then go into a ticker pattern
	updateValues()

	// Ticker
	for {
		select {
		case <-done:
			return
		case t := <-ticker.C:
			updateValues()
		}
	}
}

We call the updater once, to make sure that we have values in our vars, before going into the ticker pattern.

Attaching the updater to your application

This is done simply by adding this line to your “main.go” file:

go UpdateSecretValues()

And off it goes.

The full implementation

This is the main logic provider. Call this “vault.go” if you copy-paste this block

package vault

import (
	"github.com/hashicorp/vault/api"
	"os"
)

var (
	vaultAddr = os.Getenv("VAULT_ADDR")
	roleId    = os.Getenv("VAULT_ROLE_ID")
	secretId  = os.Getenv("VAULT_SECRET_ID")
)

type Client struct {
	Token         string
	Client        *api.Client
	LeaseDuration int
}

type Token struct {
	Auth struct {
		ClientToken   string `json:"client_token"`
		LeaseDuration int    `json:"lease_duration"`
	} `json:"auth"`
}

type appRoleLogin struct {
	RoleID   string `json:"role_id"`
	SecretID string `json:"secret_id"`
}

func NewVaultClient() (*Client, error) {
	vaultClient := Client{}

	client, err := api.NewClient(&api.Config{
		Address: vaultAddr,
	})

	vaultClient.Client = client

	return &vaultClient, err
}

func (vaultClient *Client) AuthUser() (string, error) {
	// step: create the token request
	request := vaultClient.Client.NewRequest("POST", "/v1/auth/approle/login")
	login := appRoleLogin{SecretID: secretId, RoleID: roleId}

	if err := request.SetJSONBody(login); err != nil {
		return "", err
	}

	// step: make the request
	resp, err := vaultClient.Client.RawRequest(request)
	if err != nil {
		return "", err
	}

	defer resp.Body.Close()

	// step: parse and return auth
	secret, err := api.ParseSecret(resp.Body)
	if err != nil {
		return "", err
	}

	return secret.Auth.ClientToken, nil
}

// Generic secret value reader
func ReadSecretValues(secretPath string, userDesignation string, passDesignation string) (map[string]interface{}, error) {
	data := make(map[string]interface{})
	client, err := NewVaultClient()

	if err != nil {
		return nil, err
	}

	token, err := client.AuthUser()

	if err != nil {
		return nil, err
	}

	client.Client.SetToken(token)
	secretValues, err := client.Client.Logical().Read(secretPath)

	if err != nil || secretValues == nil {
		return nil, err
	}

	data["username"] = secretValues.Data[userDesignation]
	data["password"] = secretValues.Data[passDesignation]

	return data, err
}

This is the lease updater. Call this “leaseUpdater.go” if you copy-paste this block

package vault

import (
	"errors"
	"fmt"
	"os"
	"strconv"
	"time"
)

// An automated process to update secret values periodically
var (
	ttl, _ = strconv.Atoi(os.Getenv("VAULT_TTL")) // IN MINUTES

	leasePath 		= os.Getenv("EXAMPLE_VAULT_LEASE")
	leaseSecretPattern         = []string{"username", "password"}
	leaseSecretInjectionPoints = map[string]string{"username": "USER_VAULT", "password": "PASS_VAULT"}

	step   = time.Duration(ttl) * time.Minute
	ticker = time.NewTicker(step)
	done   = make(chan bool)
)

func UpdateSecretValues() {
	// Update the values immediately, and then go into a ticker pattern
	updateValues()

	// Ticker
	for {
		select {
		case <-done:
			return
		case t := <-ticker.C:
			updateValues()
		}
	}
}

// Simple high-level wrapper script
func updateValues() {
	// Run the injector at least once
	err := injectWrapper(leasePath, leaseSecretPattern, leaseSecretInjectionPoints)

	if err != nil {
		return
	}
}

// Injection wrapper
func injectWrapper(leasePath string, secretPattern []string, injectorPoints map[string]string) error {
	if secretPattern[0] == "" || secretPattern[1] == "" {
		return errors.New("secret pattern UNDEFINED")
	}

	secretValues, err := ReadSecretValues(leasePath, secretPattern[0], secretPattern[1])

	// Fail-safes 1/2
	if err != nil {
		return err
	}

	// Fail-safes 2/2
	if secretValues == nil {
		return errors.New("no secret values in path")
	}

	// Inject the values
	injErr := secretValuesInjector(secretValues, injectorPoints)

	if injErr != nil {
		return err
	}

	return nil
}

// Takes an arbitrary number of values and injection points and sets env vars values
func secretValuesInjector(values map[string]interface{}, injectionPoints map[string]string) error {
	if len(values) != len(injectionPoints) {
		return errors.New("length mismatch, aborting")
	}

	for i, envKey := range injectionPoints {
		err := os.Setenv(envKey, fmt.Sprintf("%v", values[i]))

		if err != nil {
			break
		}
	}

	return nil
}

Home

Categories

Vault&Go(role id based)