Running Vault on Nomad, Part 3

This post is the third and final part of a series on how to deploy the underlying HashiCorp Nomad infrastructure and configuration to run HashiCorp Vault as a Nomad job. Part 1 focused on deploying the infrastructure to run Vault on Nomad, while part 2 took a deep dive into running Vault as a Nomad job. This installment looks at automating operational tasks for Vault using Nomad jobs.

Specifically, it covers how to automate: 1. Unsealing Vault 2. Taking snapshots of Vault

What is unsealing Vault?

As a secrets management platform, Vault stores sensitive and often mission-critical data in its storage backend. This data is encrypted using an encryption key, which is required to decrypt the data in the storage backend. Vault stores this key with the encrypted data and further encrypts it using another key known as the root key.

Unsealing is the process of decrypting the encryption key using the root key and then decrypting the data using the encryption key. Until this process has been completed, you can perform only two operations on the Vault server:

1. Checking the seal status of Vault
2. Unsealing Vault

The root key is normally split into a configurable number of shards, known as unseal keys, using Shamir's Secret Sharing algorithm, and these are distributed to engineers responsible for unsealing Vault. The unseal process consists of a pre-specified threshold of unseal keys being entered by the key holders.

Auto unsealing Vault

When a Vault server is started or restarted, it comes up in a sealed state, which means that only the two operations mentioned above can be performed on it. There are many reasons why Vault might need to be restarted, from OS patching to resource consumption issues. Whatever the reason, unsealing presents a potentially huge management overhead burden for the Vault servers.

To address this, Vault’s auto-unseal feature delegates the responsibility of unsealing Vault to a service like a cloud key management service (KMS) or a device like a hardware security module (HSM). As the name suggests, using auto-unseal means that the Vault servers will be automatically unsealed when they are started or restarted.

Vault Unsealer

There are many reasons why some organizations cannot use auto-unseal. Their security policies may not allow cloud services, or they may not want to pay the high procurement and operational costs of HSMs. Vault Unsealer, is a proof-of-concept tool designed to automate the process of unsealing Vault using the unseal keys.

How Vault Unsealer works

Vault Unsealer checks the seal status of each Vault server in the cluster and unseals any servers reporting their status as sealed. Under the hood, Vault Unsealer uses the Vault API to perform these tasks.

Configuring Vault Unsealer

In order to use Vault Unsealer, you’ll configure a JSON file to tell it which servers to manage the unseal state on, the unseal keys to use, how often it should check the seal status, and the log level to output to stdout. Here is an example configuration file:

{
 "log_level": "debug",
 "probe_interval": 10,
 "nodes": [
   "http://192.168.1.141:8200",
   "http://192.168.1.142:8200",
   "http://192.168.1.143:8200"
 ],
 "unseal_keys": [
   "aa109356340az6f2916894c2e538f7450412056cea4c45b3dd4ae1f9c840befc1a",
   "4948bcfe36834c8e6861f8144672cb804610967c7afb0588cfd03217b4354a8c35",
   "7b5802f21b19s522444e2723a31cb07d5a3de60fbc37d21f918f998018b6e7ce8b"
 ]
}

NOTE: The unseal keys are sensitive pieces of data, so we recommend that the config file is rendered with the unseal keys’ values coming from an encrypted store that you trust.

Deploying Vault Unsealer as a Nomad job

For this post, the code is located within the 2-nomad-configuration directory.

Writing a Nomad jobspec for Vault Unsealer is similar to the process in part 2 of the blog series with some subtle differences because the requirements for this job are slightly less than that of the Vault cluster. Here is the vault-unsealer.nomad file:

job "vault-unsealer" {
 namespace   = "vault-cluster"
 datacenters = ["dc1"]
 type        = "service"
 node_pool   = "vault-servers"

 group "vault-unsealer" {
   count = 1

   constraint {
     attribute = "${node.class}"
     value     = "vault-servers"
   }

   task "vault-unsealer" {
     driver = "docker"

     config {
       image      = "devopsrob/vault-unsealer:0.2"

       command = "./vault-unsealer"
       volumes = [
         "local/config:/app/config"
       ]
     }

     template {
       data = 

from HashiCorp Blog https://ift.tt/yZptJxM
via IFTTT