Testing Vault: Next-Gen Secret Management

I mainly followed this tutorial. But This is very old. And I just posted an upgraded steps here that I learnt when getting error following the process in that old post.

Create a Digital Ocean Droplet and Get SSL cert

• One Ubuntu 16.04 server set up by following  the initial setup guide for Ubuntu 16.04, including a sudo non-root user and a firewall.
• An SSL certificate, which we will use to secure Vault's HTTP API. You can get one for free by following  this Certbot standalone mode tutorial.

Install Vault

wget https://releases.hashicorp.com/vault/1.1.0/vault_1.1.0_linux_amd64.zip
sudo apt-get update
sudo apt-get install unzip
unzip vault_*.zip
sudo chown root:root vault
sudo mv vault /usr/local/bin/
vault -autocomplete-install
complete -C /usr/local/bin/vault vault

Finally, set a Linux capability flag on the binary. This adds extra security by letting the binary perform memory locking without unnecessarily elevating its privileges.

sudo setcap cap_ipc_lock=+ep /usr/local/bin/vault

You can now use the  vault command. Try checking Vault's version to make sure it works.

vault --version

Creating the Vault Unit File

First, create a  vault system user.

sudo useradd -r -d /var/lib/vault -s /bin/nologin vault

Here, we use  /var/lib/vault as the user's home directory. This will be used as the Vault data directory. We also set the shell to  /bin/nologin to restrict the user as a non-interactive system account. Set the ownership of  /var/lib/vault to the  vault user and the  vault group exclusively.

sudo install -o vault -g vault -m 750 -d /var/lib/vault

Then create vault.hcl file. My let's-encrypt certificate is issued for vault.knsakib.com. So, my configuration file is.

sudo mkdir --parents /etc/vault.d
sudo touch /etc/vault.d/vault.hcl
sudo chown --recursive vault:vault /etc/vault.d
sudo chmod 640 /etc/vault.d/vault.hcl
sudo nano /etc/vault.d/vault.hcl

The content is

ui = true
	
backend "file" {
	path = "/var/lib/vault"
}
	
api_addr = "https://vault.knsakib.com:8200"
listener "tcp" {
	address = "vault.knsakib.com:8200"
	tls_disable = 0
	tls_cert_file = "/etc/letsencrypt/live/vault.knsakib.com/fullchain.pem"
	tls_key_file = "/etc/letsencrypt/live/vault.knsakib.com/privkey.pem"
}

Next, to let Systemd manage the persistent Vault daemon, create a  unit file at  /etc/systemd/system/vault.service.

sudo nano /etc/systemd/system/vault.service
	

Add the following,

[Unit]
	Description="HashiCorp Vault - A tool for managing secrets"
	Documentation=https://www.vaultproject.io/docs/
	Requires=network-online.target
	After=network-online.target
	ConditionFileNotEmpty=/etc/vault.d/vault.hcl
	
[Service]
	User=vault
	Group=vault
	ProtectSystem=full
	ProtectHome=read-only
	PrivateTmp=yes
	PrivateDevices=yes
	SecureBits=keep-caps
	AmbientCapabilities=CAP_IPC_LOCK
	Capabilities=CAP_IPC_LOCK+ep
	CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
	NoNewPrivileges=yes
	ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
	ExecReload=/bin/kill --signal HUP $MAINPID
	KillMode=process
	KillSignal=SIGINT
	Restart=on-failure
	RestartSec=5
	TimeoutStopSec=30
	StartLimitIntervalSec=60
	StartLimitBurst=3
	LimitNOFILE=65536
[Install]
	WantedBy=multi-user.target

Using TLS

Finally, Vault needs permission to read the certificates you created with Certbot. By default, these certificates and private keys are only accessible by  root. To make these available securely, we'll create a special group called  pki to access these files. We will create the group and then add the  vault user to it.

Save and close the file, then create the  pki group.

sudo groupadd pki	
sudo chgrp -R pki /etc/letsencrypt/{archive,live}	
sudo chmod -R g+rx /etc/letsencrypt/{archive,live}

Then add the  vault user to the  pki group. This will grant Vault access to the certificates so that it can serve requests securely over HTTPS.

sudo gpasswd -a vault pki	

As a final step for convenience, add a rule in  /etc/hosts to direct requests to Vault to  localhost.

By default, Vault will only listen for requests from the loopback interface ( lo , or address  127.0.0.1 ). This is to ensure that the service is not exposed to the public internet before it has been properly secured. You can update this later, but for now, this configuration change will let us use the  vault  command and correctly resolve the HTTPS-secured domain name.

Replace  vault.knsakib.com in the following command with domain you acquired the Let's Encrypt certificate for:

echo 127.0.0.1 vault.knsakib.com | sudo tee -a /etc/hosts

This appends the line  127.0.0.1 vault.knsakib.com to  /etc/hosts so that any HTTP requests to  example.com are routed to  localhost.

Initializing Vault

	sudo systemctl enable vault
	sudo systemctl start vault
	sudo systemctl status vault
	Output
	. . .
	Active: active (running)
	. . .

Let's export VAULT_ADDR=https://vault.knsakib.com:8200

export VAULT_ADDR=https://vault.knsakib.com:8200

Initialize Vault with the aforementioned parameters:

vault operator init -key-shares=3 -key-threshold=2
vault operator unseal

Run the command 2 times and put the unsealing keys until it is unseal. Keep the root token safe when it is revealed. It is always a good practice to revoke when it's initial job done. That means initial policy and auth creation.

Reading and Writing Secrets

I want to read and write secrets, like it has to be done in the application using rest API, instead of CLI. But at first let's finish initial root token's work. Then we can revoke it.

export VAULT_TOKEN=####
vault secrets enable kv 
vault write kv/my-secret value="passwordblah!"
vault write kv/hello target=world

Let's write some policy.

sudo nano my-policy

The content is

path "secret/*" {
    capabilities = ["create"]
}
path "secret/foo" {
    capabilities = ["read"]
}
# Dev servers have version 2 of KV mounted by default, so will need these
# paths:
path "secret/data/*" {
    capabilities = ["create"]
}
path "secret/data/foo" {
    capabilities = ["read"]
}

vault policy fmt my-policy.hcl
vault policy list

Let's create a Auth Token associated with that policy,

vault token create -policy=my-policy

It will generate a new token. Let use that token in simple skeleton Node app. I will use this vault Node.js Client.

var Vault = require('node-vault-js');
var vault = new Vault({  
	endpoint: 'https://vault.knsakib.com:8200',  
	token: '####'
});	
vault.read('kv/my-secret', function(err, result) {        
	if (err) { 
	    throw err 
	}  
console.log('Read with response: ', result)
});

It is a very simple app. I just want to see how I can call the API in my app. In fact. It is a next generation of secret management. We can now easily create another app whose sole purpose is using IAM service API (Google, AWS, Azure) to generate new short lived token every time. And Our app(the app that will consume the secret) will received each new tokens every time, which encrypted+short lived :). And the secrets will never leave the Vault! :)

by the way the output looks like after I ran the node index.js,

Read with response:  { 
	request_id: '########',
	lease_id: '',
	renewable: false,
	lease_duration: 2764800,
	data: { value: 'PASSWORD_VALUE' },
	wrap_info: null,
	warnings: null,
	auth: null 
}