January 31, 2020

Technical Memo: Disk Encryption using Cryptsetup with Vault as Key Management Service

In Linux Operating system, full disk encryption could be achieved by various solutions: encryptfs, dm-crypt, cryptsetup… While a step-by-step setup of disk/partition encryption can be referred from official documentation, integration disk encryption with Key Management solution like Hashicorp Vault is unobvious. Fortunately, there are a package named vaultlocker in Ubuntu Universe repository that ease this integration. I decided to spend my free time to make cryptsetup work with Vault.

Notes: A similar request has been made to Cryptsetup but it goes outside of cryptsetup so it was closed.

1. Install Vault

As JuJu charm store has working Vault/MySQL charms, Vault deployment is as simple and easy as following commands

$ juju deploy percona-cluster
$ juju deploy vault
$ juju relate vault:shared-db percona-cluster

Vault needs to be initialized and unsealed before it can provide its secure services. I followed below steps to initialize Vault

$ juju ssh vault/0
$ export VAULT_ADDR=
$ vault operator init -key-shares=3 -key-threshold=2

Unseal Key 1: hALty7n4UHvYQAN89xy3SE7yb2SUYM+9DR/tEpPGIQDB 
Unseal Key 2: b+OcCcpjHQ+eG/dZ1bcGuwbL9LM1p2xR0yEha5XnsvO4 
Unseal Key 3: DGo4UAsb894lXw4musCeUp30jJqoi5Qiyg5YQDpSRRqT 

Initial Root Token: s.k5awSEin8LFTHQXy3qXM8GFv 
This gives me 3 keys. I need to use 2 keys to unseal Vault in case it is locked. As we are initiating Vault, I use 2 keys to unseal vault.

$ vault operator unseal hALty7n4UHvYQAN89xy3SE7yb2SUYM+9DR/tEpPGIQDB 
$ vault operator unseal b+OcCcpjHQ+eG/dZ1bcGuwbL9LM1p2xR0yEha5XnsvO4 

Now Vault should be unsealed and its result can be confirmed by vault status

$ vault status 

Key Value 
--- ----- 
Seal Type shamir 
Initialized true 
Sealed false 
Total Shares 3 
Threshold 2 
Version 1.1.1 
Cluster Name vault-cluster-44e9a806 Cluster ID 1087afa0-c7f1-01e4-abce-187b7b5f63a3 
HA Enabled false

Authorize Juju against Vault to allow JuJu actions to run.

$ export VAULT_TOKEN=s.k5awSEin8LFTHQXy3qXM8GFv

$ vault token create -ttl=10m
--- ----- 
token s.7jChN3pD8cbLLVUSYrXqsBig 
token_accessor gj76tBhiygYh1WsK1vamP20e 
token_duration 10m 
token_renewable true 
token_policies ["root"] 
identity_policies [] 
policies ["root"] 

$ juju status

Model Controller Cloud/Region Version SLA Timestamp 

default nucctr nuc/default 2.7.1 unsupported 14:24:12+09:00 

App Version Status Scale Charm Store Rev OS Notes 

percona-cluster 5.7.20 active 1 percona-cluster jujucharms 282 ubuntu 
vault 1.1.1 active 1 vault jujucharms 32 ubuntu 

Unit Workload Agent Machine Public address Ports Message 

percona-cluster/0* active idle 0 3306/tcp Unit is 
ready vault/0* active idle 1 8200/tcp Unit is ready (active: true, mlock: enabled) 

Machine State DNS Inst id Series AZ Message 

0 started needed-guinea bionic default Deployed 
1 started first-wahoo bionic default Deployed 

Reference: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-vault.html

2. Install VaultLocker and Setting Vault

On the test machine, install vaultlocker.

$ sudo apt update
$ sudo apt install vaultlocker

Vaultlocker will need information of Vault in order to talk to Vault. The information includes app role and proper policy to create information in Vault. So in order for vaultlocker to function, we need to create an approle and assign it a policy.

Create AppRole

I will create a approle named vaultluks with no timeout for secret-id

$ vault auth enable approle
$ vault write auth/approle/role/vaultluks

$ vault read auth/approle/role/vaultluks/role-id
Key Value 
--- ----- 
role_id 7cbb1c1c-2270-6120-04f6-c3b1c753191d 

$ vault write -f auth/approle/role/vaultluks/secret-id
Key Value 
--- ----- 
secret_id f657f357-4748-dd09-733c-58ff994bd985 
secret_id_accessor 5eb5f63f-478b-4f41-a1c9-63f151991cf6  

Create Secret Backend and Policy

In order for vaultlocker to write secret to Vault, a secret backend and a policy that allows vaultlocker to access that secret backend

$ vault secrets enable -path=luks kv 
Path Type Accessor Description 
---- ---- -------- ----------- 
charm-pki-local/ pki pki_c7e293a2 Charm created PKI backend 
cubbyhole/ cubbyhole cubbyhole_ffd5c863 per-token private secret storage 
identity/ identity identity_70188f48 identity store 
luks/ kv kv_0ad49111 n/a 
secret/ kv kv_7b586ad0 n/a 
sys/ system system_0cc7f9f9 system endpoints used for control, policy and debugging 

$ vim vaultluks.hcl
path "luks/*" { 
    capabilities = ["create", "read", "update", "delete", "list"] 

$ vault policy write vault luks vaultluks.hcl 

Configure Vaultlocker and Encrypt Disk

The url, approle and secret_id should be modified accordingly.

$ sudo vim /etc/vaultlocker/vaultlocker.conf
url = 
approle = 7cbb1c1c-2270-6120-04f6-c3b1c753191d 
secret_id = f657f357-4748-dd09-733c-58ff994bd985  
backend = luks 

$ sudo vaultlocker encrypt /dev/vdb2
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 
DEBUG:urllib3.connectionpool: "POST /v1/auth/approle/login HTTP/1.1" 200 484 
INFO:vaultlocker.dmcrypt:LUKS formatting /dev/vdb2 using UUID:bed01f5f-6766-4a3d-aa22-05703568da74 
INFO:vaultlocker.dmcrypt:udevadm trigger block/add for /dev/vdb2 INFO:vaultlocker.dmcrypt:udevadm settle /dev/disk/by-uuid/bed01f5f-6766-4a3d-aa22-05703568da74 
DEBUG:urllib3.connectionpool: "PUT /v1/luks/vault/bed01f5f-6766-4a3d-aa22-05703568da74 HTTP/1.1" 204 0 
DEBUG:urllib3.connectionpool: "GET /v1/luks/vault/bed01f5f-6766-4a3d-aa22-05703568da74 HTTP/1.1" 200 866 
INFO:vaultlocker.dmcrypt:LUKS opening bed01f5f-6766-4a3d-aa22-05703568da74 
INFO:root:Enabling systemd unit for vaultlocker-decrypt@bed01f5f-6766-4a3d-aa22-05703568da74.service Created symlink /etc/systemd/system/multi-user.target.wants/vaultlocker-decrypt@bed01f5f-6766-4a3d-aa22-05703568da74.service → /lib/systemd/system/vaultlocker-decrypt@.service. 

$ lsblk
NAME                                           MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda                                            252:0    0  7.5G  0 disk
└─vda1                                         252:1    0  7.5G  0 part  /
vdb                                            252:16   0    5G  0 disk
├─vdb1                                         252:17   0  1.9G  0 part
└─vdb2                                         252:18   0  3.1G  0 part
  └─crypt-bed01f5f-6766-4a3d-aa22-05703568da74 253:0    0  3.1G  0 crypt

Now vdb2 is encrypted and its key is stored in vaults under /luks path. Here bed01f5f-6766-4a3d-aa22-05703568da74 is the UUID of the disk and it is also the key that vaultlocker uses to retrieve the encryption key.

In vault server, you can confirm the key file stored in Vault. There should be a key with the same uuid as the device and value is the encryption key.

$ vault kv list luks/

ubuntu@first-wahoo:~$ vault kv list luks/vault

$ vault kv get luks/vault/bed01f5f-6766-4a3d-aa22-05703568da74
======= Data =======
Key            Value
---            -----
dmcrypt_key    LONGLONGKEY

How to use

Everytime you want to open the partition you could use

$ sudo vaultlocker decrypt ${UUID}


$ echo -n $KEY | sudo cryptsetup --batch-mode --key-file - open \
  UUID=bed01f5f-6766-4a3d-aa22-05703568da74 \ 
  crypt-bed01f5f-6766-4a3d-aa22-05703568da74 --type luks 

Where $KEY is the variable with value got from vault

January 22, 2020

SSH Port Forwarding

Local / Remote port forwarding are powerful features of ssh, yet I am confused of the their usage. Today I saw a great picture explaining in details many options of port forwarding. The author of the accepted answers drew a so great picture that I want to copy here for my own future reference.

"a picture worth a thousand words"

Source: https://unix.stackexchange.com/questions/115897/whats-ssh-port-forwarding-and-whats-the-difference-between-ssh-local-and-remot

ssh tunnel starting from remote

January 20, 2020


My old desktop becomes sluggish as I deploy multiple test environment on it. As I also use various desktop applications for my daily works on the same desktop machine, sometimes resource insufficiency freeze the all the applications, interfering my work flows. I am always a fan of small low-energy NUC devices so I take this chance to buy a new NUC device solve my resource issue.

Before buying new device, I had some considerations for a NUC.
  • Whether it's good time to buy a NUC? If yes, what version should I buy?
  • Whether it works with my 4K Dell Monitor?
  • Whether it generates too much heat?
  • Whether it supports enough resource for my daily work flows (I need a lot of Linux/Windows virtual machines)
  • How I should use 

It is a good timing?
At CES 2020, Intel announced an impressive NUC 9 and it made me think for a while whether I should wait for the release of this device. However, after checking around for price/performance ratio, I realize that it's not a good time to wait as newly released device will have a price premium, which is easily over my budget. Moreover, I probably don't need a device with such high performance. So I persuade myself to go with a NUC 8. Since NUC8, it seems that the heat problem has gone much better so I should have no issue with heat issue. As I am going to build software on my desktop, for energy saving purpose I went with an Corei5 model.

Whether it works with my 4K Monitor? 
I want the NUC should support 4K resolution with at least 60Hz refresh rate. I confirmed the tech spec for my NUC at Intel Nuc Tech Spec. Any model on the list should support 4K HDR. I went to the model field in the product page and saw BOXNUC8I5BEH as the device model, so this model supports 4K. All I need is a HDR 4K HDMI cable, so I bought this NUC device with a HDMI cable from Amazon.

Does it generate heat?
Memory does not generate much heat so I bought 2 Silicon Power DDR4 Laptop Memory, whose memory size is 16GB each. I wanted to buy memory from more well-known vendors like Kingston, but as my NUC transfer speed is 2400Hz and the only memory with the same transfer speed at the time I search for is from Silicon Power, I decided to buy these 2. If it breaks, I can easy replace them as memory now is inexpensive.

Resource Sufficiency?
I don't think I will have performance bottleneck with cpu and memory resources. For hard disk, I wondered between a standard SATA SSD device and a NVME device. NVME SSD is known to generate heat but with higher read/write performance. For a NUC, a normal SATA should be fine. However, I want to see whether I have a heat bottleneck, so I decided to go for a nvme device.

With some helps from Youtube, I could install SSD and RAM without any issue. I boot the device, plugged in a bootable USB with Ubuntu LTS 18.04. The OS install goes as usual with mouse clicks.

After the device boots, I realize that by default Ubuntu only generates display resolution of 4k at 30Hz, which is unacceptable for me. As 4K resolution is unreadable for my eyes even in 27-inches display, I performed following step to generate 2K resolution at 60Hz

$ cvt 2560 1440 60

# 2560x1440 59.96 Hz (CVT 3.69M9) hsync: 89.52 kHz; pclk: 312.25 MHz Modeline "2560x1440_60.00" 312.25 2560 2752 3024 3488 1440 1443 1448 1493 -hsync +vsync 

After that I add the following lines to ~/.xprofile
xrandr --newmode "2560x1440_60.00"  312.25  2560 2752 3024 3488  1440 1443 1448 1493 -hsync +vsync<br />
xrandr --addmode DP-1 2560x1440_60.00<br />

Where DP-1 is my display device. Now my display could show 2K resolution at 60Hz.

Other settings are as usual and will be topics for another blog post.

I bought a new NUC device and I am totally satisfied with it.