February 2, 2020

Technical Memo: Convert an ova image to qcow2 format

From Wikipedia of Open Virtualization Format, I learnt that an the ova file is just a tar file of ovf directory. So I went open a Windows 10 Developer Evaluation ova file to see what exist inside it.

$ tar -tf WinDev2001Eval.ova


So inside the ova file includes
  • A virtual machine specification definition .ovf file
  • A hash checksum .mf file
  • A disk image in .vmdk image
As qemu-img supports vmdk as disk image file, we actually could convert vmdk file to any other format that we like.

1. Convert .vmdk to .img

$ qemu-img convert -O raw WinDev2001Eval-disk001.vmdk WinDev2001Eval.img

2. Convert .vmdk to .qcow2 
$ qemu-img convert -O qcow2 WinDev2001Eval-disk001.vmdk WinDev2001Eval.qcow2

All supported formats can be confirmed by qemu-img -h

The converted qcow2 image could be used to create a new virtual machine using virt-manager.

I used to think that .ova file is the file format of Virtual Box (There are also many articles on the internet instruct you to install Virtual Box on Linux to open .ova file!). Actually, by simple conversion, we can run .ova packaging image on default simple performant Linux KVM. The OVF was originated from 2007 and it has been more than a decade from its creation, yet I am still amazed at how little I know about it.



January 31, 2020

Technical Memo: Disk Encryption using Cryptsetup with Vault as Key Management Service

In Linux Operating system, full disk encryption could be achieved by various solutions: encryptfs, dm-crypt, cryptsetup… While a step-by-step setup of disk/partition encryption can be referred from official documentation, integration disk encryption with Key Management solution like Hashicorp Vault is unobvious. Fortunately, there are a package named vaultlocker in Ubuntu Universe repository that ease this integration. I decided to spend my free time to make cryptsetup work with Vault.

Notes: A similar request has been made to Cryptsetup but it goes outside of cryptsetup so it was closed.

1. Install Vault

As JuJu charm store has working Vault/MySQL charms, Vault deployment is as simple and easy as following commands

$ juju deploy percona-cluster
$ juju deploy vault
$ juju relate vault:shared-db percona-cluster

Vault needs to be initialized and unsealed before it can provide its secure services. I followed below steps to initialize Vault

$ juju ssh vault/0
$ export VAULT_ADDR=
$ vault operator init -key-shares=3 -key-threshold=2

Unseal Key 1: hALty7n4UHvYQAN89xy3SE7yb2SUYM+9DR/tEpPGIQDB 
Unseal Key 2: b+OcCcpjHQ+eG/dZ1bcGuwbL9LM1p2xR0yEha5XnsvO4 
Unseal Key 3: DGo4UAsb894lXw4musCeUp30jJqoi5Qiyg5YQDpSRRqT 

Initial Root Token: s.k5awSEin8LFTHQXy3qXM8GFv 
This gives me 3 keys. I need to use 2 keys to unseal Vault in case it is locked. As we are initiating Vault, I use 2 keys to unseal vault.

$ vault operator unseal hALty7n4UHvYQAN89xy3SE7yb2SUYM+9DR/tEpPGIQDB 
$ vault operator unseal b+OcCcpjHQ+eG/dZ1bcGuwbL9LM1p2xR0yEha5XnsvO4 

Now Vault should be unsealed and its result can be confirmed by vault status

$ vault status 

Key Value 
--- ----- 
Seal Type shamir 
Initialized true 
Sealed false 
Total Shares 3 
Threshold 2 
Version 1.1.1 
Cluster Name vault-cluster-44e9a806 Cluster ID 1087afa0-c7f1-01e4-abce-187b7b5f63a3 
HA Enabled false

Authorize Juju against Vault to allow JuJu actions to run.

$ export VAULT_TOKEN=s.k5awSEin8LFTHQXy3qXM8GFv

$ vault token create -ttl=10m
--- ----- 
token s.7jChN3pD8cbLLVUSYrXqsBig 
token_accessor gj76tBhiygYh1WsK1vamP20e 
token_duration 10m 
token_renewable true 
token_policies ["root"] 
identity_policies [] 
policies ["root"] 

$ juju status

Model Controller Cloud/Region Version SLA Timestamp 

default nucctr nuc/default 2.7.1 unsupported 14:24:12+09:00 

App Version Status Scale Charm Store Rev OS Notes 

percona-cluster 5.7.20 active 1 percona-cluster jujucharms 282 ubuntu 
vault 1.1.1 active 1 vault jujucharms 32 ubuntu 

Unit Workload Agent Machine Public address Ports Message 

percona-cluster/0* active idle 0 3306/tcp Unit is 
ready vault/0* active idle 1 8200/tcp Unit is ready (active: true, mlock: enabled) 

Machine State DNS Inst id Series AZ Message 

0 started needed-guinea bionic default Deployed 
1 started first-wahoo bionic default Deployed 

Reference: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-vault.html

2. Install VaultLocker and Setting Vault

On the test machine, install vaultlocker.

$ sudo apt update
$ sudo apt install vaultlocker

Vaultlocker will need information of Vault in order to talk to Vault. The information includes app role and proper policy to create information in Vault. So in order for vaultlocker to function, we need to create an approle and assign it a policy.

Create AppRole

I will create a approle named vaultluks with no timeout for secret-id

$ vault auth enable approle
$ vault write auth/approle/role/vaultluks

$ vault read auth/approle/role/vaultluks/role-id
Key Value 
--- ----- 
role_id 7cbb1c1c-2270-6120-04f6-c3b1c753191d 

$ vault write -f auth/approle/role/vaultluks/secret-id
Key Value 
--- ----- 
secret_id f657f357-4748-dd09-733c-58ff994bd985 
secret_id_accessor 5eb5f63f-478b-4f41-a1c9-63f151991cf6  

Create Secret Backend and Policy

In order for vaultlocker to write secret to Vault, a secret backend and a policy that allows vaultlocker to access that secret backend

$ vault secrets enable -path=luks kv 
Path Type Accessor Description 
---- ---- -------- ----------- 
charm-pki-local/ pki pki_c7e293a2 Charm created PKI backend 
cubbyhole/ cubbyhole cubbyhole_ffd5c863 per-token private secret storage 
identity/ identity identity_70188f48 identity store 
luks/ kv kv_0ad49111 n/a 
secret/ kv kv_7b586ad0 n/a 
sys/ system system_0cc7f9f9 system endpoints used for control, policy and debugging 

$ vim vaultluks.hcl
path "luks/*" { 
    capabilities = ["create", "read", "update", "delete", "list"] 

$ vault policy write vault luks vaultluks.hcl 

Configure Vaultlocker and Encrypt Disk

The url, approle and secret_id should be modified accordingly.

$ sudo vim /etc/vaultlocker/vaultlocker.conf
url = 
approle = 7cbb1c1c-2270-6120-04f6-c3b1c753191d 
secret_id = f657f357-4748-dd09-733c-58ff994bd985  
backend = luks 

$ sudo vaultlocker encrypt /dev/vdb2
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 
DEBUG:urllib3.connectionpool: "POST /v1/auth/approle/login HTTP/1.1" 200 484 
INFO:vaultlocker.dmcrypt:LUKS formatting /dev/vdb2 using UUID:bed01f5f-6766-4a3d-aa22-05703568da74 
INFO:vaultlocker.dmcrypt:udevadm trigger block/add for /dev/vdb2 INFO:vaultlocker.dmcrypt:udevadm settle /dev/disk/by-uuid/bed01f5f-6766-4a3d-aa22-05703568da74 
DEBUG:urllib3.connectionpool: "PUT /v1/luks/vault/bed01f5f-6766-4a3d-aa22-05703568da74 HTTP/1.1" 204 0 
DEBUG:urllib3.connectionpool: "GET /v1/luks/vault/bed01f5f-6766-4a3d-aa22-05703568da74 HTTP/1.1" 200 866 
INFO:vaultlocker.dmcrypt:LUKS opening bed01f5f-6766-4a3d-aa22-05703568da74 
INFO:root:Enabling systemd unit for vaultlocker-decrypt@bed01f5f-6766-4a3d-aa22-05703568da74.service Created symlink /etc/systemd/system/multi-user.target.wants/vaultlocker-decrypt@bed01f5f-6766-4a3d-aa22-05703568da74.service → /lib/systemd/system/vaultlocker-decrypt@.service. 

$ lsblk
NAME                                           MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda                                            252:0    0  7.5G  0 disk
└─vda1                                         252:1    0  7.5G  0 part  /
vdb                                            252:16   0    5G  0 disk
├─vdb1                                         252:17   0  1.9G  0 part
└─vdb2                                         252:18   0  3.1G  0 part
  └─crypt-bed01f5f-6766-4a3d-aa22-05703568da74 253:0    0  3.1G  0 crypt

Now vdb2 is encrypted and its key is stored in vaults under /luks path. Here bed01f5f-6766-4a3d-aa22-05703568da74 is the UUID of the disk and it is also the key that vaultlocker uses to retrieve the encryption key.

In vault server, you can confirm the key file stored in Vault. There should be a key with the same uuid as the device and value is the encryption key.

$ vault kv list luks/

ubuntu@first-wahoo:~$ vault kv list luks/vault

$ vault kv get luks/vault/bed01f5f-6766-4a3d-aa22-05703568da74
======= Data =======
Key            Value
---            -----
dmcrypt_key    LONGLONGKEY

How to use

Everytime you want to open the partition you could use

$ sudo vaultlocker decrypt ${UUID}


$ echo -n $KEY | sudo cryptsetup --batch-mode --key-file - open \
  UUID=bed01f5f-6766-4a3d-aa22-05703568da74 \ 
  crypt-bed01f5f-6766-4a3d-aa22-05703568da74 --type luks 

Where $KEY is the variable with value got from vault

January 22, 2020

SSH Port Forwarding

Local / Remote port forwarding are powerful features of ssh, yet I am confused of the their usage. Today I saw a great picture explaining in details many options of port forwarding. The author of the accepted answers drew a so great picture that I want to copy here for my own future reference.

"a picture worth a thousand words"

Source: https://unix.stackexchange.com/questions/115897/whats-ssh-port-forwarding-and-whats-the-difference-between-ssh-local-and-remot

ssh tunnel starting from remote