January 22, 2020

SSH Port Forwarding

Local / Remote port forwarding are powerful features of ssh, yet I am confused of the their usage. Today I saw a great picture explaining in details many options of port forwarding. The author of the accepted answers drew a so great picture that I want to copy here for my own future reference.

"a picture worth a thousand words"

Source: https://unix.stackexchange.com/questions/115897/whats-ssh-port-forwarding-and-whats-the-difference-between-ssh-local-and-remot

ssh tunnel starting from remote

January 20, 2020


My old desktop becomes sluggish as I deploy multiple test environment on it. As I also use various desktop applications for my daily works on the same desktop machine, sometimes resource insufficiency freeze the all the applications, interfering my work flows. I am always a fan of small low-energy NUC devices so I take this chance to buy a new NUC device solve my resource issue.

Before buying new device, I had some considerations for a NUC.
  • Whether it's good time to buy a NUC? If yes, what version should I buy?
  • Whether it works with my 4K Dell Monitor?
  • Whether it generates too much heat?
  • Whether it supports enough resource for my daily work flows (I need a lot of Linux/Windows virtual machines)
  • How I should use 

It is a good timing?
At CES 2020, Intel announced an impressive NUC 9 and it made me think for a while whether I should wait for the release of this device. However, after checking around for price/performance ratio, I realize that it's not a good time to wait as newly released device will have a price premium, which is easily over my budget. Moreover, I probably don't need a device with such high performance. So I persuade myself to go with a NUC 8. Since NUC8, it seems that the heat problem has gone much better so I should have no issue with heat issue. As I am going to build software on my desktop, for energy saving purpose I went with an Corei5 model.

Whether it works with my 4K Monitor? 
I want the NUC should support 4K resolution with at least 60Hz refresh rate. I confirmed the tech spec for my NUC at Intel Nuc Tech Spec. Any model on the list should support 4K HDR. I went to the model field in the product page and saw BOXNUC8I5BEH as the device model, so this model supports 4K. All I need is a HDR 4K HDMI cable, so I bought this NUC device with a HDMI cable from Amazon.

Does it generate heat?
Memory does not generate much heat so I bought 2 Silicon Power DDR4 Laptop Memory, whose memory size is 16GB each. I wanted to buy memory from more well-known vendors like Kingston, but as my NUC transfer speed is 2400Hz and the only memory with the same transfer speed at the time I search for is from Silicon Power, I decided to buy these 2. If it breaks, I can easy replace them as memory now is inexpensive.

Resource Sufficiency?
I don't think I will have performance bottleneck with cpu and memory resources. For hard disk, I wondered between a standard SATA SSD device and a NVME device. NVME SSD is known to generate heat but with higher read/write performance. For a NUC, a normal SATA should be fine. However, I want to see whether I have a heat bottleneck, so I decided to go for a nvme device.

With some helps from Youtube, I could install SSD and RAM without any issue. I boot the device, plugged in a bootable USB with Ubuntu LTS 18.04. The OS install goes as usual with mouse clicks.

After the device boots, I realize that by default Ubuntu only generates display resolution of 4k at 30Hz, which is unacceptable for me. As 4K resolution is unreadable for my eyes even in 27-inches display, I performed following step to generate 2K resolution at 60Hz

$ cvt 2560 1440 60

# 2560x1440 59.96 Hz (CVT 3.69M9) hsync: 89.52 kHz; pclk: 312.25 MHz Modeline "2560x1440_60.00" 312.25 2560 2752 3024 3488 1440 1443 1448 1493 -hsync +vsync 

After that I add the following lines to ~/.xprofile
xrandr --newmode "2560x1440_60.00"  312.25  2560 2752 3024 3488  1440 1443 1448 1493 -hsync +vsync<br />
xrandr --addmode DP-1 2560x1440_60.00<br />

Where DP-1 is my display device. Now my display could show 2K resolution at 60Hz.

Other settings are as usual and will be topics for another blog post.

I bought a new NUC device and I am totally satisfied with it.  

November 16, 2019


I find this picture is intriguing.

To one chicken, the other chicken is on the other side
To the viewer of this picture, we have two stupid chickens (or one?).

I find it is interesting because it's just one picture yet we could perceive it differently, depending on the place we put our perspectives.

October 7, 2019

Technical Note: LXD Database and patch SQL

I deleted some unused zfs storage pools without realizing that they are used by LXD, so today after a machine rebooted, LXD refused to startups with following log messages were output to lxd.log files

t=2019-10-07T23:02:43+0900 lvl=info msg="Initializing storage pools"
t=2019-10-07T23:02:43+0900 lvl=eror msg="Failed to start the daemon: ZFS storage pool \"juju-zfs\" could not be imported: "
t=2019-10-07T23:02:43+0900 lvl=info msg="Starting shutdown sequence"

As lxd settings are stored in dqlite database (distributed sqlite) at /var/snap/lxd/common/lxd/database/global/db.bin, so I go confirm the record settings.

sqlite> .tables
certificates                        networks
config                              networks_config
images                              networks_nodes
images_aliases                      nodes
images_nodes                        operations
images_properties                   profiles
images_source                       profiles_config
instances                           profiles_config_ref
instances_backups                   profiles_devices
instances_config                    profiles_devices_config
instances_config_ref                profiles_devices_ref
instances_devices                   profiles_used_by_ref
instances_devices_config            projects
instances_devices_ref               projects_config
instances_profiles                  projects_config_ref
instances_profiles_ref              projects_used_by_ref
instances_snapshots                 schema
instances_snapshots_config          storage_pools
instances_snapshots_config_ref      storage_pools_config
instances_snapshots_devices         storage_pools_nodes
instances_snapshots_devices_config  storage_volumes
instances_snapshots_devices_ref     storage_volumes_config
sqlite> select * from storage_pools;
sqlite> select * from storage_pools_config;

It seems that the storage pools settings are stored in 2 tables: storage_pools and storage_pools_config.

It tried to delete the related records from the above tables and restarted lxd process but lxd still failed with the same errors. I went back to to database and confirmed that those records were still there even after I deleted it. It seems that lxd recover those records from its log files. I could read the code to see how it recover but it takes time so I decided to look for database documentation in lxd source code while creating a new topic to ask for helps from lxd community.

I skimmed through lxd database documentation and found that I could create a patch.global.sql to remove unnecessary records as these sql statements are run at the very early stage of lxd startup. I created a file call patch.global.sql with statement to remove unneeded settings and start lxd.

And lxd process starts again with all my in-development containers!

Lesson learned: before removing anything, look for all its usages.

October 3, 2019

Visualize Enviro pHat sensors data

Enviro pHat is an low-cost environmental sensing boards that let you measure temperature, light color, motion and analog sensors. The accompanied python SDK makes writing program to get data from enviro pHat as easy as pie. Enviro pHat could be used in combination with RaspPi zero to become a simple room conditions monitoring sensing board.

With grafana and prometheus, it looks so easy to at least build a dashboard for environ pHat sensors data, so I decided to build a enviro pHat dashboard.

System Diagram

Though Enviro pHat provides SDK to get data from sensors, we need methods to that data to server. We also need to care about the connection between the device and server side. In home environment, I used Wifi as Pi Zero supports Wifi connections. Prometheus also provides scrape methods to fetch data from prometheus client, so I will add a prometheus client in the device side for prometheus server to scrape. The data is stored in prometheus data store and visualized by grafana.

System Diagram: Prometheus fetch data from RaspPi Zero over Wifi

Device Side

Soldering sensor board and setup OS

If you buy an un-soldered Enviro pHat, you probably need to solder it with the accompanied 2x20 pin female header. The solder part is not that difficult and Pimoroni has a detailed guide for the task. Below are my raspberry Pi Zero and the Enviro pHat sensor board.

My RaspBerry Pi Zero
Enviro pHAT board
Enviro pHat

For the PiZero device, you need to download Raspbian and write it into the SDCard. Because we are going to connect to the Rasp Pi through Wifi, we need to setup wifi for Rasp Pi. After downloading the OS image and writing the OS image to the sd card, put the sd card into your PC, open the sd card directory and added 2 following files to the boot directory.

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
        ssid="WIFI SSID"
        psk="WIFI SSID PASSWORD"

empty content

The purpose of the ssh file is to tell raspbian to enable ssh server by default.

Plugin the power supply in. You should see Raspberry Pi IP address through your home router (Or ping the whole IP address segments to see which IP is assigned).

You should be able to ssh into the raspberry Pi through its IP address. Perform basic server setups

$ ssh pi@$PI_IP_ADDRESS
pi@$PI_IP_ADDRESS password: ****
$ pi@'s password:  0.567, 3: 0.51
Linux pizero 4.19.66+ #1253 Thu Aug 15 11:37:30 BST 2019 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Oct  3 05:50:11 2019 from

$ sudo apt update
$ sudo vim /etc/hostname
$ ...
$ sudo reboot 

Final result
Final result after soldering enviro phat, setting up OS for rasp pi zero, and plugging in the power supply

Install Enviro pHat SDK

$ curl https://get.pimoroni.com/envirophat | bash
$ pip3 install prometheus_client
$ git clone git@github.com:telescreen/raspizero-playground.git
$ cd raspizero-playground
$ python3 enviro-collectd.py -d -f enviro-collectd.log --pid enviro-collectd.pid

I write a program to export environ data through prometheus client: https://github.com/telescreen/raspizero-playground . After running the program like above, you should be able to confirm the prometheus data at


Server Side

I have LXD running in my local machine. I also have Juju bootstrapped with local cloud as cloud environment. As Juju store already provided prometheus2 and grafana charms, Installation of these middlewares is simply typing following 2 commands.

$ juju deploy prometheus2
$ juju deploy granafa
$ juju add-relation prometheuss:grafana-source grafana:grafana-source

It will take a while for Juju to download / install charm and necessary software. After Juju finishes installing granafa and prometheus, we should see  both applciation active in Juju status

$ juju status
Model    Controller  Cloud/Region         Version  SLA          Timestamp
default  lxd         localhost/localhost  2.6.9    unsupported  16:19:07+09:00

App          Version  Status  Scale  Charm        Store       Rev  OS      Notes
grafana               active      1  grafana      jujucharms   32  ubuntu
prometheus2           active      1  prometheus2  jujucharms   11  ubuntu

Unit            Workload  Agent  Machine  Public address  Ports               Message
grafana/0*      active    idle   1      3000/tcp            Started grafana-server
prometheus2/0*  active    idle   0     9090/tcp,12321/tcp  Ready

Machine  State    DNS          Inst id        Series  AZ  Message
0        started  juju-63ab09-0  bionic      Running
1        started   juju-63ab09-1  bionic      Running

Next, we need to add the exposed prometheus data in the device to Prometheus's scrape job. I prepare a simple scrape job setup at


Adding the scrape job with Juju by following command

$ juju config prometheus2 scrape-jobs=@scrape-job.yaml

After 1 minute, the device IP address should appear in the scrape target with state UP

Login into grafana with password retrieved by following actions. The other actions are performed through the grafana UI and quite intuitively, so I excluded those settings from this post.

$ juju run-action --wait grafana/0 get-admin-password
  id: 43e7d1e6-3251-437b-816d-d2aa182c5554
    password: NJBMMVNqW6ssKywL
  status: completed
    completed: 2019-10-03 07:28:44 +0000 UTC
    enqueued: 2019-10-03 07:28:41 +0000 UTC
    started: 2019-10-03 07:28:44 +0000 UTC
  unit: grafana/0

Final Results

After like 3 hours, I could build the following dashboard for data captured by enviro phat's sensors. The light change is due to I put the sensors in a dark zone in my room. My room is definitely cooler than the temperature that Enviro is capturing. Probably, the temperature is affected by the heat from pi zero.

The storm is approaching my city so I am expecting to see a lower than usual air pressure. The pressure should increase back in a few days after the storm.

I was able to build a simple home conditions' monitoring system in a short time with opensource technology. I can felt more and more the importance of opensource softwares after this project.

Dashboard with temperature, air pressure and light information