Technical Memo: Apparmor starting errors in Ubuntu Bionic 18.04

1. System Degradation Issue

As usual, after installing some modules and modifying some user privileges, I performed a reboot for my home desktop computer. After the reboot, I always check for system start health and today is no different.

$ systemctl status
● stark
    State: degraded
     Jobs: 0 queued
   Failed: 1 units
    Since: Tue 2019-04-23 13:37:13 JST; 0h 4min ago
   CGroup: /
           ├─user.slice
           │ ├─user-1000.slice
           │ │ ├─user@1000.service
... 
 

2. What was the problem?

hm... I was quite surprised at this degradation of my system, so I look for the unit that fails to start at boot time.

$ systemctl list-units --failed
  UNIT             LOAD   ACTIVE SUB    DESCRIPTION           
● apparmor.service loaded failed failed AppArmor initialization

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

$ systemctl status apparmor
● apparmor.service - AppArmor initialization
   Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-04-24 10:06:09 JST; 5min ago
     Docs: man:apparmor(7)
           http://wiki.apparmor.net/
  Process: 20197 ExecStop=/etc/init.d/apparmor stop (code=exited, status=0/SUCCESS)
  Process: 20208 ExecStart=/etc/init.d/apparmor start (code=exited, status=123)
 Main PID: 20208 (code=exited, status=123)
...

 4月 24 10:06:07 stark apparmor[20208]: Warning from /var/lib/snapd/apparmor/profiles/snap.lxd.migrate (/var/lib/snapd/apparmor/profiles/snap.lxd.migrate line 484): Unconfined exec qualifier (ux) allows some dangerous environment variables to be passed to the unconfined process; 'man 5 apparmor.d' for details.
 4月 24 10:06:09 stark apparmor[20208]:    ...fail!
 4月 24 10:06:09 stark systemd[1]: apparmor.service: Main process exited, code=exited, status=123/n/a
 4月 24 10:06:09 stark systemd[1]: apparmor.service: Failed with result 'exit-code'.
 4月 24 10:06:09 stark systemd[1]: Failed to start AppArmor initialization.

Apparmor seems fail to initialize at startup. It's quite surprising because I didn't modify any apparmor-related things. I look for the status of apparmor to see what problem might happen.

Well, nothing special here. the apparmor service is complaining of some unconfined exec qualifier and it dies. Warnings cannot make a process fail to start because they are warnings! So I look at apparmor journal logs.

$ journalctl -u apparmor
...
 4月 23 16:25:41 stark apparmor[14865]: AppArmor parser error  for /etc/apparmor.d/usr.sbin.dhcpd in /etc/apparmor.d/usr.sbin.dhcpd at line 70: Could not open 'dhcpd.d'

The error at line 70 of profile file for dhcpd is the culprit causing apparmor failed to start. At that line is the following #include

  # allow packages to re-use dhcpd and provide their own specific directories
  #include

It seems that this apparmor's dhcpd profile looks for dhcpd.d directory in /etc/apparmor.d/ but fails to see this directory. So on theory, if I create this directory again, the apparmor parser will parse and I will not see any problem.

3. Temporary Solution

So I go and create the directory

$ sudo mkdir /etc/apparmor.d/dhcpd.d
$ sudo systemctl start apparmor
$ systemctl status apparmor
● apparmor.service - AppArmor initialization
   Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2019-04-24 10:50:18 JST; 5s ago
     Docs: man:apparmor(7)
           http://wiki.apparmor.net/

  Process: 20197 ExecStop=/etc/init.d/apparmor stop (code=exited, status=0/SUCCESS)
  Process: 11297 ExecStart=/etc/init.d/apparmor start (code=exited, status=0/SUCCESS)
 Main PID: 11297 (code=exited, status=0/SUCCESS)
...

$ systemctl status
● stark
    State: running
     Jobs: 0 queued
   Failed: 0 units

Ok, it works!

4. Why?

I try to google around to see whether I'm the only person who see this issue. I did found out similar issue date back to 2015. The reason at that time was that Ubuntu developers forgot to add the empty directory dhcpd.d to the built package. However, the issue got fixed and released at 2.8.95~
2430-0ubuntu5.2. My apparmor package version was 2.12-4ubuntu5.1 (apt-cache showpkg apparmor) so it should be no issue at all. I'm curious to see whether my apt update affects this issue. so I issue a search command for apparmor to apt log

$ grep apparmor /var/log/apt/term.log
Skipping profile in /etc/apparmor.d/disable: usr.sbin.squid
Installing new version of config file /etc/apparmor.d/usr.lib.snapd.snap-confine.real ...

2 promising lines did appear but it didn't relate to my issue.
At this time, I still need to finish my new employee training plan, so I leave this issue as it is. If due to some modification, the issue reappears, I will definitely raise a bug ticket to Ubuntu Launchpad.

Comments

Post a Comment

Popular posts from this blog

Technical Memo: Disk Encryption using Cryptsetup with Vault as Key Management Service

In Linux Operating system, full disk encryption could be achieved by various solutions: encryptfs, dm-crypt, cryptsetup… While a step-by-step setup of disk/partition encryption can be referred from official documentation, integration disk encryption with Key Management solution like Hashicorp Vault is unobvious. Fortunately, there are a package named vaultlocker in Ubuntu Universe repository that ease this integration. I decided to spend my free time to make cryptsetup work with Vault. Notes: A similar request has been made to Cryptsetup but it goes outside of cryptsetup so it was closed. 1. Install Vault As JuJu charm store has working Vault/MySQL charms, Vault deployment is as simple and easy as following commands $ juju deploy percona-cluster $ juju deploy vault $ juju relate vault:shared-db percona-cluster Vault needs to be initialized and unsealed before it can provide its secure services. I followed below steps to initialize Vault $ juju ssh vault/0 $ export VAU
Read more

Use MaaS to manage local computing resource

Image
Besides managing bare metal machines, MaaS (Metal as a Service) can also manage KVM guests as long as those machines are boot from network PXE. This feature allow us to efficiently manage local computing resources, while at the same time, leverage JuJu for automatically middle ware deployment. In this blog post, I will detail steps to manage KVM guest machines using MaaS. System diagram MaaS Managed Machine : MaaS Virtual Machine creates other Virtual machines through KVM Pod. All VMs access the internet through physical machine's NIC using NAT. The MaaS server will be deployed to a virtual machine running inside the physical machine. The reason for installing MaaS into a virtual machine instead of directly install to the physical machine is that we want to have the flexibility of VMs as we want to update MaaS version, or backup... The MaaS server will manage its "parent" physical machine resources through KVM pod. All machines will access to the Internet thr
Read more

Technical Memo: Convert an ova image to qcow2 format

From Wikipedia of Open Virtualization Format , I learnt that an the ova file is just a tar file of ovf directory. So I went open a Windows 10 Developer Evaluation ova file to see what exist inside it. $ tar -tf WinDev2001Eval.ova WinDev2001Eval.ovf WinDev2001Eval-disk001.vmdk WinDev2001Eval.mf So inside the ova file includes A virtual machine specification definition .ovf file A hash checksum .mf file A disk image in .vmdk image As qemu-img supports vmdk as disk image file, we actually could convert vmdk file to any other format that we like. 1. Convert .vmdk to .img $ qemu-img convert -O raw WinDev2001Eval-disk001.vmdk WinDev2001Eval.img 2. Convert .vmdk to .qcow2    $ qemu-img convert -O qcow2 WinDev2001Eval-disk001.vmdk WinDev2001Eval.qcow2 All supported formats can be confirmed by qemu-img -h The converted qcow2 image could be used to create a new virtual machine using virt-manager. I used to think that .ova file is the file format of Virtual Box (There
Read more