Technical Notes: How to remove juju application in error state

I have been playing with juju for application deployment recently as parts of my current job. Most of the time, I just need to `juju deploy application` and JuJu gives me all settings, no matter what cloud I am using. However, in some cases, juju refuses to follow my orders, which makes me really frustrasted.

For example, the following deployment will definitely fail because grafana charm version 24 was not updated to changes in upstream grafana.

$ juju deploy grafana-24

The system will stuck with following juju status

$ juju status
Model       Controller  Cloud/Region  Version  SLA          Timestamp
experiment  stark-kvm   stark-kvm     2.6.5    unsupported  16:53:31+09:00

App      Version  Status  Scale  Charm    Store       Rev  OS      Notes
grafana           error       1  grafana  jujucharms   24  ubuntu

Unit        Workload  Agent  Machine  Public address  Ports  Message
grafana/0*  error     idle   0        10.4.23.120            hook failed: "install"

Machine  State    DNS          Inst id    Series  AZ       Message
0        started  10.4.23.120  tidy-tick  bionic  default  Deployed


When juju stucks at this stage, due to the hook error, we could not remove application. Worse, if you try `juju remove-application grafana` without --force or --no-wait flags, any subsequent command will also fail. In other word, the application refuses to be removed, until you resolve its internal errors.

My colleagues suggests 3 ways to resolve this issue in this case.

1. Resolve the issue and `juju resolve grafana/0`
2. Perform an operational hack 1: change hook script to a bash that always returns normal status code (exitcode 0).
3. Perform an operational hack 2: `juju debug-hook grafana/0`, wait for the hook context loaded, and exit immediately. This will send the positive feedback to juju controllers, let the controller perform the next action (which is our removal command).

I believe that users should be able to remove application every time they want, no matter what problem is occurring. It seems that JuJu developers thought the same so they triaged several related bugs. However, up to the writing of this blog, the issue has not been fixed yet (my version 2.6.5-bionic-amd64 still has the issue). Until the issue is fixed, there is no way but to really resolve the issue or to "hack" it.

Comments

Post a Comment

Popular posts from this blog

Technical Memo: Disk Encryption using Cryptsetup with Vault as Key Management Service

In Linux Operating system, full disk encryption could be achieved by various solutions: encryptfs, dm-crypt, cryptsetup… While a step-by-step setup of disk/partition encryption can be referred from official documentation, integration disk encryption with Key Management solution like Hashicorp Vault is unobvious. Fortunately, there are a package named vaultlocker in Ubuntu Universe repository that ease this integration. I decided to spend my free time to make cryptsetup work with Vault. Notes: A similar request has been made to Cryptsetup but it goes outside of cryptsetup so it was closed. 1. Install Vault As JuJu charm store has working Vault/MySQL charms, Vault deployment is as simple and easy as following commands $ juju deploy percona-cluster $ juju deploy vault $ juju relate vault:shared-db percona-cluster Vault needs to be initialized and unsealed before it can provide its secure services. I followed below steps to initialize Vault $ juju ssh vault/0 $ export VAU
Read more

Use MaaS to manage local computing resource

Image
Besides managing bare metal machines, MaaS (Metal as a Service) can also manage KVM guests as long as those machines are boot from network PXE. This feature allow us to efficiently manage local computing resources, while at the same time, leverage JuJu for automatically middle ware deployment. In this blog post, I will detail steps to manage KVM guest machines using MaaS. System diagram MaaS Managed Machine : MaaS Virtual Machine creates other Virtual machines through KVM Pod. All VMs access the internet through physical machine's NIC using NAT. The MaaS server will be deployed to a virtual machine running inside the physical machine. The reason for installing MaaS into a virtual machine instead of directly install to the physical machine is that we want to have the flexibility of VMs as we want to update MaaS version, or backup... The MaaS server will manage its "parent" physical machine resources through KVM pod. All machines will access to the Internet thr
Read more

Technical Memo: Convert an ova image to qcow2 format

From Wikipedia of Open Virtualization Format , I learnt that an the ova file is just a tar file of ovf directory. So I went open a Windows 10 Developer Evaluation ova file to see what exist inside it. $ tar -tf WinDev2001Eval.ova WinDev2001Eval.ovf WinDev2001Eval-disk001.vmdk WinDev2001Eval.mf So inside the ova file includes A virtual machine specification definition .ovf file A hash checksum .mf file A disk image in .vmdk image As qemu-img supports vmdk as disk image file, we actually could convert vmdk file to any other format that we like. 1. Convert .vmdk to .img $ qemu-img convert -O raw WinDev2001Eval-disk001.vmdk WinDev2001Eval.img 2. Convert .vmdk to .qcow2    $ qemu-img convert -O qcow2 WinDev2001Eval-disk001.vmdk WinDev2001Eval.qcow2 All supported formats can be confirmed by qemu-img -h The converted qcow2 image could be used to create a new virtual machine using virt-manager. I used to think that .ova file is the file format of Virtual Box (There
Read more