I like fresh, reproducible installs when a new version of your preferred Linux distro (in my case openSUSE) comes out. Ensuring the perceived (!) feeling that a system is a “clean” state is hard enough during its lifecycle already and ensuring it is so after doing an upgrade installation is even harder. That’s why I don’t really like upgrades and, even more so, prefer scripted, automatic installations and configuration management tools over manual installations even if it’s additional work in the beginning.
For VMs I have developed a concept of parallel installs: when Version $NEW of my Linux distro is released, I fire up a new VM with an automatic install based on $NEW that automatically syncs all important data from an existing VM running $OLD, test it, and if things work out (they usually do) I simply change the DNS record to point the $NEW VM and turn off the one with $OLD. So the VMs themselves are not pets, they’re cattle.
This works fine for VMs in the cloud where I don’t have to care about the VM host but at home/on-premise I run a VM host myself. And neither do I have a second identical physical machine that I could do a parallel install on nor would I want to physically move data here or move underlying storage.
So instead, I have to take the bullet once in a while and risk reinstalling it from scratch in a semi-automated fashion. Semi-automated as, for the VM host only, I intentionally still have to manually confirm the beginning of the installation, just in case that my Autoyast Pre-Scripts mis-detected hard disks or some other Yast proposal seems off.
I could have an endless debate on how far a configuration management tool such as Ansible should manage a system and where the thin red line to “user data” that is beyond automatic configuration begins. In the case of VM hosts, I think the virtual machines, their virtual networks and also storage pools should be considered user data. I considered the option of having Ansible also configure e.g. a Mail server VM for me but a.) this contradicts my parallel install approach described above and b.) I may also have defined some VMs ad-hoc for testing purposes. So if I reinstall the VM host from scratch I would have to backup libvirt configuration manually. Which, of course, contradicts my automation approach.
So instead, I looked at having the semi-automated OS install also backup libvirt configuration files, that is, definitions of:
- virtual machines
- virtual networks
- storage pools
The actual disk images would be beyond the scope because they reside on separate storage that is not be formatted, e.g. a VM partition. The libvirt configuration itself, e.g. /etc/libvirt/libvirtd.conf, is also not covered here since that it’s clearly subject to configuration management and not “user data”.
I initially solved this entirely through Autoyast Pre- and Chroot-Scripts but as I saw the existing Yast installation module to import (= keep) SSH configurations and keys from existing Linux installations, I was pondering taking the Yast team up on their promise of Yast’s Open source nature and extensibility and try to develop a similar module for Libvirt and see how hard it gets for an “outsider” (I’m not actively engaged in openSUSE or Yast and I’m also not proficient in Ruby).
Turns out: not hard at all. May I present to you: yast-libvirt-import! What does it do?
First, it scans existing Linux partitions for libvirt configuration files, that is, XML files below /etc/libvirt/storage (storage pools), /etc/libvirt/qemu (virtual machines) and /etc/libvirt/qemu/networks (virtual networks) as well as related symlinks indicating desired autostart on VM host boot. This happens during the preparation phase in which Yast generates an installation proposal, i.e. along steps such as setting up the desired partitioning, software to install etc.:
Afterwards the results are shown as part of the installation proposal summary (don’t be confused that it’s the only text in English language here, I didn’t bother localizing it to integrate with the other German language texts):
Remember that, as mentioned above, I’m requiring explicit confirmation of the installation proposal here although it is an otherwise automated installation. In a fully automated installation you wouldn’t see this screen at all.
My initial implementation simply always kept all detected configuration files. But where would be the fun in that? I’d miss out on all the glory of GUI design! Thus I also implemented a dialog showing more details that is reachable by clicking the “Import libvirt configuration files” link:
Here the admin can use the checkboxes at the bottom to specify for each detected configuration file whether he wants to import it (keep it for the new installation) and can also change the Autostart behavior, i.e. whether the corrosponding virtual machine, virtual network or storage pool will be started at boot time. Double-clicking a table entry will also toggle the Import flag.
Inbetween I also tried a variant of this dialog using three different tables for each supported type of configuration file but it was suggested that this would look too complex:
He got a point there. Also the single table approach was easier to implement.
When the admin finally confirms the installation, the selected configuration files will then be automatically restored at their original locations as part of the installation process and before libvirtd starts for the first time. That is, I do not use virsh define and the libvirt API here but simply take the same approach a file-based backup would take. Note that yast-libvirt-import itself does not install libvirt and also doesn’t force libvirt to be part of Yast’s software selection: how and where libvirt gets installed is at the admin’s discretion. In my case it’s part of my Ansible “libvirtd” role.
So, looks cool? Of course, there must be a catch, right? Yep, there is! I wrote I developed a custom Yast module, meaning that it is not part of openSUSE (yet?) and while I put the sources on Github, I see some big question marks as to whether it makes sense to even propose upstreaming it:
- would it be useful for a sufficient number of people (what number would that be? 100? 1000? 10000?) or is this too “exotic” of a use case?
- what would justify such a special module for libvirt and wouldn’t that mean we’d also need special modules for the Apache webserver, OpenLDAP and other apps?
- wouldn’t it thus make more sense to somehow integrate a generic backup and restore mechanism for files in /etc?
I wrote this module primary because I have a perfect use case for it but it’s hard for me to tell whether this applies to others, too, and what the answers to the questions above would look like. I’d probably say that libvirt is special because it’s the only software you’ll run directly on physical hardware these days and everything else can run in virtual machines. But then you’d (hello Stefan!) probably counter that virtual machines are so 1999ish and everyone is running containers these days anyway and so maybe there should be a yast-k8s-import… but let me know what you think, whether it makes sense to propose integration into openSUSE. And if anyone from the Yast team reads this, I’d be even more interested in knowing your opinion!
So if it isn’t part of openSUSE and certainly not part of the latest Leap 15.2 release, how did I still manage to integrate it? I’ll be dedicating an upcoming separate post to answer this question and also talk a bit about the structure of the module and answer the “was it hard?” question.