Consistent udev storage names during and after OS installation: when minimal is too minimal

Getting your custom OS installation ISO to boot from both DVDs and USB sticks is but the first step in making the whole installation working from USB sticks as well. There can still be unexpected pitfalls in the course of the installation process as I encountered yesterday and while my experience was with a specific Linux distro, the actual problem behind it can affect any OS installation and is as such worth documenting in a blog post.

So I had a custom SUSE Linux Enterprise Server (SLES) 12 SP2 installation here, automated and with some addon repositories defined that bring additional software, but so far deployed from DVDs only and I was to make it support installation from USB sticks as well. Stage 1 of the installation was working just fine: it booted from either DVD or USB, found the Autoyast control file after I put it into the initrd, located the addons repos correctly because I was using relurl:// URLs to refer to their locations, installed everything on the hard disks and rebooted. But then, in stage 2, I got this error:

Stage 2 error on media not being found

The first thing I learned: Yast was obviously not only intelligent enough to replace the relurl:// URL with an URL that describes the actual installation media where it found the repos (in this case a USB stick), it also didn’t use a static reference, such as /dev/sdb1 which could break at any time if additional disks would be plugged in, but a dynamic reference, /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-usb-disk0. However apparantly exactly this reference now could not be resolved.

So I restarted the installation and took a look around on the text consoles while stage 1 was running. Clearly there was a /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-usb-disk0. I let the installation proceed to stage 2 and looked around again: nope, doesn’t exist! Instead, the USB stick was now referred to by /dev/disk/by-id/usb-QEMU_QEMU_HARDDISK_1-0000:00:09.7-3-0:0. Huh? Inconsistent udev storage names between the installation system (active during stage 1) and the installed system (stage 2)?!

At first I thought that those stages for some reason used different udev rules, but nope, one and the same /usr/share/udev/rules.d/60-persistent-storage.rules file. Maybe different properties? So I ran udevadm info /dev/sdb in both stages and yes, different properties. But why?

Next I ran udevadm info -ax which still didn’t give me direct answers but from which I could determine the sysfs path to pass to udevadm test. So I ran udevadm test sys/devices/pci0000\:00\:09.7/usb4/4-3/4-3\:1.0/host2/target2\:0\:0/2\:0\:0\:0/block/sdb and that was actually enlightening:

For stage 1:

IMPORT '/usr/bin/sg_inq --export /dev/sdb' /usr/lib/udev/rules.d/55-scsi-sg3_id.rules
starting '/usr/bin/sg_inq --export /dev/sdb'
'/usr/bin/sg_inq --export /dev/sdb'(out) 'SCSI_TPGS=0'
'/usr/bin/sg_inq --export /dev/sdb'(out) 'SCSI_VENDOR=QEMU'
'/usr/bin/sg_inq --export /dev/sdb'(out) 'SCSI_MODEL=QEMU_HARDDISK'
LINK 'disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-usb-disk0' /usr/lib/udev/rules.d/58-scsi-sg3_symlink.rules:37
LINK 'disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-usb-disk0' /usr/lib/udev/rules.d/60-persistent-storage.rules:37

For stage 2:

IMPORT builtin `usb_id` /usr/lib/udev/rules.d/60-persistent-storage.rules:31:
/sys/devices/pci0000:00/0000:00:09.7/usb4/4-3/4-3:1.0: if_class 8 protocol 6
LINK 'disk/by-id/usb-QEMU_QEMU_HARDDISK_1-0000:00:09.7-3-0:0` /usr/lib/udev/rules.d/60-persistent-storage.rules:37

So quite obviously in stage 1, where the installation system was active, there was a file /usr/lib/udev/rules.d/55-scsi-sg3_id.rules triggered a call to sg_inq which returned some data to create the /dev/disk/by-id/ link from. In stage 2, the installed system, this udev rules file apparantly didn’t exist and instead usb_id was used to construct a different link. But where did it go?

Turns out /usr/lib/udev/rules.d/55-scsi-sg3_id.rules belongs to the sg3_utils package which indeed was not installed in the installed system. Why? Because the minimal software installation pattern was used and apparantly the never-ending, often absurd discussions about how minimal a “minimal” system needs to be led to sg3_utils not to be included. Which is a bit strange, considering that obviously it was important enough to be part of the installation system, which is also intended to be “minimal”. And in the end here lead to inconsistent udev storage names…

Note that the problem at hand is not necessarily restricted to automated installations: if you’d manually add an addon product on an external USB hard drive and perform a minimal software installation, you could be affected as well, depending on whether Yast adds the repo with a /dev/disk/by-id syntax or not.

It won’t affect you, however, if you install to a USB hard drive since the paths written to /etc/fstab default to “mount by filesystem label” and could otherwise also be influenced by Autoyast options. Which, of course, is something Yast could do when resolving relurl:// paths: use /dev/disk/by-label/ paths…

For the time being, however, the obvious fix is to explictly install sg3_utils as well if you do a minimal installation.