PXE booting inside VirtualBox and KVM virtual machines stopwatched (1/2)

During the past days at FOSDEM, I wanted to do some work on automatic OS installs via PXE servers. Therefore, I set up a virtual machine in VirtualBox and enabled network booting. VirtualBox has a sparely documented DHCP/TFTP server built-in when you set up the network adapter in NAT mode, but for all what it’s worth it doesn’t give any useful debug information in the logs and is more convenient than flexible. Thus, I configured the VM to do host-only networking, which gives me an interface vboxnet0 at host side (configured as 192.168.56.1/24), on which I let dnsmasq listen.

Here’s the dnsmasq config:


interface=vboxnet0
bind-interfaces
no-resolv
local=/localnet/

enable-tftp
tftp-root=/srv/install

dhcp-range=192.168.56.10,192.168.56.19,12h
dhcp-match=ipxe,175
dhcp-boot=net:#ipxe,http://192.168.56.1/ipxe.pxe

I git clone‘d the latest version of iPXE (commit b757df7 as of this writing), because I wanted to chainload it for two reasons: a.) it now has menu support, so I can completely replace PXELinux and b.) Windows PE. I also set up an Apache VHost instance listening on 192.168.56.1 and serving /srv/install, seeing iPXE speaks HTTP, not just TFTP.

I then played around with various configurations and measured the time it roughly took to boot both a Linux kernel plus initrd and a Windows PE loader. The first results were as follows:

  • Intel PRO/1000MT Desktop (82540EM) with Intel PXE –> ipxe.pxe –> Kernel+Initrd via HTTP = 20 secs
  • Intel PRO/1000MT Desktop (82540EM) with Intel PXE –> ipxe.pxe –> Kernel+Initrd via TFTP = 57 secs
  • Intel PRO/1000MT Desktop (82540EM) with Intel PXE –> ipxe.pxe –> Wimload/Windows PE via HTTP = 54 secs
  • Intel PRO/1000MT Desktop (82540EM) with Intel PXE –> ipxe.pxe –> Wimload/Windows PE via TFTP = 155 secs

These results hold no surprise: HTTP is about 65% faster than TFTP. I then got curious what would happen if I chainload pxelinux.0. So I downloaded syslinux 5.01, because 5.x integrates the patches for initrdfile= support required to let wimboot load Windows PE. The results here:

  • Intel PRO/1000MT Desktop (82540EM) with Intel PXE –> ipxe.pxe –> pxelinux.0 -> Kernel+Initrd via TFTP = 75 secs
  • Intel PRO/1000MT Desktop (82540EM) with Intel PXE –> ipxe.pxe –> pxelinux.0 -> Wimload/Windows PE via TFTP = 181 secs

These results are a little bit surprising, especially if we consider what happens if we boot pxelinux.0 directly:

  • Intel PRO/1000MT Desktop (82540EM) with Intel PXE –> pxelinux.0 –> Kernel+Initrd via TFTP = 20 secs
  • Intel PRO/1000MT Desktop (82540EM) with Intel PXE –> pxelinux.0 –> Wimboot/Windows PE via TFTP = 52 secs

Obviously, pxelinux.0 operated best when it was loaded by the Intel PXE code directly (20 and 52 vs. 75 and 181 seconds). If ipxe.pxe were in the game, one would best let it load the targets itself (57 and 155 vs. 75 and 181 seconds). Since I wouldn’t know a reason for chain loading pxelinux.0 anyway, let’s focus on these numbers. 155 is still way too much seeing pxelinux.0 does the TFTP job in 52 seconds, so it seems iPXE’s TFTP code suffers from VirtualBox’s Intel PXE code.

For comparison, I did some more measurements with KVM:

  • Intel e1000 with iPXE –> ipxe.pxe –> Kernel+Initrd via HTTP = 1096 secs
  • Intel e1000 with iPXE –> ipxe.pxe –> Kernel+Initrd via TFTP = 6 secs
  • Virtio with iPXE –> ipxe.pxe –> Kernel+Initrd via HTTP = 2 secs
  • Virtio with iPXE –> ipxe.pxe –> Kernel+Initrd via TFTP = 6 secs

The first measurement I can’t explain: yes, it’s really more than 18 minutes and it’s reproducible. Maybe some buffer in the e1000 emulation is too small. I’ll ignore it for now.

The more interesting point is that with KVM iPXE can really show off, in the case of TFTP even with the e1000 emulation: 6 seconds is really amazing. Yet, the real killer value, of course, is combining Virtio with HTTP: 2 seconds for Kernel+Initrd.

Now remember the 54 and 52 seconds for Wimboot/Windows PE via HTTP above? Let’s see how that performs with KVM:

  • Intel e1000 with iPXE –> ipxe.pxe –> Wimboot/Windows PE via HTTP = >1200 secs
  • Intel e1000 with iPXE –> ipxe.pxe –> Wimboot/Windows PE via TFTP = 15 secs
  • Virtio with iPXE –> ipxe.pxe –> Wimboot/Windows PE via HTTP = 9 secs
  • Virtio e1000 with iPXE –> ipxe.pxe –> Wimboot/Windows PE via TFTP = 16 secs

So we see the same HTTP anomaly as above, but as a workaround we could comfortably use the e1000 emulation with TFTP or, if we integrated virtio drivers into Windows, Virtio with HTTP/TFTP. Keep in mind that for Wimboot/Windows PE, the last file loaded alone (winpe.wim) is of 135MB size. 135MB loaded in 9 secs, now we’re talking!

So, in essence, we would like to do Virtio and HTTP, but this result also stems from the usage of KVM which brings along its own iPXE instead of VirtualBox’s Intel PXE code. Plus, VirtualBox doesn’t support PXE booting with this Virtio-Net driver. What, if we could replace VirtualBox’s PXE ROM? Actually we can, but that’s subject of an upcoming article.

By the way: both VirtualBox and KVM, of course, allow configuration of other NIC emulations. I ran some tests with all the other VirtualBox NIC types: PRO/1000T Server (82543GC) and PRO/1000MT Server (82545EM) always made no difference to the PRO/1000MT Desktop (82540EM) emulation. With PCnet-PCI II (Am79C970A) and PCnet-Fast III (Am79C793) I got strange results: it loaded so extremely slow that I aborted testing with the only exception being the TFTP transfer of Wimboot, for which the value of 150 secs almost equals the PRO/1000MT Desktop (82540EM) one. The PCnet emulations had no problems with pxelinux.0, however:

  • PCnet-PCI II (Am79C970A) with Intel PXE –> pxelinux.0 –> Kernel+Initrd via TFTP = 17 secs
  • PCnet-PCI II (Am79C970A) with Intel PXE –> pxelinux.0 –> Wimboot/Windows PE via TFTP = 47 secs
  • PCnet-Fast III (Am79C973) with Intel PXE –> pxelinux.0 –> Kernel+Initrd via TFTP = 17 secs
  • PCnet-Fast III (Am79C973) with Intel PXE –> pxelinux.0 –> Wimboot/Windows PE via TFTP = 46 secs

These are slightly better values than with PRO/1000MT, but none that allow for clear conclusions.

For the sake of completeness, I will post results with undionly.kpxe/undionly.kkpxe later.

Leave a comment