Running Linux KVM on the ARM Fast Model
2014-05-20

Starting with the Cortex-A15, ARM extended the ARMv7 architecture to include virtualization extensions. In order to use KVM the Linux kernel must be booted in the new ARM HYP mode. If the kernel is booted in the traditional SVC mode then KVM will not work.

Note that KVM has been ported to ARM and already merged to Linux 3.9 for ARMv7 and merged to Linux 3.11 for ARMv8. Also the Linaro uefi-next git branch supports HYP mode so I'll be using UEFI instead of U-Boot.

For more information on the ARM virtualization extensions and how they relate to KVM and the Linux kernel see this LWN.net article.

My initial efforts to test KVM and ARM was focused on the Arndale Octa board. This board contains a Samsung Exynos 5420 containing a quad Cortex-A15 and a quad Cortex-A7 implementing ARM's big.LITTLE architecture. The original Arndale board (Samsung 5250 w/ dual A15) was the first hardware available that developers could test ARMv7 virtualization. Needless to say, the original Arndale board is heavily supported by Linaro and others. There are many guides online detailing how to get KVM up and running on that board. The Arndale Octa board does not yet have this wide support as this board is still too new and the community is catching up. Linaro has released some OpenEmbedded and Ubuntu images for the Arndale Octa but at this time no U-Boot or UEFI bootloaders enabled with HYP mode are available.

So I shifted my focus to the ARM Fixed Virtual Platforms as the Foundation platform fully supports ARMv8 and the virtualization extensions. While the Foundation platform is quite slow, no doubt KVM works as expected. Here I detail the steps required to get it running.

Any newer Linux distribution can be used to perform this KVM testing. I used an x86_64 Arch Linux installation.

Linaro Toolchain

Get the latest release of the Linaro aarch64 Linux GNU toolchain and add it to your PATH.

# mkdir -p ~/work/linaro
# cd ~/work/linaro
# wget http://releases.linaro.org/latest/components/toolchain/binaries/gcc-linaro-aarch64-linux-gnu-4.8-2014.02_linux.tar.xz
# tar -zxvf gcc-linaro-aarch64-linux-gnu-4.8-2014.02_linux.tar.xz
# ln -s gcc-linaro-aarch64-linux-gnu-4.8-2014.02_linux gcc-linaro-aarch64
# export PATH=~/work/linaro/gcc-linaro-aarch64/bin:$PATH

Linux arm64 KVM Enabled Kernel

You can use either the Linaro Tracking or LNG kernels for KVM. As of the time of this writing, the Linaro LSK kernel does not yet have KVM support merged in yet. Here we use the latest head from the Tracking kernel and build an arm64/aarch64 Linux image. Details can be found on the Building From Source tab on the Linaro release OpenEmbedded aarch64 page.

# mkdir -p ~/work/arm_kvm
# cd ~/work/arm_kvm
# git clone git://git.linaro.org/kernel/linux-linaro-tracking.git
... snip ...
# cd linux-linaro-tracking
# git branch -a
... snip ...
# git checkout ll_20140314.0
# ARCH=arm64 ./scripts/kconfig/merge_config.sh \
    linaro/configs/linaro-base.conf \
    linaro/configs/linaro-base64.conf \
    linaro/configs/distribution.conf \
    linaro/configs/vexpress64.conf \
    linaro/configs/kvm-host.conf \
    linaro/configs/kvm-guest.conf
... snip ...
# make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j4
... snip ...

The kernel image and device tree files will be located under arch/arm64/boot and arch/arm64/boot/dts respectively. Note that the same kernel will be used for both the KVM host/hypervisor and the guest.

HOST Setup (rootfs)

The HOST is the hypervisor that will be running in the ARM Foundation. We are going to use the Ubuntu Core image so we have a semi-sane package-friendly environment. Additionally, the qemu-system-aarch64 package is already available for the arm64 platform. The latest Ubuntu Core image can be found on the Ubuntu Core daily build page. We will be using the latest Trusty Tahr Ubuntu Core image.

# mkdir -p ~/work/arm_kvm
# cd ~/work/arm_kvm
# wget http://cdimages.ubuntu.com/ubuntu-core/daily/current/trusty-core-arm64.tar.gz
# qemu-img create -f raw arm_trusty_host.img 1G
# mkfs.ext4 -F arm_trusty_host.img
# mkdir mnt
# sudo mount -o loop arm_trusty_host.img mnt
# sudo tar -zxf trusty-core-arm64.tar.gz -C mnt
# sed -e 's/tty1/ttyAMA0/g' -e 's/38400/115200/g' mnt/etc/init/tty1.conf \
    | sudo tee mnt/etc/init/ttyAMA0.conf >/dev/null
# echo -e "\nauto lo0\niface lo0 inet loopback\n\nauto eth0\niface eth0 inet dhcp\n" \
    | sudo tee -a mnt/etc/network/interfaces >/dev/null
# echo 'Acquire::CompressionTypes::Order { "gz"; "bz2"; }' \
    | sudo tee mnt/etc/apt/apt.conf.d/99gzip >/dev/null
# sudo sed -in -e \
's@root:\*:@root:$6$pDWQLJGt$813e.4.vXznRlkCpxRtBdUZmHf6DnYg.XM58h6SGLF0Q2tCh5kTF2hCi7fm9NeaSSHeGBaUfpKQ9/wA54mcb51:@' \
    mnt/etc/shadow
# echo 'trusty' | sudo tee mnt/etc/hostname >/dev/null
# sudo umount mnt
# rmdir mnt

The arm_trusty_host.img file contains a 1GB sized HOST rootfs image. This image is formatted as ext4, the ttyAMA0 console has been enabled, the eth0 interface configured for DHCP, the root user's password set to root, and the hostname set to trusty.

GUEST Setup (rootfs)

The GUEST is the KVM guest that will be running on the HOST hypervisor which in turn is running in the ARM Foundation. We are, again, going to use the Ubuntu Core image so we have a semi-sane package-friendly environment. The latest Ubuntu Core image can be found on the Ubuntu Core daily build page.

# cd ~/work/arm_kvm
# qemu-img create -f raw arm_trusty_guest.img 500M
# mkfs.ext4 -F arm_trusty_guest.img
# mkdir mnt
# sudo mount -o loop arm_trusty_guest.img mnt
# sudo tar -zxf trusty-core-arm64.tar.gz -C mnt
# sed -e 's/tty1/ttyAMA0/g' -e 's/38400/115200/g' mnt/etc/init/tty1.conf \
    | sudo tee mnt/etc/init/ttyAMA0.conf >/dev/null
# echo -e "\nauto lo0\niface lo0 inet loopback\n\nauto eth0\niface eth0 inet dhcp\n" \
    | sudo tee -a mnt/etc/network/interfaces >/dev/null
# echo 'Acquire::CompressionTypes::Order { "gz"; "bz2"; }' \
    | sudo tee mnt/etc/apt/apt.conf.d/99gzip >/dev/null
# sudo sed -in -e \
's@root:\*:@root:$6$pDWQLJGt$813e.4.vXznRlkCpxRtBdUZmHf6DnYg.XM58h6SGLF0Q2tCh5kTF2hCi7fm9NeaSSHeGBaUfpKQ9/wA54mcb51:@' \
    mnt/etc/shadow
# echo 'kvmguest' | sudo tee mnt/etc/hostname >/dev/null
# sudo umount mnt
# rmdir mnt
# qemu-img convert -f raw -O qcow2 arm_trusty_guest.img arm_trusty_guest.qcow2 

The arm_trusty_guest.qcow2 file contains a 500M sized GUEST rootfs image. This image is formatted as ext4, the ttyAMA0 console has been enabled, the eth0 interface configured for DHCP, the root user's password set to root, and the hostname set to kvmguest.

Notice that the GUEST image is converted from raw to qcow2 format. This greatly reduces the file size of the image and saves time/space when transferring the image file to the HOST.

UEFI

We are going to use the pre-compiled Linaro UEFI images. These images are built from the Linaro uefi-next git repository and contain the necessary patches needed to enable HYP mode for KVM. If the CPUs aren't enabled in HYP mode properly then KVM will not work. The latest images can be found on the Linaro release OpenEmbedded aarch64 page.

# mkdir -p ~/work/arm_kvm/fastmodel
# cd ~/work/arm_kvm/fastmodel
# wget http://releases.linaro.org/latest/openembedded/aarch64/bl1.bin
# wget http://releases.linaro.org/latest/openembedded/aarch64/bl2.bin
# wget http://releases.linaro.org/latest/openembedded/aarch64/bl31.bin
# wget http://releases.linaro.org/latest/openembedded/aarch64/uefi_fvp-base.bin

Kernel Image and Device Trees

Copy the Linux kernel image and device tree over for use with the Foundation model.

# cd ~/work/arm_kvm/fastmodel
# cp -f ~/work/arm_kvm/linux-linaro-tracking/arch/arm64/boot/Image .
# cp -f ~/work/arm_kvm/linux-linaro-tracking/arch/arm64/boot/dts/fvp-foundation-gicv2-psci.dtb .
# ln -s fvp-foundation-gicv2-psci.dtb fdt.dtb

Note that the symlink named fdt.dtb must be created as this is the device tree filename pre-compiled in the UEFI image. An alternative to creating the symlink is the name could be changed in the UEFI bootloader menu during each boot.

ARM Foundation Platform

Either the Foundation or Base platform could be used for testing. We'll be using the Foundation platform here since it's free and doesn't require a license server to run. Download the Foundation model. A link to free Foundation_v8 model is at the bottom of the ARM Fast Models download page.

# cd ~/work/arm_kvm
# mv ~/Downloads/FM000-KT-00035-r0p8-52rel06.tgz .
# tar -zxf FM000-KT-00035-r0p8-52rel06.tgz

Run the HOST on the Foundation Platform

Now we can start the Foundation platform booting into UEFI and then loading the Linux Linaro Tracking kernel we built along with the HOST Ubuntu Core rootfs. Details can be found on the Binary Image Installation tab on the Linaro release OpenEmbedded aarch64 page.

# cd ~/work/arm_kvm/fastmodel
# ~/work/arm_kvm/Foundation_v8pkg/models/Linux64_GCC-4.1/Foundation_v8 \
    --cores=4 \
    --network=nat \
    --gicv3 \
    --data=bl1.bin@0x0 \
    --data=uefi_fvp-base.bin@0x8000000 \
    --block-device ~/work/arm_kvm/arm_trusty_host.img

Note that it is intentional specifying GICv3 while using the GICv2 based device tree pointed to by fdt.dtb. In this mode we're using the GICv3 controller in GICv2 compatibility mode.

When the Foundation is started a window will pop up showing the console. This window is nothing special as it's simply an xterm running a telnet session to 127.0.0.1 port 5000. If you accidently kill the xterm you can always get back to the console as the Foundation is running in the background.

In the xterm that pops up you'll see a countdown presented by UEFI. Hit the <Enter> key which will drop you into the UEFI boot menu. Here we need to change the default kernel boot string that was pre-compiled in the UEFI binaries by Linaro.

The string compiled in UEFI is:

console=ttyAMA0 earlyprintk=pl011,0x1c090000 debug user_debug=31 loglevel=9 root=/dev/vda2

We need to change it to:

console=ttyAMA0 earlyprintk=pl011,0x1c090000 debug user_debug=31 loglevel=9 root=/dev/vda rw

The reason for the change is the Ubuntu Core rootfs image we created isn't partitioned and /dev/vda2 doesn't exit. The root exists at /dev/vda.

Here is an example session showing how to navigate the UEFI menus and change the kernel command line:

The default boot selection will start in   8 seconds
<Enter>
[1] Linaro disk image on virtio
        - VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/Image
        - Arguments: console=ttyAMA0 earlyprintk=pl011,0x1c090000 debug user_debug=31 loglevel=9 root=/dev/vda2
        - FDT: VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/fdt.dtb
        - LoaderType: Linux kernel with Local FDT
-----------------------
Global FDT Config
        - VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/fdt.dtb
-----------------------
[a] Boot Manager
[b] Shell
[c] Reboot
[d] Shutdown
Start: a
[1] Add Boot Device Entry
[2] Update Boot Device Entry
[3] Remove Boot Device Entry
[4] Update FDT path
[5] Return to main menu
Choice: 2
[1] Linaro disk image on virtio
Update entry: 1
File path of the EFI Application or the kernel: Image
Keep the local FDT: [y/n] y
File path of the local FDT: fdt.dtb
Add an initrd: [y/n] n
Arguments to pass to the binary: console=ttyAMA0 earlyprintk=pl011,0x1c090000 debug user_debug=31 loglevel=9 root=/dev/vda rw
Description for this new Entry: Linaro disk image on virtio
[1] Add Boot Device Entry
[2] Update Boot Device Entry
[3] Remove Boot Device Entry
[4] Update FDT path
[5] Return to main menu
Choice: 5
[1] Linaro disk image on virtio
        - VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/Image
        - Arguments: console=ttyAMA0 earlyprintk=pl011,0x1c090000 debug user_debug=31 loglevel=9 root=/dev/vda rw
        - FDT: VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/fdt.dtb
        - LoaderType: Linux kernel with Local FDT
-----------------------
Global FDT Config
        - VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/fdt.dtb
-----------------------
[a] Boot Manager
[b] Shell
[c] Reboot
[d] Shutdown
Start: 1
   PEI   1412 ms
   DXE    456 ms
   BDS      8 ms
Total Time = 1877 ms

[    0.000000] Linux version 3.14.0-rc6 (edavis@eadmaka) (gcc version 4.8.3 20140203 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2014.02 - Linaro GCC 2014.02) ) #1 SMP Fri Mar 14 13:45:22 PDT 2014
... snip ...

The Foundation platform is slow. You'll immediately see the kernel log messages and after a couple minutes you should see a login prompt. Be patient!

Note that you'll see the following kernel log messages when the HOST boots:

kvm [1]: Using HYP init bounce page @ffd76000
kvm [1]: interrupt-controller@2c010000 IRQ25
kvm [1]: timer IRQ27
kvm [1]: Hyp mode initialized successfully

These messages detail that the CPU has been initialized properly and the kernel has been loaded in ARM HYP mode. If the kernel was loaded at another exception level then KVM would not work.

Login as the user root with the password root and install QEMU along with some other packages:

# apt-get update
... snip ...
# apt-get install qemu-system-aarch64 rsync openssh-server openssh-client
... snip ...

Since the HOST is connected to the network via NAT, I like to create an ssh backdoor just in case the console become unresponsive:

# ssh -f -N -R 8888:127.0.0.1:22 edavis@10.12.137.93

Now on the 10.12.137.93 system I can ssh to 127.0.0.1:8888 to connect to the HOST.

GUEST Setup on the HOST

Now we need to copy over the Linux Linaro Tracking kernel image, the device tree, and the GUEST rootfs image to the HOST. The following commands are performed on the HOST console:

# mkdir ~/guest
# cd ~/guest
# rsync -avP edavis@10.12.137.93:work/arm_kvm/fastmodel/Image .
# rsync -avP edavis@10.12.137.93:work/arm_kvm/linux-linaro-tracking/arch/arm64/boot/dts/rtsm_ve-aemv8a.dtb .
# rsync -avP edavis@10.12.137.93:work/arm_kvm/arm_trusty_guest.qcow2

Note that we will be using the RTS_VE-aemv8a.dtb device tree file.

Run the GUEST on QEMU/KVM

# qemu-system-aarch64 \
    -enable-kvm \
    -m 1G \
    -smp 1 \
    -cpu host \
    -machine vexpress-a15 \
    -nographic \
    -serial stdio
    -monitor telnet:127.0.0.1:4444,server,nowait \
    -drive if=none,file=arm_trusty_guest.qcow2,id=foo \
    -device virtio-blk-device,drive=foo \
    -netdev tap,id=tap0,script=no,downscript=no,ifname=tap0 \
    -device virtio-net-device,netdev=tap0 \
    -append 'console=ttyAMA0 earlyprintk=pl011,0x1c090000 loglevel=9 root=/dev/vda rw rootwait' \
    -kernel Image \
    -dtb rtsm_ve-aemv8a.dtb &

This spawns the GUEST in the background. Note that the CPU is set to host which implies the ARMv8/arm64 architecture that the HOST is running on. Also the emulated machine is vexpress-a15 which doesn't mean the CPU is a Cortex-A15. It just means the emulated platform and surrounding devices are similar to those found on the ARM Versatile Express platform. Later changes in QEMU include support for specifying the CPU as cortex-a57 and the machine as vexpress-a57.

To access the QEMU monitor:

# telnet 127.0.0.1 4445

To access the GUEST console:

# telnet 127.0.0.1 4444

Telnet to the console and you'll see the kernel log messages. After a couple minutes you should see a login prompt. Be patient! To see how much faster KVM is, run the same qemu-system-aarch64 command without the -enable-kvm switch. It takes on the order of hours to get to the login prompt instead of minutes.

Note that you'll see the following kernel log message when the GUEST boots:

Kvm [1]: Hyp Mode not available

This is expected with the GUEST kernel and it's saying that it will not be able to run as a hypervisor.

Happy KVM'ing!