Making NetBSD as a fast(er) booting microvm By: Emile Heitor

YouTube Video

This YouTube presentation details a project to significantly speed up NetBSD kernel boot times, particularly on AMD64 architecture. Key points include:

Project Goals & Background:

• The overarching goal is to make NetBSD boot as fast as possible.
• The presenter, Emil, has a long history with NetBSD (since 1998) and has worked on various projects aimed at making it smaller and faster, including a NetBSD live USB key and a minimal containerization system called Sailor.
• Inspiration came from AWS Firecracker’s fast boot times and Colin Percival’s work on FreeBSD.

Technical Approach & Challenges:

• The project leverages PVH (Paravirtualized Host), a virtualization mode that utilizes hardware acceleration (Intel VT-x or AMD-V) and paravirtualized drivers (like virtio) for faster boot times, bypassing much of the BIOS initialization process.
• Initial attempts involved patching the NetBSD kernel to support PVH, adapting techniques used in FreeBSD. This involved modifying the lcore.S assembly code, the very low-level initial kernel startup code.
• A major hurdle was the incorrect assumption about the location of the physical memory address table (pointed to by EBX) passed from the VMM to the kernel. The solution involved copying this table to the location expected by the kernel.
• A new “genPVH” VM guest type was implemented to handle PVH boot in a more generic way, abstracting away Xen-specific details.
• The presenter used GDB and QEMU’s debug mode to troubleshoot kernel boot issues.
• Further optimizations involved replacing methods for getting CPU frequency, improving timer calibration, and removing unnecessary delays.

Results & Future Work:

• Successful NetBSD boot in PVH mode with QEMU and Firecracker was achieved, reaching boot times of ~20ms (down from ~373ms). This involved using the MMIO (memory-mapped I/O) driver for device access in Firecracker, which doesn’t expose a PCI bus.
• There are ongoing efforts to improve the MMIO driver and handle bus DMA cleanly.
• A website (smallnbsd.org) provides the patched kernel and related resources.
• Future work includes aiming for integration into NetBSD 11. A proof-of-concept for running NetBSD microVMs in Kubernetes is also mentioned.

Overall: The project successfully achieved dramatic improvements in NetBSD boot times through a combination of leveraging PVH, addressing kernel code issues, and implementing performance optimizations. The presenter highlights the collaborative nature of the project, drawing inspiration and code from Colin Percival’s work on FreeBSD.