VM-HDL Co-Simulation
FPGAs are gaining popularity as an accelerator technology to offload complex computation and data flows. The combination of programmability, high degree of parallelism, and low power consumption make FPGAs suitable for environments with rapidly changing workloads and strict power consumption limits, such as data centers. To put FPGAs into existing systems, PCIe has become the most common connection choice, due to its wide availability in server systems. Today, the majority of FPGAs in data centers are communicating with the host system through PCIe.
Unfortunately, developing applications for FPGAs requires the time-consuming FPGA compilation processes, including synthesis, place, and route. Moreover, it is challenging to develop and debug the host software and the FPGA hardware designs at the same time. The hardware designs running on the FPGAs provide little to no visibility, and even small changes to the hardware may need hours to go through the FPGA compilation process. The development process becomes even more difficult when taking operating systems into account. Changes to the operating system kernel, the loadable kernel modules, and the application software and hardware can frequently hang the system without providing enough information for debug, forcing a tedious reboot process. The combination of these problems results in long debug iterations and a slow development process.
To solve these problems, we developed a co-simulation framework using communication channels between a VM and an HDL simulator. On the VM side, we created a PCIe FPGA pseudo device to represent the FPGA board. The operating system and software running inside the VM see the same PCIe device as if they were running in a real system with an FPGA board plugged in. On the HDL side, we developed a PCIe simulation bridge to talk to the VM. The PCIe simulation bridge is pin-compatible with the PCIe block used in the physical FPGA hardware. The rest of the FPGA platform sees the same interface toward PCIe and requires no modification. To the FPGA development tools, the PCIe simulation bridge appears as a regular hardware block and has no impact on the simulation flow. Finally, we linked the VM’s PCIe FPGA pseudo device and the PCIe simulation bridge together using a high-level queue library that provides reliable message passing, which has an additional useful benefit: either side of the simulation can be independently restarted without affecting the other side.