For the upcoming stages of QEMU SPARC64 for sun4v architecture I started to think about the possibility of enabling I/O, but for some weeks I didn’t really know where to start on this milestone.

The Legion (Sun software emulator) firmware blob from the OpenSPARC T1 archive, loaded by QEMU, only supports the hsimd ramdisk-based driver, without any network (honestly, I’d never heard of this driver’s existence before QEMU).

The driver is practically just mapped to a file located in RAM: it’s a ramdrive, nothing more. Any other I/O modification hits a wall: we need to work with the hypervisor.

Some info about the sun4v hypervisor

A sun4v LDOM implementation is a genuinely classic Type-1 hypervisor… so classic, in fact, that it still makes me laugh a little whenever Xen (with Dom0/DomU separation it’s more relevant to Type-1, but I/O is still alive on Dom0) or even KVM get called that.

Okay, but what does hypervisor mean on the sun4v arch? I know it as an axiom we’ve used for the past 21 years, since T1 was introduced: every piece of code on T-based servers runs under the hypervisor.

For this project, either path, LDC (Logical Domain Channels - bus between guest and host in LDOM terminology) or a real PCIe device (the real choice also noted below), needed the same thing first: a hypervisor with enabled I/O properties.

The sun4v hypervisor, aka the q.bin file, is written almost entirely in assembly language, and it’s small.

No, honestly, it’s tiny, and it lives outside the OBP blob:

-rwxrwxr-x 1 dpim dpim 216912 Aug  4 13:42 q.bin

And at this point you might be thinking: great, so just skip the hypervisor and wire a PCIe root complex straight into QEMU’s sun4v machine like any other QEMU-related PCI emulation?

Not a chance.

A bit of concept

And yet it’s the thing doing the real work: not just the LDC virtual bus, but PCI itself, config-space access, MMIO, talking to the IOMMU, interrupt routing, and of course mapping CPUs and PCI devices through the machine description (MD, see my previous posts), all handled by the hypervisor. It’s also the layer responsible for the actual memory mapping: real address to physical address (RA→PA).

Which leads to a genuinely strange result: what you’re building is nominally a guest domain description… but it’s paradoxically also the physical domain, because the “primary” domain simply owns every physical resource on the machine, entirely through its MD (after reconfiguration, of course it can “give” part of these resources (i.e memory, pci buses, cpus…) to another guest, or several). So, a guest domain can end up with its own PCI root complex, but touching it costs the processor real hypercalls and real interrupts: one call for config space, a different one for the IOMMU, another set again for MSI and interrupt delivery.

Return to the reality was painful: the T2 archive also carries the OpenSPARC T1 (Ontario/T2000) FPGA builds, just not a Legion build, which is the one QEMU actually needs.

After looking at the OpenSPARC T2 archive source (never touched by the QEMU community) I was surprised: both PCIe (aka “Fire” aka px nexus) and LDC I/O source code, the full MSI/MSIQ support, and a more advanced IOMMU, all sitting right there. But of course the I/O was completely disabled too (the firmware blob was built with -UCONFIG_FIRE and -UCONFIG_FIRE_EBUS in q.bin).

So: T1 has the oldest hypervisor generation, no LDC, real Fire hardware access switched off, but its own declarations for Legion/Ontario are correct and complete: a real, working q.bin actually ships in that tree. T2 has much more modern I/O code, but no correct, shipped declarations for the T1/Ontario/Legion combination specifically: only a Makefile sits there, no binary was ever built for it.

In principle, any assembler that understands the SPARC v9 hyperprivileged instructions can build it.

But the dialect matters here: you can’t just swap Sun’s own qas/Sun Studio toolchain for stock GNU binutils, because stock GNU as simply doesn’t speak qas’s dialect.

Sun’s source leans hard on assembler macros. Delete one, or get it slightly wrong, and the result assembles cleanly, links cleanly, and jumps to the wrong place at runtime: the worst kind of bug, since nothing upstream of “it runs” ever complains.

That’s what makes this a chicken-egg problem :) — to build SPARCv9 code, you need SPARCv9 environment :) — which is exactly what Sun/Oracle’s own readme says.

The build saga

Building the “cadaver” from the OpenSPARC T1/T2 source images took some time. The missing macros got reimplemented for GNU’s assembler, but then came the harder question: how do you know a reimplemented macro is actually right, with no working reference build to compare against and no real hardware to test on? Surprisingly, the answer was already sitting in the source tree.

OpenSPARC’s release didn’t ship source alone: it included some of Sun’s own compiled object files, leftovers of Sun’s internal build, still sitting there untouched for ~20 years. Those became ground truth. GNU-built object files, diffed against Sun’s originals, function by function, checking for semantic equivalence rather than a byte-identical match. Several hundred functions checked out. It’s an unusual way to validate code…

With the macros fixed and the sections landing where they belonged, q.bin built… linked…

The first run, and the first surprise

And then it needed somewhere to actually talk about its own state, which led to something I hadn’t seen before: it turns out there’s a SECOND SERIAL PORT that nobody had ever noticed, and the hypervisor reports its crashes straight to it during initial builds.

Alive and well ...
Strand start set = 0x1
Total physical mem = 0x0
Scrubbing the rest of memory
Number of strands = 0x1
membase = 0x400000
memsize = 0x3bfffc0
physmem = 0x0
 done


            Hypervisor 2.0 (LDoms capable + console)

<...ended with...>

Setting remaining details
Start heart beat for control domain

With the macros fixed, the sections landed correctly, and a console finally listening, q.bin ran. It boots!

The choice: LDOM vs. PCI

We started discussing this with Claude and spent the next few days estimating effort for each path: which way would be better for us, LDC or PCI? The reasoning turned out to be stranger than I thought: it seems the LDC path turned out to be the more expensive route to emulate, not the cheaper one.

A channel-based transport means an entire protocol built from nothing on the host side: packet framing, cookie/handshake state machines, plus a disk server (a QEMU-side vds-like backend) and a network switch (vsw-like) that actually speak LDC. None of that exists anywhere in QEMU today. And most of the guest-side code already runs inside the hypervisor (vdc/vnet); you’d still need to emulate the vsw/vds part on the QEMU backend as well. (I spent a day or two trying to make a one-domain “short-circuit” prototype vds/vdc pair in /devices, but it never worked correctly.)

On the other side, the PCIe device model turned out cheaper for a simple reason: most of the work belongs to someone else already. QEMU ships a mature, heavily exercised Intel e1000 model…

PCI shows signs of life into the device tree

With a self-built hypervisor and the PCIe bet in hand, the first concrete milestone was visible from OBP itself, with zero OBP changes (I still use the OBP blob from OpenSPARC T2): once the (correctly-built) hypervisor answered the PCI config-space hypercalls and the machine description declared a PCIe arc (yes, described in MD), OBP’s own, unmodified, Sun-shipped PCI probing code walked it and built /pci@0!

And this is really the whole point of the exercise: the bridge between PCIe and QEMU’s own guest machine. From there it was a matter of finishing that bridge: real MMIO windows, so config-space and device registers actually map through to something outside the guest, via a plain -device e1000,netdev=net0. Yes, Intel’s PCI VID (8086) landed here after some modifications on the QEMU side (mostly byte-swapping, hello big-endianness, for the PCI properties)

ok cd /pci@0/ethernet@0
ok .properties
assigned-addresses       82000010 00000000 00100000 00000000 00020000
                         81000014 00000000 00000000 00000000 00000040
reg                      00000000 00000000 00000000 00000000 00000000
                         02000010 00000000 00000000 00000000 00020000
                         01000014 00000000 00000000 00000000 00000040
compatible               pci8086,100e.1af4.1100.3
                         pci8086,100e.1af4.1100
                         pci1af4,1100
                         pci8086,100e.3
                         pci8086,100e
                         pciclass,020000
                         pciclass,0200
name                     ethernet
devsel-speed             00000000
latency-timer            00000000
max-latency              00000000
min-grant                00000000
interrupts               00000001
cache-line-size          00000010
class-code               00020000
subsystem-id             00001100
subsystem-vendor-id      00001af4
revision-id              00000003
device-id                0000100e
vendor-id                00008086