SCSI CD-ROM driver broken in recent Windows builds

[kroese]

I reported this issue before ( #1037 ) but I think its better to start from scratch, because I understand the problem better now.

In recent Windows version, like Windows 11 24H2 (IoT) and Windows Server 2025, there is a new Setup and this one rejects the Virtio SCSI driver (wether it is WHQL signed or not).

It does not reject the driver when you manually browse to it and select it by hand. But in automatic (unattended) installations, where the drivers are loaded from a $WinPEDriver$ folder or from a unattended.xml, it refuses to load the driver.

I spend the whole day trying to debug why this is happening, but I cannot figure it out.

So can please someone of the team look into this? If there is anything I can do to help in diagnosing the issue, please let me know!

[kroese]

@vrozenfe @kostyanf14

[kroese]

I just tried it with the WHQL drivers, but the issue stays the same, so maybe its not related to signatures or certificates after all?

It is possible that they changed the way how you must slipstream drivers into an ISO, but it has worked in this manner from Vista all the way up to Windows 11, so I would be highly surprised if they suddenly changed that.

[vrozenfe]

We have attestation signed drivers targeted to be installed on Win11 24h2 and WS2025 (partially). They are not included in the latest virtio-win package available from fedorapeople site. The reason for that is mainly because our current virtio-win installer in based on dixf framework which is not compatible withe the latest Windows OSes (higher than Win 8.1) . We are working on this issue, but at the moment we cannot build the installer with the latest attestation signed drivers. But I hope we can try sharing those drivers as is, without the installer., for the preview reason only. Let me check this option.

Cheers,
Vadim.

[kroese]

@vrozenfe Hi Vadim, that is great news!! I am not using the installer at all, so if I can preview these drivers from a .zip file I would be very happy!

[kroese]

@vrozenfe Is there any news already?

[vrozenfe]

@kroese
sorry for delay.
it took me a bit longer than I expected.
please give a try to
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/upstream-virtio/virtio-win11-attestation-0.1-258.zip
it contains Win 11 attestation signed drivers for arm64 and amd64 platforms from the latest internal build (b258).

Best,
Vadim.

[kroese]

@vrozenfe Thank you Vadim, unfortunately it makes no difference for the issue.

The scsi driver fails to be automaticly loaded in both 2025 and 24H2, and when loading it manually I get:

Windows 11 Setup Windows installation encountered an unexpected error. Error code: 0x80070103 - 0x40031

Under previous Windows 11 versions they work fine.

[vrozenfe]

@kroese

Does it happen during fresh install or the driver update?
Can you post the qemu command line to try reproducing the problem?

Thanks,
Vadim.

[kroese]

@vrozenfe During fresh install. The commandline is:

-nodefaults
-cpu host,kvm=on,l3-cache=on,migratable=no,+hypervisor,hv_passthrough,+invtsc
-smp 2,sockets=1,dies=1,cores=2,threads=1
-m 4G
-machine type=q35,smm=off,graphics=off,vmport=off,dump-guest-core=off,hpet=off,accel=kvm
-enable-kvm
-global kvm-pit.lost_tick_policy=discard
-display vnc=:0,websocket=5700
-vga virtio
-monitor telnet:localhost:7100,server,nowait,nodelay
-daemonize
-D /run/shm/qemu.log
-pidfile /run/shm/qemu.pid
-name windows,process=windows,debug-threads=on
-serial pty
-device qemu-xhci,id=xhci
-device usb-tablet
-netdev tap,ifname=qemu,script=no,downscript=no,id=hostnet0
-device virtio-net-pci,romfile=,netdev=hostnet0,mac=02:BA:F0:B0:CC:13,id=net0
-drive file=/storage/windows.iso,id=cdrom0,format=raw,readonly=on,media=cdrom,if=none
-device virtio-scsi-pci,id=cdrom0b,bus=pcie.0,addr=0x5,iothread=io2
-device scsi-cd,drive=cdrom0,bus=cdrom0b.0,bootindex=10
-drive file=/storage/data.img,id=data3,format=raw,cache=none,aio=native,discard=on,detect-zeroes=on,if=none
-device virtio-scsi-pci,id=data3b,bus=pcie.0,addr=0xa,iothread=io2
-device scsi-hd,drive=data3,bus=data3b.0,channel=0,scsi-id=0,lun=0,rotation_rate=1,bootindex=3
-object iothread,id=io2
-rtc base=localtime
-global ICH9-LPC.disable_s3=1
-global ICH9-LPC.disable_s4=1
-drive file=/storage/windows.rom,if=pflash,unit=0,format=raw,readonly=on
-drive file=/storage/windows.vars,if=pflash,unit=1,format=raw
-object rng-random,id=objrng0,filename=/dev/urandom
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pcie.0,addr=0x1c

[kroese]

Upon further testing I discovered something interesting...

As said before, the vioscsi driver does not get loaded (rejected) when placed in the $WinPEDriver$ folder. Then when manually browsing to that folder, and trying to load it, it gets error-code 0x80070103 (which means: you try to install a driver that is already present) and aborts the installation.

BUT when changing that folders name to something random, for example XXX, and then manually loading the driver, it works!

That did not work with the official 0.1.248 released drivers as far as I remember, so there is at least something that is improved by the beta drivers.

But my main goal is to slipstream the drivers into the ISO, like I always did before, and that still fails.

[kroese]

@vrozenfe I tried to debug the issue with the Windows 11 24H2 Setup a bit further, by inspecting the setupact.log and setuperr.log files that are generated during the installation.

There are no obvious errors about why it ignores the vioscsi driver in the WinPE folder, the only relevant part of setupact.log seems to be:

ConX: :Compatibility: :CDeviceCompatibilityDatabase::Import: Importing devices from X:\Sources\hwcompat64.txt ...
ConX: :Compatibility: :CDeviceCompatibilityDatabase::Import: File X:\Sources\hwcompat64.txt does not exist.
ConX: :Compatibility: :CDeviceCompatibilityDatabase::Import: Importing devices from X:\Sources\hwcompat.txt ...
ConX: :Compatibility: :CDeviceCompatibilityDatabase::Import: Importing devices from X:\Sources\hwexclude64.txt ...
ConX: :Compatibility: :CDeviceCompatibilityDatabase::Import: File X:\Sources\hwexclude64.txt does not exist.
ConX: :Compatibility: :CDeviceCompatibilityDatabase::Import: Importing devices from X:\Sources\hwexclude.txt ...
ConX: :Compatibility: :CDeviceCompatibilityDatabase::Import: Importing devices from X:\Sources\duhwcompat.txt ...
ConX: :Compatibility: :CDeviceCompatibilityDatabase::Import: File X:\Sources\duhwcompat.txt does not exist.
pIsDriverBootCritical: Start type of service i8042prt: 3
pIsDriverBootCritical: Start type of service mouhid: 3
pIsDriverBootCritical: Start type of service spaceport: @
ConX: :Compatibility: :CBootCriticalDeviceList::Initialize: Detected boot-critical device (ClassGuid = {4d36e97b-e325-11ce-bfc1-08002be10318}, ServiceName = spaceport): root\spaceport
pIsDriverBootCritical: Start type of service HidUsb: 3
pIsDriverBootCritical: Start type of service storahci: @
ConX: :Compatibility: :CBootCriticalDeviceList::Initialize: Detected boot-critical device (ClassGuid = {4d36e96a-e325-11ce-bfc1-08002be10318}, ServiceName = storahci): pci\ven_ 8086&dev_2922&subsys 11001af4&rev_ 02
ConX: :Compatibility: :CBootCriticalDeviceList::Initialize: Detected boot-critical device (ClassGuid = {4d36e96a-e325-11ce-bfc1-08002be10318}, ServiceName = storahci): pci\ven 8086&dev_2922&subsys 11001af4
ConX: :Compatibility: :CBootCriticalDeviceList::Initialize: Detected boot-critical device (ClassGuid = {4d36e96a-e325-11ce-bfc1-08002be10318}, ServiceName = storahci): pci\ven_ 8086&dev_2922&cc_910601
ConX: :Compatibility: :CBootCriticalDeviceList::Initialize: Detected boot-critical device (ClassGuid = {4d36e96a-e325-11ce-bfc1-08002be10318}, ServiceName = storahci): pci\ven 8086&dev_2922&cc_0106
pIsDriverBootCritical: Start type of service i8042prt: 3
Compatibility scan finished. Categories: @x80800000

So no errors at all or a clue why it is not loading the driver. Except for one message that might be relevant (or not):

[SetupHost.exe] UnattendSearchSetupSourceDrive: Unable to convert ARC path [MULTI(@)DISK(@)CDROM(@)] to NT path; status = 0x80070002.

Then when manually installing the vioscsi driver from the $WinPEDriver$ folder, at first everything looks fine:

MOUPG Driver: Scan requested
MOUPG Driver: Scanning requested folder
IsDriverPackageSigned: File [X:\$WinPEDriver$\vioscsi\vioscsi.inf] is signed by a catalog [X:\$WinPEDriver$\vioscsi\vioscsi.cat]
IBS GatherDeviceIDsInDriverPackage: Driver package path is [X:\$WinPEDriver$\vioscsi\vioscsi.inf ]
GetModelSectionNameEx:Using section name [VirtioScsi.NTamd64.10.0...16299]
IBS AddDeviceIDsToInjectedDriverNodeHelper:Model section name is [[email protected]]
IBS GatherDeviceIDsInDriverPackage: Successfully gathered device ID‘s from [X:\$WinPEDriver$\vioscsi\vioscsi. inf]
DeviceIDPresent:Found device with ID [PCI\VEN_1AF4&DEV_1004&SUBSYS Q0081AF4&REV_00] in the list.
MOUPG Driver: Scan successfully completed.
MOUPG Driver: Starting Wait
MOUPG Driver: Install Requested
IsDriverPackageSigned: File [X:\$WinPEDriver$\vioscsi\vioscsi.inf] is signed by a catalog [X:\$WinPEDriver$\vioscsi\vioscsi.cat]
MOUPG Driver: Received driver inf path [X:\$WinPEDriver$\vioscsi\vioscsi.inf].
MOUPG Driver: Succeeded in loading driver from path [X:\$WinPEDriver$\vioscsi\vioscsi.inf].
MOUPG Driver: Install successfully completed.

but then the problems starts:

UnattendDriverInstall: Entering Execute Method
Enter ExecuteUnattendDriver
IsDriverPackageSigned: File [X:\$WinPEDriver$\vioscsi\vioscsi.inf] is signed by a catalog [X:\$WinPEDriver$\vioscsi\vioscsi.cat]
Driver: Received driver inf path [X:\$WinPEDriver$\vioscsi\vioscsi.inf].
CDlpActionDriverInstallation: :ExecuteDriverInstal1(1062): Result = Qx80070103[ gle=0x00000103 |
CDlpActionDriverInstallation: :ExecuteUnattendDriverInstall1(1393): Result = 0x80070103[gle=0x00000103 |
CDlpActionDriverInstallation: :ExecuteUnattendDriver(1272): Result = 0x80070103[gle=0x00000103 |
CDlpActionDriverInstallation: :ExecuteRoutine(284): Result = 0x80070103[gle=0x00000103 |
CDlpActionImpl<class CDlpErrorImpl<class CDlpObjectInternalImpl<class CUnknownImpl<class IDriverAction> > > >::Execute(503): Result = Q@x80070103
UnattendDriverInstall: Leaving Execute Method

From what I can find, this error code means that it cannot install the driver because it already exists (which is true, as it should already have been installed automaticly, but was ignored).

So its a still a mystery for me, my best guess as to what is happening: maybe Microsoft made a change in 24H2 to only allow boot-critical drivers to be loaded during WinPE, and the vioscsi driver is not marked as boot-critical in the catalog file for example. That would explain why it refuses to load the driver, unless you do it manually and from a different folder than the WinPE folder. Hopefully these snippets from the logfiles can aid in diagnosing the issue somehow.

[vrozenfe]

@kroese
Are you working on unintended install or trying to add virtio-win drivers to WinPE image?
Btw, how does it work on Win11 22H2/23H2 ?

Thanks,
Vadim.

[kroese]

@vrozenfe Im working on an unattend install. I use the VirtIO drivers in my project: https://github.com/dockur/windows

As you can see it has unattended installs for the VirtIO drivers for EVERY Windows version (XP/Vista/Win7/Win8/Win10/11) and every Windows Server version (2003/2008/2012/2016/2019/2022), so I did this many times before.

Exactly the same steps work fine for Win11 22H2/23H2 and every version below it.

[kostyanf14]

@vrozenfe @kroese This is very strange. We use the unattended installation for HCK-CI. Windows Server 2025/Win11 24H2 can install boot drivers successfully.

[kroese]

@kostyanf14 There are multiple ways to inject the drivers, so I am curious which one you are using? Via the $WinPEDriver$ folder in boot.wim, via the Microsoft-Windows-PnpCustomizationsWinPE section in the autounattend.xml?

Because those both fail in my case. The only method I haven't tried is via the DISM tools Add-Driver command, because I'm building these unattended ISO's on Linux, and the Linux alternative for DISM called wimlib does not support driver injection.

So I am very curious, because I have been struggling with this for weeks now, so I am very surprised you made it work?

[kostyanf14]

We used Microsoft-Windows-PnpCustomizationsWinPE section in the autounattend.xml: https://github.com/HCK-CI/HLK-Setup-Scripts/blob/master/answer-files/autounattend.xml.uefi.in#L15

[kroese]

@kostyanf14 Yes, that is the same way as I am doing it. And if I just switch the ISO between 23H2 and 24H2 (and keep everything else the same) it stops working in my case.

What are the contents in d:\drivers\ ? Is it just the extracted contents of https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.248-1/virtio-win.iso or the https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/upstream-virtio/virtio-win11-attestation-0.1-258.zip file?

[kostyanf14]

@kroese I tested with WHQLed drivers instead of the attestation signed, but yes just extracted the content of ISO.

[kroese]

@kostyanf14 Are you sure you are a SCSI drives (and a SCSI CD-ROM for the Windows ISO)? Because I can make the unattended install work with the viostor driver, but not with the vioscsi driver.

[kostyanf14]

@kroese checking right now with Windows11_InsiderPreview_Client_x64_en-us_26100.iso and virtio-win11-attestation-0.1-258.zip

[kroese]

I always get this screen:

From there I can load the SCSI driver manually, and it works. But ONLY if I didnt already try to load it unattendedly (it will display an errorcode that the driver already exists in that case). So that proofs it finds the driver, but refuses to load it. And that it works when doing it manually is maybe because it has less strict signature checking when you do it that way.

[kostyanf14]

@kroese Works.

Install QEMU CLI:

/usr/libexec/qemu-kvm -enable-kvm -machine q35,accel=kvm,vmport=off,smm=on,kernel-irqchip=split,pflash0=pflash_code,pflash1=pflash_vars  -m 8G,maxmem=8G -smp 4,cores=4  -cpu host,+x2apic,+fsgsbase,model=13,hv_spinlocks=0x1FFF,hv_relaxed,hv_vapic,hv_time -boot order=cd,menu=on  -nodefaults -no-user-config -usb -device usb-tablet -vnc :5  -global kvm-pit.lost_tick_policy=discard -rtc base=localtime,clock=host,driftfix=slew  -global ICH9-LPC.disable_s3=0 -global ICH9-LPC.disable_s4=0  -monitor telnet::10004,server,nowait -monitor vc -blockdev node-name=pflash_code,driver=file,filename=/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd,read-only=on -blockdev node-name=pflash_vars,driver=file,filename=/home/HCK-CI/workspace/hckinstall//Win11nextx64_host_uefi_q35_viommu/2024_06_05_09_35_44/uefi_0002_cl01.nvram -global driver=cfi.pflash01,property=secure,value=on -uuid cdef127c-8795-4e67-95da-8dd0a8000201 -device pcie-root-port,slot=3,chassis=1,addr=0x4,bus=pcie.0,id=root1.0 -netdev tap,id=cs_r0002_c01,vhost=on,script=/home/HCK-CI/workspace/hckinstall//Win11nextx64_host_uefi_q35_viommu/2024_06_05_09_35_44/control_ifup_0002.sh,downscript=no -device e1000e,netdev=cs_r0002_c01,mac=56:00:02:01:cc:cc,addr=03,bus=pcie.0,id=cs_r0002_c01 -drive file=/home/HCK-CI/images/HLK11_next-C1-Win11nextx64-uefi-q35-viommu.qcow2,if=none,id=ide_0002_01 -device ide-hd,drive=ide_0002_01,serial=01ide0002 -drive file=/home/HCK-CI/workspace/hckinstall//Win11nextx64_host_uefi_q35_viommu/2024_06_05_09_35_44/client01_test_image.qcow2,if=none,format=qcow2,id=virtio_scsi_0002_01 -device virtio-scsi-pci,iommu_platform=on,ats=on,id=scsi,bus=root1.0 -device scsi-hd,drive=virtio_scsi_0002_01,serial=01scsi0002 -device intel-iommu,intremap=on,aw-bits=48,caching-mode=on,device-iotlb=on -chardev socket,id=chrtpm,path=/tmp/swtpm_0002_01_sock -tpmdev emulator,id=tpm0,chardev=chrtpm -device tpm-tis,tpmdev=tpm0 -vga std -drive file=/home/HCK-CI/workspace/hckinstall//Win11nextx64_host_uefi_q35_viommu/2024_06_05_09_35_44/setup-client.iso,media=cdrom,readonly=on -drive file=/home/HCK-CI/iso/win11x64.iso,media=cdrom,readonly=on -name QemuMachine0002_CL01 -chardev socket,id=qmp,fd=11,server=off -mon chardev=qmp,mode=control

Device: -device virtio-scsi-pci,iommu_platform=on,ats=on,id=scsi,bus=root1.0 -device scsi-hd,drive=virtio_scsi_0002_01,serial=01scsi0002

[kroese]

@kostyanf14 I think i FINALLY found it.. In your config you are installing Windows from a IDE cd-rom to an IDE disk I guess?

I have always been installing Windows from a vioscsi CD-ROM to a vioscsi harddisk. So I replaced the CD-ROM to be IDE and kept the harddisk vioscsi, and now the unattended install works.

So the next question is: why does Windows suddenly refuses to load the vioscsi driver when the Windows ISO is located on a vioscsi CD-ROM device?? That is a perfectly normal scenario, and has always worked in every version until now.

[kostyanf14]

@kroese

in your config you are installing Windows from a IDE cd-rom to an IDE disk I guess?

yes, and then boot from the same QCOW2 but switch the device to vioscsi

why does Windows suddenly refuses to load the vioscsi driver when the Windows ISO is located on a vioscsi CD-ROM device??

No idea. This is a question to Microsoft.

[kroese]

To summarize the findings.. In 24H2 and Server 2025:

You can do a manual install from a scsi-cd to a scsi-hd.
You can do an unattended install from a ide-cd to a scsi-hd
You can NOT do an unattended install from a scsi-cd to a scsi-hd anymore.

I am not sure if this is a bug in the vioscsi driver that never surfaced until now, or otherwise a bug in Windows 24H2. But I will be using IDE cd-roms until it is fixed, hopefully the performance penalty will not be to noticeable.

[kostyanf14]

I will be using IDE cd-roms until it is fixed, hopefully the performance penalty will not be to noticeable.

In my tests, I don't see any problem because you need CDROM only during the installation process.

[kroese]

@kostyanf14 I know, but as the read speeds for paravirtualized virtio devices are supposed to be much higher than the speeds of emulated ide devices, my expectation was that the Windows install would complete faster when using scsi-cd.

But I never benchmarked it to be honest, so big chance that the installation speed will be bottlenecked by something else (the write speeds to the disk for example), and there will be no measurable difference in real-life.

No idea. This is a question to Microsoft.

I am really not sure if this issue can be blamed purely on Microsoft. Because there are at least 3 other drivers from 0.1.248 that cannot be installed in 24H2, like pvpanic, smbus and the viogpudo driver (gives a black screen).

So it looks more like a compatibility problem which can only be solved by the virtio-win team, instead of just accepting that they don't fully function anymore.

[kostyanf14]

I am really not sure if this issue can be blamed purely on Microsoft. Because there are at least 3 other drivers from 0.1.248 that cannot be installed in 24H2, like pvpanic, smbus and the gpudod driver (gives a black screen). So it looks more like a compatibility problem which can only be solved by the virtio-win team, instead of just accepting that they don't fully function anymore.

Please test this with 0.1.258. Win11 24H2 required new driver and build 0.1.248 is not compatible with it, but 0.1.258 is a pre-release for Win11 24H2/Server 2025.

[kroese]

@kostyanf14 Also with 0.1.258 driver I get the same black screen with virtio-vga display device as soon as the viogpudo driver is loaded, I just tested that. When using the std VGA device, it works okay. But it was just an example why Im not sure the vioscsi issue is not a problem with the driver, as other (less critical) drivers have problems too.

[kroese]

@YanVugenfirer Can you explain a bit why you applied the "Not virtio-win" label?

All we know until now is that Windows rejects the vioscsi driver for the cd-rom, but until we know WHY it rejects the driver, how can you guys be so certain its not a problem with virtio-win?

[YanVugenfirer]

@kroese Based on experience I don't think the change in driver code or INF file will fix the issue.
Labels can be always changed.

[kroese]

Problem still present in v0.1.262

[kroese]

Issue still present in v0.1.266

[krendahl-godaddy]

Any updates on this? I cannot use an IDE cdrom with our virt-install method since we are using machine type q35. Like @kroese this has worked for previous versions using vioscsi for the install ISO. Thanks

[nyawox]

@krendahl-godaddy we probably have a similar use case. Why not install from a sata cdrom to a virtio block (or virtio-scsi)? This combination worked for me. I had to limit the drivers I load on Microsoft-Windows-PnpCustomizationsWinPE to an absolute minimum (NetKVM and viostor/virtio-scsi) though. Just install the rest later during the specialization phase with pnputil. Otherwise, the setup fails very early with an error code indicating that the driver trying to load is already loaded. For reference, this worked on Windows 11 24H2 Pro, Education, and Pro Education.

edit: for error code indicating that the driver is already loaded i mean 0x80070103, probably the same one on quickemu issue

[krendahl-godaddy]

@krendahl-godaddy we probably have a similar use case. Why not install from a sata cdrom to a virtio block (or virtio-scsi)? This combination worked for me. I had to limit the drivers I load on Microsoft-Windows-PnpCustomizationsWinPE to an absolute minimum (NetKVM and viostor/virtio-scsi) though. Just install the rest later during the specialization phase with pnputil. Otherwise, the setup fails very early with an error code indicating that the driver trying to load is already loaded. For reference, this worked on Windows 11 24H2 Pro, Education, and Pro Education.

edit: for error code indicating that the driver is already loaded i mean 0x80070103, probably the same one on quickemu issue

Thank you @nyawox your tip was the key for me to get this working. I tried using the virtio-win ISO during virt-install as well as inserting the virtio drivers into both the boot.wim/install.wim in the install src ISO... but neither worked for me. Once i created a custom virtio-win ISO with only the 2k25 netkvm/viostor/vioscsi drivers the unattended install completed successfully (using a sata cdrom for install ISO as well).

It would be nice to not have to jump through hoops and just drop in the latest virtio-win ISO so i'll be keeping an eye out on this bug for future resolution.