WiX installer fails to load custom action DLL

[mtfriesen]

Hi, we recently upgraded from WiX 3 to WiX 5, and some Windows machines started failing early in the install process with the WiX 5 MSIs. This seems to be a problem loading a custom action for WixQueryOsWellKnownSID. Do you have any idea what could cause this, or how to debug further?

MSI (s) (88:64) [10:41:51:823]: PROPERTY CHANGE: Adding ACTION property. Its value is 'INSTALL'.
MSI (s) (88:64) [10:41:51:823]: Doing action: INSTALL
MSI (s) (88:64) [10:41:51:823]: Note: 1: 2205 2:  3: ActionText 
Action start 10:41:51: INSTALL.
MSI (s) (88:64) [10:41:51:823]: Running ExecuteSequence
MSI (s) (88:64) [10:41:51:823]: Doing action: FindRelatedProducts
MSI (s) (88:64) [10:41:51:823]: Note: 1: 2205 2:  3: ActionText 
Action start 10:41:51: FindRelatedProducts.
MSI (s) (88:64) [10:41:51:823]: Doing action: AppSearch
MSI (s) (88:64) [10:41:51:823]: Note: 1: 2205 2:  3: ActionText 
Action ended 10:41:51: FindRelatedProducts. Return value 1.
Action start 10:41:51: AppSearch.
MSI (s) (88:64) [10:41:51:823]: Note: 1: 2262 2: Signature 3: -2147287038 
MSI (s) (88:64) [10:41:51:823]: Doing action: Wix4QueryOsWellKnownSID_X64
MSI (s) (88:64) [10:41:51:823]: Note: 1: 2205 2:  3: ActionText 
Action ended 10:41:51: AppSearch. Return value 1.
MSI (s) (88:30) [10:41:51:823]: Invoking remote custom action. DLL: C:\windows\Installer\MSIA9C2.tmp, Entrypoint: WixQueryOsWellKnownSID
MSI (s) (88:9C) [10:41:51:823]: Generating random cookie.
MSI (s) (88:9C) [10:41:51:838]: Created Custom Action Server with PID 15024 (0x3AB0).
MSI (s) (88:70) [10:41:51:854]: Running as a service.
MSI (s) (88:70) [10:41:51:869]: Hello, I'm your 64bit Impersonated custom action server.
Action start 10:41:51: Wix4QueryOsWellKnownSID_X64.
MSI (s) (88:64) [10:41:51:901]: Note: 1: 1723 2: Wix4QueryOsWellKnownSID_X64 3: WixQueryOsWellKnownSID 4: C:\windows\Installer\MSIA9C2.tmp 
MSI (s) (88:64) [10:41:51:901]: Note: 1: 2205 2:  3: Error 
MSI (s) (88:64) [10:41:51:901]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1723 
CustomAction Wix4QueryOsWellKnownSID_X64 returned actual error code 1157 (note this may not be 100% accurate if translation happened inside sandbox)
MSI (s) (88:64) [10:41:51:901]: Note: 1: 2205 2:  3: Error 
MSI (s) (88:64) [10:41:51:901]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709 
MSI (s) (88:64) [10:41:51:901]: Product: XDP for Windows -- Error 1723. There is a problem with this Windows Installer package. A DLL required for this install to complete could not be run. Contact your support personnel or package vendor.  Action Wix4QueryOsWellKnownSID_X64, entry: WixQueryOsWellKnownSID, library: C:\windows\Installer\MSIA9C2.tmp 

Error 1723. There is a problem with this Windows Installer package. A DLL required for this install to complete could not be run. Contact your support personnel or package vendor.  Action Wix4QueryOsWellKnownSID_X64, entry: WixQueryOsWellKnownSID, library: C:\windows\Installer\MSIA9C2.tmp 
Action ended 10:41:51: Wix4QueryOsWellKnownSID_X64. Return value 3.

[chrpai]

Do you have a repro repos?

[mtfriesen]

Unfortunately I do not have repro steps - this occurs only on one of our customer's systems. They are running Windows Server 2022, but my own installation of that OS does not repro.

Are there any more detailed logs we can collect from the repro system, or any hypotheses of potential root causes I could try to simulate on another system?

[bevanweiss]

Unfortunately I do not have repro steps - this occurs only on one of our customer's systems. They are running Windows Server 2022, but my own installation of that OS does not repro.

Are there any more detailed logs we can collect from the repro system, or any hypotheses of potential root causes I could try to simulate on another system?

I'd check the Windows Event Logs..
Since this is a custom action that references a DLL which is separate from the MSI (it's contained within the MSI, but is extracted at runtime) then it's possible that an overly restrictive anti-virus / security rules could cause issues with the DLL being loaded.
Since it's not reproducing for you under the same stock OS, then it seems customer configuration of the OS is a likely culprit.

[SJFriedl]

I can describe how I just fixed this in my own code and hope it helps with yours. I'm using WIX 5

My DLL was created in C++ with Visual Studio, and it ended up being a C runtime issue: it appears that MSIEXEC is built with one version of MSVCRT while my code was generated using a different one which was not found.

One solution - as far as I understand - is to somehow bundle the C runtime required as part of packaging up your MSI, which would ensure that the runtime is available for this or any other tool requiring that runtime, and I expect to do this myself once I know more about how WIX works.

But in the interim I updated my DLL properties to use "Multi-Threaded" (static linking) and not "Multi-Threaded DLL": the former includes its own runtime stuff, though at the cost of MSI package bloat.

In Configuration Properties --> C/C++ --> Code Generation --> Runtime Library

Release build: /MT (Multi-threaded)
Debug build: /MTd (Multi-threaded Debug)

My spidey sense tells me this is not an Easy Button™ because now there are disjoint memory allocators, and it's not out of the question that one could get into trouble if memory allocated by one component (MSIEXEC or the DLL) was release by the other. I hear hints of danger music.

But this at least confirms it's a C runtime issue.

[SJFriedl]

A bit more testing to confirm the above: it's for sure the C Runtime.

When going back to the default /MD (Multi-threaded) option in a Release build, it fails as you've seen, but after installing the X64 version of the MSVC++ Redistributable, it works fine. The installer is currently found on this page:

https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170

However, the MSVC redistributable only work for Release builds, not Debug; this is intentional as I guess you should not be shipping Debug software, so the only debug MSVC are on your dev machines.

If this is a one-time problem you could likely pass the installer to the end user, otherwise the notes above could work. Alternately, ChatGPT seems to know how to create a Merge Module that would allow you to package the runtime installer with your own.

[robmen]

We statically link the C runtime in all our custom actions to avoid this problem.