Error in macOS 15.0: pointer being freed was not allocated (uhd #778) #59
Comments
joergho
changed the title
|
Thanks for the report! I quickly scanned the linked UHD bug report and nothing has jumped out to me yet. I'll have to look more closely when I have a little bit more time later this week to form some theories. Certainly it doesn't hurt to try rebuilding just to see if something conda-forge-wide has changed. The way conda-forge works, I can't modify the Azure agent and expect to keep compatibility with other packages, but I can "re-render" the recipe to update the CI scripts to the latest versions distributed by conda-forge, so I'm trying that in #60. Fresh build logs will also give us something to inspect to look for problems. |
|
Hi! Thanks @ryanvolz for starting the re-rendering in PR #60. I see that there is a change in uhd-feedstock/.scripts/run_osx_build.sh Lines 15 to 19 in 5535245 What is your further plan for this PR? Merge it if all tests pass? If I understand it correctly, the new build should then be available in conda-forge soon after the PR was merged, correct? |
|
Yes, I will merge it if the build succeeds and then it will be available for folks to test. If that doesn't just fix it, then we can work from there. |
|
BTW, while we are already talking: We (UHD maintainers) plan to take your patches 0002...0006 mainline, if you don't mind. |
|
@ryanvolz Long time! I hope you're doing well & thanks for keeping RadioConda going! My primary wondering right now is whether the provided Python 3.12, which is built using (I'm guessing Apple) Clang 14 can work with the provided UHD if it is not built with the same Clang. In MacPorts we fixed the various compilers to use the same underlying libraries so that they can be intermixed with very high success rate; IDK if the same applies to Conda. I haven't been active on MacPorts for nearly 2 years, so I had to reinstall everything from scratch yesterday, but once done UHD worked as expected -- both the executables as well as the Python API. I'm guessing compiling everything from source helps :) Today I will move the current MP installs to a separate directory & have MP install from precompiled binary to see if that works. |
Sure! Saves me from having to figure out what's broadly useful and remember to submit them...
Thanks for leading the way in the Apple world on MacPorts since before Radioconda was even a thing! I remember it being really helpful to see what you'd done in MacPorts in getting everything to work in conda-forge initially. Especially since I'm not myself a Mac user!
Without still having looked into it further, I do suspect something along these lines. The conda-forge Python and compiler stack maintainers definitely take pains with things like that, but I haven't been tracking conda-forge closely enough over the past few months to know if anything has changed. That's why my primary hope is that they've already done some hard work that will fix it and be picked up in the rebuilt package (that should be available in 30-60 minutes). |
|
@ryanvolz I followed the instructions in your uhd-feedstock for "Installing uhd". This provides Forge python3.12, which is compiled with clang 16.0.6 & is consistent with that used to compile UHD. This install performs "import uhd" just fine. This reinforces my belief that it's a mixed compiler issue & also that this is not a UHD issue. |
|
Thanks! I've upgraded to the newest build of Unfortunately, this issue persists. (Copied from: EttusResearch/uhd#778 (comment)) |
|
@doctormin what is the text printed out when executing "python3.12". Here's what I see: "Python 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:54:21) [Clang 16.0.6 ] on darwin". Note that the compiler here is Clang 16 whereas the compiler for UHD in yours is Clang 18. While it is possible to mix compilers even after building, the background libraries have to be consistently use for both compiler builds, so that the compilers can build interoperable binaries. MacPorts does all of this intentionally so that they can intermix various Clang both from Apple and MacPorts built along with different GCC and GFORTRAN. MacPorts provides prebuilt binaries where possible; some code just has to be compiled locally for it to work & they note this in the build script. I have never used Conda / Forge / Mamba (etc) before this, so I have no idea how everything works ... in this specific case meaning: (1) how is the source built to create the binaries to distribute? (2) can we build from source locally? (3) can we control which prebuilt binary we install via a selection of the compiler used to build it? (4) are common libraries used for all compilers (a la MacPorts)? IDK the answer to (1); for the rest my gut says: no, no, and no ... basically it is up to the end-user to make sure all of the needed alignment is in place. Most of the time this works seamlessly, but when it fails it fails poorly (not robustly). Just gut beliefs: I might be totally wrong. As note: I have never used Conda / Forge / Mamba (etc) before this |
|
I looked through the most recent build log and didn't see anything that would point me to problems. There are some compiler warnings, but I think that's typical. Maybe one of those is relevant though and I'm not catching it.
(1) There's a build matrix that covers the different platforms and Python versions. The build follows the recipe in this feedstock. Packages end up with dependencies that restrict the installation to compatible builds, ensuring that e.g. the C++ stdlib is compatible. Conda-forge also has ecosystem-wide "pins" to ensure that the versions of dependencies that we build against are uniform with everything else (that's one reason to re-render and re-build, to pick up updates in the global "pins"). (2) Not as easily as something like MacPorts, but it is possible to use (3) One can install particular builds of packages by specifying the full version+build string, but there's not an easy way to select by what compiler a package was built with. (4) Yes, common libraries are used. The way this may fall apart is if the library versions that are assumed to be compatible and are manifested in dependencies aren't actually compatible. This can happen if the upstream makes an incompatible change without appropriately changing the version number, or if the conda-forge build has a subtle problem, or if the conda-forge maintainers made a compatibility assumption that is incorrect. It's harder to get and keep (4) right with MacOS arm64 since the free CI infrastructure hasn't thus far allowed automated testing of those packages. Hopefully that is close to changing. My next step will be to see if I can find any similar reports in other conda-forge packages that could point to a wider problem. I'm suspicious of that since the important change seems to be between MacOS versions. Any further clues from folks who can test this would be most welcome. |
|
Hi @ryanvolz , I just spotted In the log file. The cmake command line arguments sets this to |
I think |
|
Another thing I just thought of: I've known Does the uhd 4.6 package show this error? If it does, that would probably rule out |
|
|
|
@ryanvolz thanks for the info on Conda about background libraries. That's great! I'm assuming the background libraries work for both Conda and Apple compilers ... yes? This could still be a more general compiler issue, but I agree with you that it's most likely something to do with PyBind11 as the interface between UHD and Python. Maybe mixing and matching system and Conda versions? I'm not a PyBind11 expert by any means to know how it works and what it links to. |
|
Using miniforge for all installs seems to provide working UHD 4.7 <-> Python 3.12; at least "import uhd" works |
I think conda-forge optionally supports GCC as well, but I don't think it's widespread. So they've probably done the work to make things as compiler-agnostic as practical. I'm not sure about Apple compilers. It's uncommon for anyone to build locally for use within a conda environment and even have the ability to use a different compiler.
I just reminded myself that despite the change in uhd's CMake build script, I've actually been using the external conda-forge pybind11 for quite a while, so testing uhd 4.6 probably won't tell us whether it's a pybind issue or not. |
You need to do more than the import: the crash happens in the last line. I tried with uhd 4.6, same issue. |
|
Ah ... OK it used to be that "import uhd" caused the crash, so we've progressed beyond the basics now :) Yes issuing 'usrp = uhd.usrp.MultiUSRP()' in UHD 4.6 causes the noted crash for me. Let me try UHD 4.7 |
|
Double checked now & the above works fine within MacPorts. The Python examples also work when using the Python that UHD was installed to use (python3.12 from MacPorts; not the system Python) |
|
Forced to use python=3.13.1, bug still there: |
|
@kgarrels can you run "lldb python3.12" and do the same & then when it breaks to "bt" and provide that listing here? |
|
oh my, what is that? Like so...? This is the default python in conda, |
|
@kgarrels great; thx! Which version of Boost is UHD linked to? |
|
duh: Boost_108200 |
|
hmmm ... let me try that Boost version in MacPorts ... the debug log I have is basically identical to yours so maybe the issue is not Boost |
|
hmm ... no Boost 1.82 or newer in MP yet ... can you try building UHD with Boost 1.81? that's working for me in MP |
|
we also had this earlier |
|
did that work in Python all the way to instantiating the USRP object? The boost version is 1.84 there ... looking for 1.81 |
|
it also crahed...
ok, you want to go back? I can try, not today, I hope tomorrow afternoon. |
|
I'll try a build with Boost 1.81 in #61. |
|
I will look into getting Boost 1.82 into MacPorts, even just a local install, to see if I get issues with it |
|
Boost 1.82 works fine. Running |
|
problem does not occur with uhd=4.1.0.5, and starts to happen with udh=4.2.0.0: |
The output seems to indicate that the transition was uhd 4.1 to 4.2. Very interesting result. It looks like all of the packages are the same between those two installs, except for |
|
uhd release notes for 4.2, uhdboost? cmake |
I noted it wrong, this is correct, corrected above: |
|
I have my eye on EttusResearch/uhd@2c7ce2d for having the potential to cause subtle issues, so my next build in #61 will patch that out. |
|
I everything is installed correctly with the latest versions, could I "brute force" replace uhd with the 4.1.0.5 version with conda? |
I'm not sure quite what you mean. If you're not using anything that compiles against the uhd library, then you can absolutely force install uhd==4.1.0.5 and just use that. But if you want to use the latest version of a package compiled against uhd (e.g. gnuradio), then you wouldn't be able to use the pre-built package because it was built against uhd 4.7. That said, there is some older version of gnuradio that was compiled against uhd 4.1, so if you're happy using that older version then that will work fine. |
yes, that was my plan, does not work then...tricky. |
ok, how could I install it? |
@kgarrels The artifacts are available in this build: uhd-feedstock / 20241206.2. Otherwise, follow the instructions from Ryan's comment |
|
works! You are all geniuses! |
|
also on my other machine, failure and resolution: |
|
no success with python=3.10: |
That looks like a dependency error caused by installing the package file directly (which bypasses dependency resolution). I'm guessing that will be fixed with a proper installation. I'll merge the test build so it goes live on conda-forge, then you can upgrade as normal and verify that it works. |
|
The latest build should be available through conda-forge now, so please test and report back at your convenience. Thanks! |
|
Thanks. I will try a fresh install of radioconda. |
|
status:
|
|
If I don't install boost, but libboost-devel libboost-headers, it works. Maybe something with libboost-python? Installing boost does this: |
|
Ah, I think the new package is still fine. Notice that when you install |
|
ok, this works perfectly now. |
|
Thanks @ryanvolz for finding a solution! |
Solution to issue cannot be found in the documentation.
Issue
Hi @ryanvolz ,
in EttusResearch/uhd#778 we (NI) have a bug report that there is a crash when using macOS 15.0 (it worked for macOS 14.x) when executing the following code:
Console prints:
Exact line in the code where the crash happens: https://github.com/EttusResearch/uhd/blob/a5ed1872be6d0fc36de9a7e0b508933da1f119bc/host/python/uhd/usrp/multi_usrp.py#L36
The current assumption is that building the application with Xcode is causing the issue. Another user has found that a similar problem exists with gr-osmosdr: EttusResearch/uhd#778 (comment)
When the application is built using cmake, then no crash happens. When it is built using Xcode, it crashes with the same error as above.
What is your opinion on this? Do you think a rebuild of the uhd conda package (maybe a toolchain was updated without causing a rebuild trigger?) could solve the issue (I think it might be at least worth a try)? Or would it be okay if I create a test-PR which explicitly uses a macOS 15 agent for building (https://github.com/conda-forge/uhd-feedstock/blob/main/.azure-pipelines/azure-pipelines-osx.yml#L8) so that we can check if the generated artifacts work?
Installed packages
Environment info
The text was updated successfully, but these errors were encountered: