Anyone got SW4 3.0 running on recent Macs with Apple Silicon?

[johnnlouie]

After not having success on a MacBook M2 Max under Ventura with the install instructions for 3.0 using spack, I am spiraling down a drain trying to use MacPorts to install gcc11- and trying to install each failed component package individually. Has anyone already worked out a method? If not, I'll keep plugging at it.

[andersp]

Hi John, I have not tried installing SW4 on the new Macbook, but I have one on order so I hope we can figure it out. On my current Macbook (with an Intel CPU), I have found that homebrew works better than macports for installing open source software. The MPI-wrappers for the C++ and Fortran compilers can be installed with 'brew install open-mpi'. The easiest is to install SW4 without support for OpenMP threading. Try 'make sw4 openmp=no'. Good luck!

[johnnlouie]

Thank you Anders! Those are great hints. Chelsey Assor and I will try following them and post the results.

…

On Mon, Nov 27, 2023 at 8:40 AM Anders Petersson ***@***.***> wrote: Hi John, I have not tried installing SW4 on the new Macbook, but I have one on order so I hope we can figure it out. On my current Macbook (with an Intel CPU), I have found that homebrew works better than macports for installing open source software. The MPI-wrappers for the C++ and Fortran compilers can be installed with 'brew install open-mpi'. The easiest is to install SW4 without support for OpenMP threading. Try 'make sw4 openmp=no'. Good luck! — Reply to this email directly, view it on GitHub <#204 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJXCTHX26H5FCMVUHJKI3T3YGS7BLAVCNFSM6AAAAAA73MIXNOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMOBTGE3DM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mjbajwa]

Hi @johnnlouie and @andersp,

I think I may have successfully installed SW4 on a M2 Max, without openmp. I just had to modify the EXTRA_LINK_FLAGS in make.inc to map to the correct path for gcc using homebrew, specifically:

EXTRA_LINK_FLAGS = -framework Accelerate -L/opt/homebrew/lib/gcc/13/ -lgfortran

However, running pytest/test_sw4.py fails for me with the error: optimize_mp does not exists. I assume this has to do with not installing using openmp.

I have not had the chance to fix the installation issues when trying to use open-mpi on this machine. I will write back in this thread if I get lucky...

[mjbajwa]

Sorry, I realized that after writing the above that openmp=no puts the executable in optimize folder, not the default optimize_mp.

python test_sw4.py --sw4_exe_dir=optimize seems to run fine!

[mjbajwa]

Hi, a short update on installing SW4 with mpi on a M2 Max (MacOS 14.1) that may benefit others. I'm not quite done yet, but posting in case others have ideas.

First thing I did was installed homebrew versions of openmpi and gcc/g++ as suggested by @andersp.

0. Naive call of make gives an error on <omp.h> not being found in main.h.

There are a couple issues:

1. It appears that the LLVM shipped with base MacOS doesn't support openmpi, so their base version of LLVM (i.e. in usr/local/..) and all the compilers throw errors related to mpi headers. So likely need to ensure that the process only uses compilers installed through homebrew (/opt/homebrew/..), which support openmpi.

2. When open-mpi is installed through homebrew, the configurations mpicxx refers to the base LLVM g++ or gcc. One can modify this by setting it to reference the homebrew compilers. I did this by going to /opt/homebrew/share/openmpi/mpicxx-wrapper-data.txt and setting compiler=g++-13, where g++-13 is installed in homebrew (can check the installation path with which g++-13 command). A similar thing may need to be done for mpif77 or mpif90 to use the correct fortran compilers.

At this point, my make.inc looks like the following:

etree = no proj = no FC = mpif77 CXX = mpicxx EXTRA_CXX_FLAGS = -fopenmp -L/opt/homebrew/opt/libomp/lib -I/opt/homebrew/opt/libomp/include
Now running make compiles and assembles the .C files.

However, compilation of the fortran files fail:

cd optimize_mp; mpif77 -O3 -Xpreprocessor -c ../src/rayleighfort.f

outputs:

ld: unknown options: -commons
collect2: error: ld returned 1 exit status
make: *** [optimize_mp/rayleighfort.o] Error 1

If I pass the flag EXTRA_FORT_FLAGS = -Wl,-ld_classic, which is an input to the linker (taken from googling the issues!), the fortran compiler seems to work.

But now I get errors at the final linking stage:

cd optimize_mp; mpicxx -O3 -I../src -std=c++11 -fopenmp -L/opt/homebrew/opt/libomp/lib -I/opt/homebrew/opt/libomp/include -Xpreprocessor   -I../src/double -o sw4 main.o EW.o Sarray.o version.o parseInputFile.o ForcingTwilight.o curvilinearGrid.o parallelStuff.o Source.o MaterialProperty.o MaterialData.o material.o setupRun.o solve.o Parallel_IO.o Image.o GridPointSource.o MaterialBlock.o TimeSeries.o sacsubc.o SuperGrid.o TestRayleighWave.o MaterialPfile.o Filter.o Polynomial.o SecondOrderSection.o time_functions.o Qspline.o MaterialIfile.o GeographicProjection.o Image3D.o ESSI3D.o ESSI3DHDF5.o MaterialVolimagefile.o MaterialRfile.o MaterialSfile.o AnisotropicMaterialBlock.o sacutils.o DataPatches.o addmemvarforcing2.o consintp.o oddIoddJinterp.o evenIoddJinterp.o oddIevenJinterp.o evenIevenJinterp.o CheckPoint.o geodyn.o AllDims.o Patch.o RandomizedMaterial.o MaterialInvtest.o sw4-prof.o sachdf5.o readhdf5.o TestTwilight.o TestPointSource.o curvilinear4sgwind.o TestEcons.o GridGenerator.o GridGeneratorGeneral.o GridGeneratorGaussianHill.o CurvilinearInterface2.o SfileOutput.o pseudohess.o fastmarching.o solveTT.o rhs4th3point.o MaterialGMG.o rayleighfort.o lamb_exact_numquad.o  addsgdc.o bcfortc.o bcfortanisgc.o bcfreesurfcurvanic.o boundaryOpc.o energy4c.o checkanisomtrlc.o computedtanisoc.o curvilinear4sgc.o gradientsc.o randomfield3dc.o innerloop-ani-sgstr-vcc.o ilanisocurvc.o rhs4curvilinearc.o rhs4curvilinearsgc.o rhs4th3fortc.o solerr3c.o testsrcc.o rhs4th3windc.o tw_aniso_forcec.o tw_aniso_force_ttc.o velsumc.o twilightfortc.o twilightsgfortc.o tw_ani_stiffc.o anisomtrltocurvilinearc.o scalar_prodc.o updatememvarc.o addsg4windc.o bndryOpNoGhostc.o rhs4th3windc2.o dqags.o dqagse.o  dqaws.o  dqawse.o  dqc25s.o dqcheb.o  dqelg.o  dqk15w.o  dqk21.o  dqmomo.o dqpsrt.o  dqwgts.o  qaws.o  qawse.o  qc25s.o qcheb.o  qk15w.o  qmomo.o  qpsrt.o  qwgts.o xerror.o d1mach.o r1mach.o

Error:

Undefined symbols for architecture arm64:
  "__gfortran_concat_string", referenced from:
      _xsetf_ in xerror.o
      _xsetua_ in xerror.o
  "__gfortran_date_and_time", referenced from:
      _timestamp_ in xerror.o
  "__gfortran_os_error_at", referenced from:
      _lambexact in lamb_exact_numquad.o
  "__gfortran_st_read", referenced from:
      _timestamp_ in xerror.o
  "__gfortran_st_read_done", referenced from:
      _timestamp_ in xerror.o
  "__gfortran_st_write", referenced from:
      _lambexact in lamb_exact_numquad.o
      _j4save_.part.0 in xerror.o
      _i1mach_.part.0 in xerror.o
      _xerprt_.constprop.0 in xerror.o
      _xerabt_ in xerror.o
      _xersav_ in xerror.o
      _xerprt_ in xerror.o
      ...
  "__gfortran_st_write_done", referenced from:
      _lambexact in lamb_exact_numquad.o
      _j4save_.part.0 in xerror.o
      _i1mach_.part.0 in xerror.o
      _xerprt_.constprop.0 in xerror.o
      _xerabt_ in xerror.o
      _xersav_ in xerror.o
      _xerprt_ in xerror.o
      ...
  "__gfortran_stop_numeric", referenced from:
      _d1mach_ in d1mach.o
      _r1mach_ in r1mach.o
  "__gfortran_stop_string", referenced from:
      _j4save_.part.0 in xerror.o
      _i1mach_.part.0 in xerror.o
      _xerabt_ in xerror.o
      _xerhlt_ in xerror.o
      _d1mach_ in d1mach.o
      _r1mach_ in r1mach.o
  "__gfortran_string_index", referenced from:
      _xerprn_ in xerror.o
  "__gfortran_string_len_trim", referenced from:
      _xerprn_ in xerror.o
  "__gfortran_transfer_character_write", referenced from:
      _lambexact in lamb_exact_numquad.o
      _j4save_.part.0 in xerror.o
      _i1mach_.part.0 in xerror.o
      _xerprt_.constprop.0 in xerror.o
      _xerabt_ in xerror.o
      _xersav_ in xerror.o
      _xerprt_ in xerror.o
      ...
  "__gfortran_transfer_integer", referenced from:
      _timestamp_ in xerror.o
  "__gfortran_transfer_integer_write", referenced from:
      _lambexact in lamb_exact_numquad.o
      _j4save_.part.0 in xerror.o
      _xersav_ in xerror.o
      _xerrwv_ in xerror.o
      _xersve_ in xerror.o
      _xermsg_ in xerror.o
      _timestamp_ in xerror.o
      ...
  "__gfortran_transfer_real_write", referenced from:
      _lambexact in lamb_exact_numquad.o
      _xerrwv_ in xerror.o
  "_dgbsv_", referenced from:
      __ZN7QsplineC1EiPdddiidddd in Qspline.o
  "_dgels_", referenced from:
      __ZN2EW18setup_viscoelasticEv._omp_fn.0 in EW.o
      __ZN2EW26reverse_setup_viscoelasticEv._omp_fn.0 in EW.o
      __ZN6Source15get_cc_psourcesEP2EWidddbPdRSt6vectorIP15GridPointSourceSaIS5_EE in Source.o
      __ZN6Source15get_mr_psourcesEP2EWidddbPdRSt6vectorIP15GridPointSourceSaIS5_EE in Source.o
  "_dgesv_", referenced from:
      __ZN2EW37geodyn_second_ghost_point_curvilinearERSt6vectorI6SarraySaIS1_EES4_S4_S4_dS4_S4_i._omp_fn.2 in geodyn.o
      __ZN2EW28bcsurf_curvilinear_2nd_orderEiiiiiiiR6SarrayPd in geodyn.o
      __ZN2EW14bcfortanisg_ciEiiiiiiPiiiiPddP21boundaryConditionTypeS1_S1_S1_S1_S1_S1_S1_S1_S1_S1_._omp_fn.7 in bcfortanisgc.o
      __ZN2EW14bcfortanisg_ciEiiiiiiPiiiiPddP21boundaryConditionTypeS1_S1_S1_S1_S1_S1_S1_S1_S1_S1_._omp_fn.6 in bcfortanisgc.o
      __ZN2EW20bcfreesurfcurvani_ciEiiiiiiiPdS0_iS0_S0_S0_S0_S0_._omp_fn.1 in bcfreesurfcurvanic.o
      __ZN2EW20bcfreesurfcurvani_ciEiiiiiiiPdS0_iS0_S0_S0_S0_S0_._omp_fn.0 in bcfreesurfcurvanic.o
  "_dgetrf_", referenced from:
      __ZN21CurvilinearInterface211init_arraysERSt6vectorIPdSaIS1_EES4_RS0_I6SarraySaIS5_EES8_S8_ in CurvilinearInterface2.o
  "_dgetri_", referenced from:
      __ZN21CurvilinearInterface211init_arraysERSt6vectorIPdSaIS1_EES4_RS0_I6SarraySaIS5_EES8_S8_ in CurvilinearInterface2.o
  "_dspev_", referenced from:
      __ZN2EW9computeDTEv._omp_fn.1 in setupRun.o
      __ZN2EW17checkanisomtrl_ciEiiiiiiPdS0_RdS1_S1_S1_._omp_fn.0 in checkanisomtrlc.o
      __ZN2EW7maxwaveEPddRd in computedtanisoc.o
      __ZN2EW11maxwavecurvEPdddRd in computedtanisoc.o
ld: symbol(s) not found for architecture arm64
collect2: error: ld returned 1 exit status

So I'm sort of stuck now. I don't know enough about the Fortran compilers to know where to go next. But I do wonder if there is a magic flag there that would help.

[houjun]

can you try setting "FC = gfortran" instead of mpif77 to see if SW4 compiles?

[mjbajwa]

Hi @houjun, assume you were referring to me.

No luck. I tried it with and without EXTRA_FORT_FLAGS = -Wl,-ld_classic. It's still failing at the linking stage:

*** Configuration file: ' configs/make.inc ' ***
********* User configuration variables **************
debug=no  profile=no  hdf5=  proj=no  SW4ROOT=
CXX= mpicxx EXTRA_CXX_FLAGS= -fopenmp -L/opt/homebrew/opt/libomp/lib -I/opt/homebrew/opt/libomp/include
FC= gfortran  EXTRA_FORT_FLAGS= -Wl,-ld_classic
EXTRA_LINK_FLAGS=
******************************************************
cd optimize_mp; mpicxx -O3 -I../src -std=c++11 -fopenmp -L/opt/homebrew/opt/libomp/lib -I/opt/homebrew/opt/libomp/include -Xpreprocessor   -I../src/double -o sw4 main.o EW.o Sarray.o version.o parseInputFile.o ForcingTwilight.o curvilinearGrid.o parallelStuff.o Source.o MaterialProperty.o MaterialData.o material.o setupRun.o solve.o Parallel_IO.o Image.o GridPointSource.o MaterialBlock.o TimeSeries.o sacsubc.o SuperGrid.o TestRayleighWave.o MaterialPfile.o Filter.o Polynomial.o SecondOrderSection.o time_functions.o Qspline.o MaterialIfile.o GeographicProjection.o Image3D.o ESSI3D.o ESSI3DHDF5.o MaterialVolimagefile.o MaterialRfile.o MaterialSfile.o AnisotropicMaterialBlock.o sacutils.o DataPatches.o addmemvarforcing2.o consintp.o oddIoddJinterp.o evenIoddJinterp.o oddIevenJinterp.o evenIevenJinterp.o CheckPoint.o geodyn.o AllDims.o Patch.o RandomizedMaterial.o MaterialInvtest.o sw4-prof.o sachdf5.o readhdf5.o TestTwilight.o TestPointSource.o curvilinear4sgwind.o TestEcons.o GridGenerator.o GridGeneratorGeneral.o GridGeneratorGaussianHill.o CurvilinearInterface2.o SfileOutput.o pseudohess.o fastmarching.o solveTT.o rhs4th3point.o MaterialGMG.o rayleighfort.o lamb_exact_numquad.o  addsgdc.o bcfortc.o bcfortanisgc.o bcfreesurfcurvanic.o boundaryOpc.o energy4c.o checkanisomtrlc.o computedtanisoc.o curvilinear4sgc.o gradientsc.o randomfield3dc.o innerloop-ani-sgstr-vcc.o ilanisocurvc.o rhs4curvilinearc.o rhs4curvilinearsgc.o rhs4th3fortc.o solerr3c.o testsrcc.o rhs4th3windc.o tw_aniso_forcec.o tw_aniso_force_ttc.o velsumc.o twilightfortc.o twilightsgfortc.o tw_ani_stiffc.o anisomtrltocurvilinearc.o scalar_prodc.o updatememvarc.o addsg4windc.o bndryOpNoGhostc.o rhs4th3windc2.o dqags.o dqagse.o  dqaws.o  dqawse.o  dqc25s.o dqcheb.o  dqelg.o  dqk15w.o  dqk21.o  dqmomo.o dqpsrt.o  dqwgts.o  qaws.o  qawse.o  qc25s.o qcheb.o  qk15w.o  qmomo.o  qpsrt.o  qwgts.o xerror.o d1mach.o r1mach.o
Undefined symbols for architecture arm64:
  "__gfortran_concat_string", referenced from:
      _xsetf_ in xerror.o
      _xsetua_ in xerror.o
  "__gfortran_date_and_time", referenced from:
      _timestamp_ in xerror.o
  "__gfortran_os_error_at", referenced from:
      _lambexact in lamb_exact_numquad.o
  "__gfortran_st_read", referenced from:
      _timestamp_ in xerror.o
  "__gfortran_st_read_done", referenced from:
      _timestamp_ in xerror.o
  "__gfortran_st_write", referenced from:
      _lambexact in lamb_exact_numquad.o
      _j4save_.part.0 in xerror.o
      _i1mach_.part.0 in xerror.o
      _xerprt_.constprop.0 in xerror.o
      _xerabt_ in xerror.o
      _xersav_ in xerror.o
      _xerprt_ in xerror.o
      ...
  "__gfortran_st_write_done", referenced from:
      _lambexact in lamb_exact_numquad.o
      _j4save_.part.0 in xerror.o
      _i1mach_.part.0 in xerror.o
      _xerprt_.constprop.0 in xerror.o
      _xerabt_ in xerror.o
      _xersav_ in xerror.o
      _xerprt_ in xerror.o
      ...
  "__gfortran_stop_numeric", referenced from:
      _d1mach_ in d1mach.o
      _r1mach_ in r1mach.o
  "__gfortran_stop_string", referenced from:
      _j4save_.part.0 in xerror.o
      _i1mach_.part.0 in xerror.o
      _xerabt_ in xerror.o
      _xerhlt_ in xerror.o
      _d1mach_ in d1mach.o
      _r1mach_ in r1mach.o
  "__gfortran_string_index", referenced from:
      _xerprn_ in xerror.o
  "__gfortran_string_len_trim", referenced from:
      _xerprn_ in xerror.o
  "__gfortran_transfer_character_write", referenced from:
      _lambexact in lamb_exact_numquad.o
      _j4save_.part.0 in xerror.o
      _i1mach_.part.0 in xerror.o
      _xerprt_.constprop.0 in xerror.o
      _xerabt_ in xerror.o
      _xersav_ in xerror.o
      _xerprt_ in xerror.o
      ...
  "__gfortran_transfer_integer", referenced from:
      _timestamp_ in xerror.o
  "__gfortran_transfer_integer_write", referenced from:
      _lambexact in lamb_exact_numquad.o
      _j4save_.part.0 in xerror.o
      _xersav_ in xerror.o
      _xerrwv_ in xerror.o
      _xersve_ in xerror.o
      _xermsg_ in xerror.o
      _timestamp_ in xerror.o
      ...
  "__gfortran_transfer_real_write", referenced from:
      _lambexact in lamb_exact_numquad.o
      _xerrwv_ in xerror.o
  "_dgbsv_", referenced from:
      __ZN7QsplineC1EiPdddiidddd in Qspline.o
  "_dgels_", referenced from:
      __ZN2EW18setup_viscoelasticEv._omp_fn.0 in EW.o
      __ZN2EW26reverse_setup_viscoelasticEv._omp_fn.0 in EW.o
      __ZN6Source15get_cc_psourcesEP2EWidddbPdRSt6vectorIP15GridPointSourceSaIS5_EE in Source.o
      __ZN6Source15get_mr_psourcesEP2EWidddbPdRSt6vectorIP15GridPointSourceSaIS5_EE in Source.o
  "_dgesv_", referenced from:
      __ZN2EW37geodyn_second_ghost_point_curvilinearERSt6vectorI6SarraySaIS1_EES4_S4_S4_dS4_S4_i._omp_fn.2 in geodyn.o
      __ZN2EW28bcsurf_curvilinear_2nd_orderEiiiiiiiR6SarrayPd in geodyn.o
      __ZN2EW14bcfortanisg_ciEiiiiiiPiiiiPddP21boundaryConditionTypeS1_S1_S1_S1_S1_S1_S1_S1_S1_S1_._omp_fn.7 in bcfortanisgc.o
      __ZN2EW14bcfortanisg_ciEiiiiiiPiiiiPddP21boundaryConditionTypeS1_S1_S1_S1_S1_S1_S1_S1_S1_S1_._omp_fn.6 in bcfortanisgc.o
      __ZN2EW20bcfreesurfcurvani_ciEiiiiiiiPdS0_iS0_S0_S0_S0_S0_._omp_fn.1 in bcfreesurfcurvanic.o
      __ZN2EW20bcfreesurfcurvani_ciEiiiiiiiPdS0_iS0_S0_S0_S0_S0_._omp_fn.0 in bcfreesurfcurvanic.o
  "_dgetrf_", referenced from:
      __ZN21CurvilinearInterface211init_arraysERSt6vectorIPdSaIS1_EES4_RS0_I6SarraySaIS5_EES8_S8_ in CurvilinearInterface2.o
  "_dgetri_", referenced from:
      __ZN21CurvilinearInterface211init_arraysERSt6vectorIPdSaIS1_EES4_RS0_I6SarraySaIS5_EES8_S8_ in CurvilinearInterface2.o
  "_dspev_", referenced from:
      __ZN2EW9computeDTEv._omp_fn.1 in setupRun.o
      __ZN2EW17checkanisomtrl_ciEiiiiiiPdS0_RdS1_S1_S1_._omp_fn.0 in checkanisomtrlc.o
      __ZN2EW7maxwaveEPddRd in computedtanisoc.o
      __ZN2EW11maxwavecurvEPdddRd in computedtanisoc.o
ld: symbol(s) not found for architecture arm64
collect2: error: ld returned 1 exit status
make: *** [sw4] Error 1

[houjun]

ah, yes, I was replying to you @mjbajwa, another thing to try is to set CXX=g++, FC=gfortran, and add the MPI include and library paths and library flags manually.

[mjbajwa]

@houjun thank you for the tip.

I also just found this, which is the same error I'm getting when using gfortran without the flags -Wl,-ld_classic. I'll test it shortly.