Just update all the dependencies

[liamhuber]

There was an old lady who swallowed a fly...

pyiron_base can't update until we update pympipool (#867), pyiron_atomistics can't update until we update base (#868), and pyiron_atomistics being out of date is causing our daily tests to fail because we import StructureFactory which imports MaterialsProjectFactory and then the outdated mp_api has some conflict with some other dependency on pydantic.

However, as discussed in more detail here, I fully expect the major version bump to cause failures where pyiron_contrib.nofiles and pyiron_contrib.tinybase rely on pympipool -- if this PR passes tests, it will only be because the tests weren't good enough!

@jan-janssen and @pmrv, nofiles and tinybase are owned by you, respectively; could you please look into updating them to pympipool 0.7.0 syntax so we can close off this dependency update chain?

[github-actions]

👈 Launch a binder notebook on branch pyiron/pyiron_contrib/update_pympipool

[liamhuber]

Ok, the unit tests fail, but in the usual way with the pydantic issue. Disturbingly, neither nofiles nor tinybase show up in the failures. I took a very brief peek at the source code, and for nofiles I guess this is not surprising -- the only test there is for lammps, so I don't think the tests ever import the problematic pyiron_contrib.nofiles.master. It disturbs me a bit that our tests won't catch major version bumps destroying syntax. For tinybase there is no direct test of pyiron_contrib.tinybase.executor, but it does show up in the import for pyiron_contrib.tinybase.job, so I think we would see it cause a failure in the tinybase notebooks, but the pydantic error kills things before we can get there -- @pmrv you'll be better able to assess that.

[liamhuber]

Instead of trying to close off this chain and rely on the dependabot, what about cutting to the chase and updating pympipool, pyiron_base and pyiron_atomistics all at once together here?

[pmrv]

I'll have a look next week, but unless the import location changed or pympipool doesn't adhere to the executor protocol anymore, I don't think anything should break, since the only thing [we're doing](
from pympipool import PoolExecutor) is importing and instantiating the thing.

[pmrv]

Ah, I see, the import path does change. Should be a quick fix then.

[niklassiemer]

That is what I usually do for pyiron_base and pyiron_atomistics. I release a new version of base, immediately make the conda-forge PR and merge it, and update pyiron_atomistics to this version 1-2h later once the new version is ready.

[liamhuber]

Summary

pyiron_atomistics directly specifies the dependency mp-api 0.36.1, which does not try to import anything from pydantic_settings; pyiron_contrib is installing atomistics from conda, and thus getting >= instead of == so we wind up with mp-api 0.37.0, which does try to import from pydantic_settings. (In both cases we wind up with pydantic 1.10.13.)

So mp-api starts explicitly depending on the package pydantic-settings starting in 0.37.0, but only implicitly gets that dependency by demanding emmet-core[all]==0.39.8 in its requirements (which in turn specifies the pydantic-settings dependency).

The trouble is that we are somehow winding up with emmet-core==0.38.0, which does not specify a dependency on pydantic-settings, so it never gets installed.

For the life of me, I can't figure out what's causing us to get stuck on emmet-core 0.38.0.

Full process

...
File "/usr/share/miniconda3/envs/my-env/lib/python3.10/site-packages/mp_api/client/core/settings.py", line 6, in <module>
    from pydantic_settings import BaseSettings
ModuleNotFoundError: No module named 'pydantic_settings'

This error is coming from where we touch pyiron_atomistics, so why does atomistics not have a similar problem??

Ok, so pyiron_atomistics directly specifies that we need mp-api 0.36.1 and is getting that in the latest tests (on this PR), but when we install pyiron_atomistics here it's from conda, so pyiron_atomistics dependences get a >= instead of == and we wind up with mp-api 0.37.0. Neither contrib nor atomistics depends directly on pydantic, and both wind up getting pydantic 1.10.13.

Indeed, moving from mp-api 0.36.1 to 0.37.1 changes the import from pydantic import BaseSettings, Field to from pydantic import Field; from pydantic_settings import BaseSettings. However, for both of these the specified pydantic requirement is simply >=1.8.2! So why should 1.10.13 be causing trouble for either??

Neither the pydantic 1.8.2 setup file nor requirements, nor those for 1.10.13 make any reference to pydantic-settings. It does appear in the toml file for the latest v2 of pydantic. Is it possible that mp-api is just missing an explicit dependency on pydantic-settings?

Indeed, I'm actually a little suspicious that the latest version of mp-api is the problem. Their lower-bound of pydantic 1.8.2 was released on 11 May 2021, but the PR introducing pydantic-settings didn't get merged until 7 September 2022.

Aha aha, but they do require their package emmet-core[all]==0.39.8, and the setup.py for the latest emmet-core requires pydantic-settings.

This brings us back to our pip-check failure here:

maggma 0.46.1 has requirement pyzmq==22.3.0, but you have pyzmq 25.1.1.
mp-api 0.37.1 has requirement emmet-core>=0.69.2, but you have emmet-core 0.68.0.

And sure enough, if I look at the setup for 0.68.0 it's missing pydantic-settings, but it's present for 0.69.2! It's true, in the unit failing unit tests, one can see emmet-core 0.68.0 is the one getting installed.

So, why is emmet-core not the latest and greatest?

I don't find "emmet" mentioned directly anywhere in the pyiron repository. Since it's a materialsproject package, I searched to see if there is another of their projects specifying emmet-core== that we depend on (or emmet-core[all]). atomate2 and crystaltoolkit showed up, but we don't appear to depend (directly) on either of these. Nor does pymatgen depend indirectly on crystaltoolkit, so so far a dead end here.

If we knew who was sticking us with emmet-core==0.68.0 and could upgrade that, this should resolve itself. I'm also not super pleased with the mp-api devs introducing an explicit dependency and not including it in their requirements, but I guess that's life.

[liamhuber]

Well that was a rabbit hole and a half. Now I'm going to shotgun @jan-janssen @pmrv and @niklassiemer to ask if any of you have any idea where we might be (indirectly) specifying an emmet-core dependency??

[jan-janssen]

@jan-janssen and @pmrv, nofiles and tinybase are owned by you, respectively; could you please look into updating them to pympipool 0.7.0 syntax so we can close off this dependency update chain?

The nofiles modules are superseded by the atomistics package https://github.com/pyiron/atomistics so I removed the corresponding classes in pyiron_contrib: #873

[jan-janssen]

To me maggma 0.46.1 has requirement pyzmq==22.3.0, but you have pyzmq 25.1.1. sounds like a critical issue, as we use pyzmq in the pympipool, but we can relax this dependency if necessary.

[liamhuber]

To me maggma 0.46.1 has requirement pyzmq==22.3.0, but you have pyzmq 25.1.1. sounds like a critical issue, as we use pyzmq in the pympipool, but we can relax this dependency if necessary.

Yeah, my big concern is I can't figure out why we're getting the downgraded versions of these packages-- where else is the lower requirement coming from?

[liamhuber]

To me maggma 0.46.1 has requirement pyzmq==22.3.0, but you have pyzmq 25.1.1. sounds like a critical issue, as we use pyzmq in the pympipool, but we can relax this dependency if necessary.

Yeah, my big concern is I can't figure out why we're getting the downgraded versions of these packages-- where else is the lower requirement coming from?

I was being a bit daft when I first read your comment -- you're explicitly explaining where the pyzmq requirement is coming from! My bad. Nonetheless, I still don't see where we're getting the lower emmet-core dependency from.

[jan-janssen]

I was being a bit daft when I first read your comment -- you're explicitly explaining where the pyzmq requirement is coming from!

The pyzmq part is fixed now, as shown in pyiron/pyiron_gpl#145 but we still have the same issue with emmet-core, I am still not sure where this is coming from.

[liamhuber]

but we still have the same issue with emmet-core, I am still not sure where this is coming from.

I thought to check the github dependencies but these are only direct dependencies, and no transitive search is available on the web portal. Following the advice on the second link I tried using pycharm to search for emmet-core, but consistent with our current knowledge I don't find any references outside the emmet package itself and mp-api. So I'm still super stumped.

I'm going to try updating pyiron contrib in my conda env and repeating the pycharm search...

[liamhuber]

aimsgb explicitly request mp-api but specifies no version. Still digging...

[jan-janssen]

The strange part is by explicitly pinning emmet-core the issue can be fixed pyiron/pyiron_gpl#146 - in comparison to - pyiron/pyiron_gpl#145 .

[liamhuber]

The strange part is by explicitly pinning emmet-core the issue can be fixed pyiron/pyiron_gpl#146 - in comparison to - pyiron/pyiron_gpl#145 .

Crazy. I mean, it makes perfect sense that the pydantic_settings error goes away when you pin emmet-core, because the more recent version does include that dependency. But I agree it's totally wild why it will let you pin it and still create the env, while without the pin it regresses the version?!

[liamhuber]

@jan-janssen, the most recent feedstock for pyiron_atomistics actually has a pretty out-of-date requirement for mp-api. This surprises me since the 0.3.3 tag has mp-api =0.36.1 in its conda env.

Do you know why the feedstock would regress the dependency like that? I double-checked and 0.3.3 is indeed the live version on conda-forge right now, so I would have expected the github env yaml versions for that tag to be a floor for the feedstock recipe.

[jan-janssen]

Do you know why the feedstock would regress the dependency like that? I double-checked and 0.3.3 is indeed the live version on conda-forge right now, so I would have expected the github env yaml versions for that tag to be a floor for the feedstock recipe.

I created a pull request to fix this issue conda-forge/pyiron_atomistics-feedstock#116 . @niklassiemer did the last releases for pyiron_atomistics and while the dependencies for pyiron_base were correctly updated - for example in conda-forge/pyiron_base-feedstock#170 - this does not seem to be the case for pyiron_atomistics.

[liamhuber]

I created a pull request to fix this issue conda-forge/pyiron_atomistics-feedstock#116 .

🚀 Thanks!

@niklassiemer did the last releases for pyiron_atomistics and while the dependencies for pyiron_base were correctly updated - for example in conda-forge/pyiron_base-feedstock#170 - this does not seem to be the case for pyiron_atomistics.

Yeah, that's not inconsistent with my just-updated conda env, which wound up giving me mp-api 0.35.1 and emmet-core 0.67.5.

I'm a little bit concerned that the feedstock bump to mp-api 0.36.1 will be insufficient and that we'll need to go right up to 0.37.1. My reasoning is this: the conda-feedstock for mp-api 0.37.1 clearly demands emmet-core >=0.69.2. This is sufficient, and this dependency can be seen in the github repo's pyproject.toml.

However, if I move the github tag back to 0.36.1, the pyproject.toml requirement drops all the way down to 0.54.0, which is not enough to get us pydantic-settings.

Now, mp-api dependencies are absolutely a hot mess on their repo. Their emmet-core dependency is 0.54.0 in the pyproject.toml, 0.39.8 in requirements.txt, and 0.67.5 everywhere in requirements/*.txt (e.g.). These are also outdated garbage on 0.37.1, but there at least the toml file agrees with the feedstock.

The conda-forge feedstock doesn't contain easy-to-browse tags for past versions, but just looking at the history I can see that as recently as 0.37.0, the emmet-core dependency in the recipe is only >=0.54.0.

So my understainding is that if we really want to force conda to get emmet-core 0.69. or be unable to solve, we're going to need mp-api>=0.37.1*

[liamhuber]

_So my understainding is that if we really want to force conda to get emmet-core 0.69. or be unable to solve, we're going to need mp-api>=0.37.1_*

Of course, none of this explains why we're getting emmet-core 0.68.0, which is still happening over on the pyiron_atomistics mp-api-bump I thought would solve things. 🤬

[liamhuber]

_So my understainding is that if we really want to force conda to get emmet-core 0.69. or be unable to solve, we're going to need mp-api>=0.37.1_*

Of course, none of this explains why we're getting emmet-core 0.68.0, which is still happening over on the pyiron_atomistics mp-api-bump I thought would solve things. 🤬

It isn't anymore!!!! I asked dependabot to recreate the PR and now it's correctly getting mp-api 0.37.1 and emmet-core 0.69.5. Perfect!

pyiron_atomistics still needs to update whatever path it uses to depend on pympipool to pull in your latest patch, @jan-janssen, but after that I guess atomistics will be fixed and that should start the chain of dominos to fix everything else.

[niklassiemer]

Yes pyiron_atomistics is often taken care of by the conda-forge bot, which, however does not update dependencies... maybe we disable the automerge again? Then one would be forced to look at the PR and update things...

[liamhuber]

The following packages are incompatible
├─ numpy 1.24.3**  is installable with the potential options
│  ├─ numpy 1.24.3, which can be installed;
│  ├─ numpy [1.24.3|1.26.0] would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  ├─ numpy 1.24.3 would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  └─ numpy [1.24.3|1.26.0] would require
│     └─ python >=3.9,<3.10.0a0 , which can be installed;
└─ pyiron_atomistics 0.3.4**  is installable and it requires
   └─ atomistics >=0.0.6 , which requires
      └─ numpy >=1.26.0  with the potential options
         ├─ numpy 1.26.0 conflicts with any installable versions previously reported;
         ├─ numpy [1.24.3|1.26.0], which can be installed (as previously explained);
         ├─ numpy [1.24.3|1.26.0], which can be installed (as previously explained);
         └─ numpy 1.26.0 would require
            └─ python >=3.12.0rc3,<3.13.0a0  with the potential options
               ├─ python 3.12.0, which can be installed;
               └─ python 3.12.0rc3 would require
                  └─ _python_rc  , which does not exist (perhaps a missing channel).

[liamhuber]

Could not solve for environment specs
The following packages are incompatible
├─ matplotlib 3.7.2**  is installable with the potential options
│  ├─ matplotlib 3.7.2 would require
│  │  └─ matplotlib-base >=3.7.2,<3.7.3.0a0  with the potential options
│  │     ├─ matplotlib-base [3.7.2|3.8.0] would require
│  │     │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  │     ├─ matplotlib-base [3.7.2|3.8.0] would require
│  │     │  └─ python >=3.9,<3.10.0a0 , which can be installed;
│  │     ├─ matplotlib-base 3.7.2, which can be installed;
│  │     └─ matplotlib-base 3.7.2 would require
│  │        └─ python >=3.8,<3.9.0a0 , which can be installed;
│  ├─ matplotlib 3.7.2 would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  ├─ matplotlib 3.7.2 would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  ├─ matplotlib 3.7.2 would require
│  │  └─ python >=3.9,<3.10.0a0 , which can be installed;
│  └─ matplotlib 3.7.2 conflicts with any installable versions previously reported;
└─ pyiron_atomistics 0.3.4**  is installable and it requires
   └─ matplotlib-base >=3.8.0  with the potential options
      ├─ matplotlib-base [3.7.2|3.8.0], which can be installed (as previously explained);
      ├─ matplotlib-base 3.8.0 conflicts with any installable versions previously reported;
      └─ matplotlib-base [3.7.2|3.8.0], which can be installed (as previously explained).

[liamhuber]

Could not solve for environment specs
The following packages are incompatible
├─ pyiron_atomistics 0.3.4**  is installable and it requires
│  └─ atomistics >=0.0.6 , which requires
│     └─ scipy >=1.11.3  with the potential options
│        ├─ scipy [1.11.2|1.11.3] would require
│        │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│        ├─ scipy 1.11.3, which can be installed;
│        ├─ scipy 1.11.3 would require
│        │  └─ python >=3.12,<3.13.0a0 , which can be installed;
│        └─ scipy [1.11.2|1.11.3] would require
│           └─ python >=3.9,<3.10.0a0 , which can be installed;
└─ scipy 1.11.2**  is installable with the potential options
   ├─ scipy [1.11.2|1.11.3], which can be installed (as previously explained);
   ├─ scipy [1.11.2|1.11.3], which can be installed (as previously explained);
   └─ scipy 1.11.2 conflicts with any installable versions previously reported.

[liamhuber]

Could not solve for environment specs
The following packages are incompatible
├─ aws-sam-translator 1.75.0**  is installable and it requires
│  └─ pydantic ~=1.10.2  with the potential options
│     ├─ pydantic [1.10.10|1.10.11|...|2.0.3] would require
│     │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│     ├─ pydantic [1.10.10|1.10.11|...|2.0.3] would require
│     │  └─ python >=3.11,<3.12.0a0 , which can be installed;
│     ├─ pydantic [1.10.10|1.10.11|...|2.0.3] would require
│     │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│     ├─ pydantic [1.10.10|1.10.11|...|2.0.3] would require
│     │  └─ pypy3.9 >=7.3.11 , which can be installed;
│     ├─ pydantic 1.10.2 would require
│     │  └─ python >=3.7,<3.8.0a0 , which can be installed;
│     ├─ pydantic [1.10.2|1.10.3|1.10.4] would require
│     │  └─ python_abi 3.9 *_pypy39_pp73, which can be installed;
│     ├─ pydantic [1.10.10|1.10.11|...|1.10.9], which can be installed;
│     └─ pydantic 1.10.13 would require
│        └─ pypy3.9 >=7.3.12 , which can be installed;
├─ coveralls >=3.3.1  is installable and it requires
│  └─ python >=3.5  with the potential options
│     ├─ python 3.9.18 would require
│     │  └─ python_abi 3.9 *_pypy39_pp73, which can be installed;
│     ├─ python [3.10.0|3.10.1|...|3.10.9], which can be installed;
│     ├─ python [3.11.0|3.11.1|...|3.11.6], which can be installed;
│     ├─ python [3.12.0|3.5.1|...|3.9.9], which can be installed;
│     ├─ python [3.7.0|3.7.1|...|3.7.9], which can be installed;
│     ├─ python [3.8.0|3.8.1|...|3.8.8], which can be installed;
│     ├─ python 3.12.0rc3 would require
│     │  └─ _python_rc  , which does not exist (perhaps a missing channel);
│     └─ python 3.9.18 would require
│        └─ python_abi 3.9.* *_cp39, which conflicts with any installable versions previously reported;
├─ pyiron_atomistics 0.3.4**  is installable and it requires
│  ├─ mp-api >=0.37.2 , which requires
│  │  └─ emmet-core >=0.69.2 , which requires
│  │     └─ pydantic >=2.0  with the potential options
│  │        ├─ pydantic [1.10.10|1.10.11|...|2.0.3], which can be installed (as previously explained);
│  │        ├─ pydantic [1.10.10|1.10.11|...|2.0.3], which can be installed (as previously explained);
│  │        ├─ pydantic [1.10.10|1.10.11|...|2.0.3], which can be installed (as previously explained);
│  │        ├─ pydantic [2.0|2.0.1|...|2.4.2] conflicts with any installable versions previously reported;
│  │        └─ pydantic [1.10.10|1.10.11|...|2.0.3], which can be installed (as previously explained);
│  └─ phonopy >=2.20.0  with the potential options
│     ├─ phonopy 2.20.0 would require
│     │  └─ python_abi 3.9.* *_cp39 but there are no viable options
│     │     ├─ python_abi 3.9 conflicts with any installable versions previously reported;
│     │     └─ python_abi 3.9 would require
│     │        └─ python 3.9.* *_cpython, which conflicts with any installable versions previously reported;
│     ├─ phonopy 2.20.0 would require
│     │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│     ├─ phonopy 2.20.0 would require
│     │  └─ python >=3.11,<3.12.0a0 , which can be installed;
│     └─ phonopy 2.20.0 would require
│        └─ python >=3.8,<3.9.0a0 , which can be installed;
└─ scikit-image 0.21.0**  is installable with the potential options
   ├─ scikit-image 0.21.0 would require
   │  └─ python_abi 3.9 *_pypy39_pp73, which can be installed;
   ├─ scikit-image 0.21.0 would require
   │  └─ python >=3.10,<3.11.0a0 , which can be installed;
   ├─ scikit-image 0.21.0 would require
   │  └─ python >=3.11,<3.12.0a0 , which can be installed;
   ├─ scikit-image 0.21.0 would require
   │  └─ python >=3.8,<3.9.0a0 , which can be installed;
   └─ scikit-image 0.21.0 would require
      └─ python_abi 3.9.* *_cp39 but there are no viable options
         ├─ python_abi 3.9 conflicts with any installable versions previously reported;
         └─ python_abi 3.9, which cannot be installed (as previously explained).

[liamhuber]

From what I can tell, the heart of the issue is that aws-sam-translator 1.75.0 requires pydantic ~=1.10.2, while pyiron_atomistics 0.3.4 --> mp-api >=0.37.2 --> emmet-core >=0.69.2 requires pydantic >=2.0. Now, despite the major version difference, this is not impossible! For most python versions, pydantic =2.0.3 is included in the compatibility list for pydantic ~=1.10.2. Buuuut, not for python 3.11.

[jan-janssen]

From what I can tell, the heart of the issue is that aws-sam-translator 1.75.0 requires pydantic ~=1.10.2, while pyiron_atomistics 0.3.4 --> mp-api >=0.37.2 --> emmet-core >=0.69.2 requires pydantic >=2.0. Now, despite the major version difference, this is not impossible! For most python versions, pydantic =2.0.3 is included in the compatibility list for pydantic ~=1.10.2. Buuuut, not for python 3.11.

This should be fixed with conda-forge/aws-sam-translator-feedstock#88

[liamhuber]

This should be fixed with conda-forge/aws-sam-translator-feedstock#88

Awesome! I will anyhow proceed with the extraction of the workflow stuff to its own repo, since that was planned, but this means that in the medium term it will stay possible to have an env that supports both the most recent pyiron_contrib and the most recent pyiron_workflow at the same time.

[liamhuber]

(FYI: right now I intend to remove the workflow stuff from contrib in a separate PR, and after that's working I think we can just close this PR without merging.)