Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use hatchling as build backend #12269

Merged
merged 6 commits into from
Dec 8, 2023
Merged

Conversation

hoechenberger
Copy link
Member

@hoechenberger hoechenberger commented Dec 6, 2023

Supersedes #12173

Closes #12173
Closes #12169

cc @ofek Let's see how this goes

Editable installs now work as expected.


I built the packages via

python -m build .

both on this branch and on main.

I diffed the contents of the archives.

Wheel diff

❯ diff setuptools/site-packages hatchling/site-packages
Common subdirectories: setuptools/site-packages/mne and hatchling/site-packages/mne
Only in setuptools/site-packages: mne-1.7.0.dev26+g7bf1b4ab7.dist-info
Only in hatchling/site-packages: mne-1.7.0.dev27+ge9f4a7be3.dist-info

sdist diff

❯ diff setuptools/mne-1.7.0.dev26+g7bf1b4ab7 hatchling/mne-1.7.0.dev27+ge9f4a7be3
Only in setuptools/mne-1.7.0.dev26+g7bf1b4ab7: .circleci
Only in setuptools/mne-1.7.0.dev26+g7bf1b4ab7: .git_archival.txt
Only in setuptools/mne-1.7.0.dev26+g7bf1b4ab7: .gitattributes
Only in setuptools/mne-1.7.0.dev26+g7bf1b4ab7: .github
Only in setuptools/mne-1.7.0.dev26+g7bf1b4ab7: MANIFEST.in
diff --color setuptools/mne-1.7.0.dev26+g7bf1b4ab7/PKG-INFO hatchling/mne-1.7.0.dev27+ge9f4a7be3/PKG-INFO
3c3
< Version: 1.7.0.dev26+g7bf1b4ab7
---
> Version: 1.7.0.dev27+ge9f4a7be3
5,7d4
< Author-email: Alexandre Gramfort <[email protected]>
< Maintainer-email: Dan McCloy <[email protected]>
< License: BSD-3-Clause
14,15c11,15
< Keywords: neuroscience,neuroimaging,MEG,EEG,ECoG,fNIRS,brain
< Classifier: Intended Audience :: Science/Research
---
> Author-email: Alexandre Gramfort <[email protected]>
> Maintainer-email: Dan McCloy <[email protected]>
> License: BSD-3-Clause
> License-File: LICENSE.txt
> Keywords: ECoG,EEG,MEG,brain,fNIRS,neuroimaging,neuroscience
16a17
> Classifier: Intended Audience :: Science/Research
18,20c19
< Classifier: Programming Language :: Python
< Classifier: Topic :: Software Development
< Classifier: Topic :: Scientific/Engineering
---
> Classifier: Operating System :: MacOS
24c23
< Classifier: Operating System :: MacOS
---
> Classifier: Programming Language :: Python
25a25,26
> Classifier: Topic :: Scientific/Engineering
> Classifier: Topic :: Software Development
27,28c28,31
< Description-Content-Type: text/x-rst
< License-File: LICENSE.txt
---
> Requires-Dist: decorator
> Requires-Dist: jinja2
> Requires-Dist: lazy-loader>=0.3
> Requires-Dist: matplotlib>=3.5.0
29a33,34
> Requires-Dist: packaging
> Requires-Dist: pooch>=1.5
31d35
< Requires-Dist: matplotlib>=3.5.0
33,37d36
< Requires-Dist: pooch>=1.5
< Requires-Dist: decorator
< Requires-Dist: packaging
< Requires-Dist: jinja2
< Requires-Dist: lazy_loader>=0.3
39,41c38,62
< Provides-Extra: hdf5
< Requires-Dist: h5io; extra == "hdf5"
< Requires-Dist: pymatreader; extra == "hdf5"
---
> Provides-Extra: dev
> Requires-Dist: mne[doc,test]; extra == 'dev'
> Requires-Dist: rcssmin; extra == 'dev'
> Provides-Extra: doc
> Requires-Dist: graphviz; extra == 'doc'
> Requires-Dist: ipython!=8.7.0; extra == 'doc'
> Requires-Dist: memory-profiler; extra == 'doc'
> Requires-Dist: mne-bids; extra == 'doc'
> Requires-Dist: mne-connectivity; extra == 'doc'
> Requires-Dist: mne-gui-addons; extra == 'doc'
> Requires-Dist: neo; extra == 'doc'
> Requires-Dist: numpydoc; extra == 'doc'
> Requires-Dist: pydata-sphinx-theme==0.13.3; extra == 'doc'
> Requires-Dist: pygments>=2.13; extra == 'doc'
> Requires-Dist: pytest; extra == 'doc'
> Requires-Dist: pyxdf; extra == 'doc'
> Requires-Dist: pyzmq!=24.0.0; extra == 'doc'
> Requires-Dist: seaborn!=0.11.2; extra == 'doc'
> Requires-Dist: selenium; extra == 'doc'
> Requires-Dist: sphinx-copybutton; extra == 'doc'
> Requires-Dist: sphinx-design; extra == 'doc'
> Requires-Dist: sphinx-gallery; extra == 'doc'
> Requires-Dist: sphinx>=6; extra == 'doc'
> Requires-Dist: sphinxcontrib-bibtex>=2.5; extra == 'doc'
> Requires-Dist: sphinxcontrib-youtube; extra == 'doc'
43,85c64,109
< Requires-Dist: mne[hdf5]; extra == "full"
< Requires-Dist: qtpy; extra == "full"
< Requires-Dist: PyQt6!=6.6.1; extra == "full"
< Requires-Dist: PyQt6-Qt6!=6.6.1; extra == "full"
< Requires-Dist: pyobjc-framework-Cocoa>=5.2.0; platform_system == "Darwin" and extra == "full"
< Requires-Dist: sip; extra == "full"
< Requires-Dist: scikit-learn; extra == "full"
< Requires-Dist: nibabel; extra == "full"
< Requires-Dist: openmeeg>=2.5.5; extra == "full"
< Requires-Dist: numba; extra == "full"
< Requires-Dist: h5py; extra == "full"
< Requires-Dist: pandas; extra == "full"
< Requires-Dist: numexpr; extra == "full"
< Requires-Dist: jupyter; extra == "full"
< Requires-Dist: python-picard; extra == "full"
< Requires-Dist: statsmodels; extra == "full"
< Requires-Dist: joblib; extra == "full"
< Requires-Dist: psutil; extra == "full"
< Requires-Dist: dipy; extra == "full"
< Requires-Dist: vtk; extra == "full"
< Requires-Dist: nilearn; extra == "full"
< Requires-Dist: xlrd; extra == "full"
< Requires-Dist: imageio>=2.6.1; extra == "full"
< Requires-Dist: imageio-ffmpeg>=0.4.1; extra == "full"
< Requires-Dist: traitlets; extra == "full"
< Requires-Dist: pyvista!=0.35.2,!=0.38.0,!=0.38.1,!=0.38.2,!=0.38.3,!=0.38.4,!=0.38.5,!=0.38.6,!=0.42.0,>=0.32; extra == "full"
< Requires-Dist: pyvistaqt>=0.4; extra == "full"
< Requires-Dist: mffpy>=0.5.7; extra == "full"
< Requires-Dist: ipywidgets; extra == "full"
< Requires-Dist: ipympl; extra == "full"
< Requires-Dist: ipyevents; extra == "full"
< Requires-Dist: trame; extra == "full"
< Requires-Dist: trame-vtk; extra == "full"
< Requires-Dist: trame-vuetify; extra == "full"
< Requires-Dist: mne-qt-browser; extra == "full"
< Requires-Dist: darkdetect; extra == "full"
< Requires-Dist: qdarkstyle!=3.2.2; extra == "full"
< Requires-Dist: threadpoolctl; extra == "full"
< Requires-Dist: eeglabio; extra == "full"
< Requires-Dist: edfio>=0.2.1; extra == "full"
< Requires-Dist: pybv; extra == "full"
< Requires-Dist: snirf; extra == "full"
< Requires-Dist: defusedxml; extra == "full"
---
> Requires-Dist: darkdetect; extra == 'full'
> Requires-Dist: defusedxml; extra == 'full'
> Requires-Dist: dipy; extra == 'full'
> Requires-Dist: edfio>=0.2.1; extra == 'full'
> Requires-Dist: eeglabio; extra == 'full'
> Requires-Dist: h5py; extra == 'full'
> Requires-Dist: imageio-ffmpeg>=0.4.1; extra == 'full'
> Requires-Dist: imageio>=2.6.1; extra == 'full'
> Requires-Dist: ipyevents; extra == 'full'
> Requires-Dist: ipympl; extra == 'full'
> Requires-Dist: ipywidgets; extra == 'full'
> Requires-Dist: joblib; extra == 'full'
> Requires-Dist: jupyter; extra == 'full'
> Requires-Dist: mffpy>=0.5.7; extra == 'full'
> Requires-Dist: mne-qt-browser; extra == 'full'
> Requires-Dist: mne[hdf5]; extra == 'full'
> Requires-Dist: nibabel; extra == 'full'
> Requires-Dist: nilearn; extra == 'full'
> Requires-Dist: numba; extra == 'full'
> Requires-Dist: numexpr; extra == 'full'
> Requires-Dist: openmeeg>=2.5.5; extra == 'full'
> Requires-Dist: pandas; extra == 'full'
> Requires-Dist: psutil; extra == 'full'
> Requires-Dist: pybv; extra == 'full'
> Requires-Dist: pyobjc-framework-cocoa>=5.2.0; platform_system == 'Darwin' and extra == 'full'
> Requires-Dist: pyqt6!=6.6.1; extra == 'full'
> Requires-Dist: pyqt6-qt6!=6.6.1; extra == 'full'
> Requires-Dist: python-picard; extra == 'full'
> Requires-Dist: pyvista!=0.35.2,!=0.38.0,!=0.38.1,!=0.38.2,!=0.38.3,!=0.38.4,!=0.38.5,!=0.38.6,!=0.42.0,>=0.32; extra == 'full'
> Requires-Dist: pyvistaqt>=0.4; extra == 'full'
> Requires-Dist: qdarkstyle!=3.2.2; extra == 'full'
> Requires-Dist: qtpy; extra == 'full'
> Requires-Dist: scikit-learn; extra == 'full'
> Requires-Dist: sip; extra == 'full'
> Requires-Dist: snirf; extra == 'full'
> Requires-Dist: statsmodels; extra == 'full'
> Requires-Dist: threadpoolctl; extra == 'full'
> Requires-Dist: traitlets; extra == 'full'
> Requires-Dist: trame; extra == 'full'
> Requires-Dist: trame-vtk; extra == 'full'
> Requires-Dist: trame-vuetify; extra == 'full'
> Requires-Dist: vtk; extra == 'full'
> Requires-Dist: xlrd; extra == 'full'
> Provides-Extra: hdf5
> Requires-Dist: h5io; extra == 'hdf5'
> Requires-Dist: pymatreader; extra == 'hdf5'
87,100c111,124
< Requires-Dist: pytest; extra == "test"
< Requires-Dist: pytest-cov; extra == "test"
< Requires-Dist: pytest-timeout; extra == "test"
< Requires-Dist: pytest-harvest; extra == "test"
< Requires-Dist: pytest-qt; extra == "test"
< Requires-Dist: ruff; extra == "test"
< Requires-Dist: numpydoc; extra == "test"
< Requires-Dist: codespell; extra == "test"
< Requires-Dist: check-manifest; extra == "test"
< Requires-Dist: tomli; python_version < "3.11" and extra == "test"
< Requires-Dist: twine; extra == "test"
< Requires-Dist: wheel; extra == "test"
< Requires-Dist: pre-commit; extra == "test"
< Requires-Dist: mypy; extra == "test"
---
> Requires-Dist: check-manifest; extra == 'test'
> Requires-Dist: codespell; extra == 'test'
> Requires-Dist: mypy; extra == 'test'
> Requires-Dist: numpydoc; extra == 'test'
> Requires-Dist: pre-commit; extra == 'test'
> Requires-Dist: pytest; extra == 'test'
> Requires-Dist: pytest-cov; extra == 'test'
> Requires-Dist: pytest-harvest; extra == 'test'
> Requires-Dist: pytest-qt; extra == 'test'
> Requires-Dist: pytest-timeout; extra == 'test'
> Requires-Dist: ruff; extra == 'test'
> Requires-Dist: tomli; python_version < '3.11' and extra == 'test'
> Requires-Dist: twine; extra == 'test'
> Requires-Dist: wheel; extra == 'test'
102,136c126,136
< Requires-Dist: mne[test]; extra == "test-extra"
< Requires-Dist: nitime; extra == "test-extra"
< Requires-Dist: nbclient; extra == "test-extra"
< Requires-Dist: sphinx-gallery; extra == "test-extra"
< Requires-Dist: eeglabio; extra == "test-extra"
< Requires-Dist: edfio>=0.2.1; extra == "test-extra"
< Requires-Dist: pybv; extra == "test-extra"
< Requires-Dist: imageio>=2.6.1; extra == "test-extra"
< Requires-Dist: imageio-ffmpeg>=0.4.1; extra == "test-extra"
< Requires-Dist: snirf; extra == "test-extra"
< Provides-Extra: doc
< Requires-Dist: sphinx>=6; extra == "doc"
< Requires-Dist: numpydoc; extra == "doc"
< Requires-Dist: pydata_sphinx_theme==0.13.3; extra == "doc"
< Requires-Dist: sphinx-gallery; extra == "doc"
< Requires-Dist: sphinxcontrib-bibtex>=2.5; extra == "doc"
< Requires-Dist: memory_profiler; extra == "doc"
< Requires-Dist: neo; extra == "doc"
< Requires-Dist: seaborn!=0.11.2; extra == "doc"
< Requires-Dist: sphinx_copybutton; extra == "doc"
< Requires-Dist: sphinx-design; extra == "doc"
< Requires-Dist: sphinxcontrib-youtube; extra == "doc"
< Requires-Dist: mne-bids; extra == "doc"
< Requires-Dist: pyxdf; extra == "doc"
< Requires-Dist: mne-connectivity; extra == "doc"
< Requires-Dist: mne-gui-addons; extra == "doc"
< Requires-Dist: pygments>=2.13; extra == "doc"
< Requires-Dist: pytest; extra == "doc"
< Requires-Dist: graphviz; extra == "doc"
< Requires-Dist: pyzmq!=24.0.0; extra == "doc"
< Requires-Dist: ipython!=8.7.0; extra == "doc"
< Requires-Dist: selenium; extra == "doc"
< Provides-Extra: dev
< Requires-Dist: mne[doc,test]; extra == "dev"
< Requires-Dist: rcssmin; extra == "dev"
---
> Requires-Dist: edfio>=0.2.1; extra == 'test-extra'
> Requires-Dist: eeglabio; extra == 'test-extra'
> Requires-Dist: imageio-ffmpeg>=0.4.1; extra == 'test-extra'
> Requires-Dist: imageio>=2.6.1; extra == 'test-extra'
> Requires-Dist: mne[test]; extra == 'test-extra'
> Requires-Dist: nbclient; extra == 'test-extra'
> Requires-Dist: nitime; extra == 'test-extra'
> Requires-Dist: pybv; extra == 'test-extra'
> Requires-Dist: snirf; extra == 'test-extra'
> Requires-Dist: sphinx-gallery; extra == 'test-extra'
> Description-Content-Type: text/x-rst
Only in setuptools/mne-1.7.0.dev26+g7bf1b4ab7: examples
Common subdirectories: setuptools/mne-1.7.0.dev26+g7bf1b4ab7/mne and hatchling/mne-1.7.0.dev27+ge9f4a7be3/mne
Only in setuptools/mne-1.7.0.dev26+g7bf1b4ab7: mne.egg-info
diff --color setuptools/mne-1.7.0.dev26+g7bf1b4ab7/pyproject.toml hatchling/mne-1.7.0.dev27+ge9f4a7be3/pyproject.toml
0a1,4
> [build-system]
> requires = ["hatchling", "hatch-vcs"]
> build-backend = "hatchling.build"
> 
171,214c175,192
< [build-system]
< requires = ["setuptools>=45", "setuptools_scm[toml]>=6.2", "wheel"]
< build-backend = "setuptools.build_meta"
< 
< [tool.setuptools.packages.find]
< where = ["."]
< include = ["mne*"]
< namespaces = false
< 
< [tool.setuptools_scm]
< version_scheme = "release-branch-semver"
< 
< [tool.setuptools]
< include-package-data = true
< 
< [tool.setuptools.package-data]
< "mne" = [
<     "data/eegbci_checksums.txt",
<     "data/*.sel",
<     "data/icos.fif.gz",
<     "data/coil_def*.dat",
<     "data/helmets/*.fif.gz",
<     "data/FreeSurferColorLUT.txt",
<     "data/image/*gif",
<     "data/image/*lout",
<     "data/fsaverage/*.fif",
<     "channels/data/layouts/*.lout",
<     "channels/data/layouts/*.lay",
<     "channels/data/montages/*.sfp",
<     "channels/data/montages/*.txt",
<     "channels/data/montages/*.elc",
<     "channels/data/neighbors/*.mat",
<     "datasets/sleep_physionet/SHA1SUMS",
<     "datasets/_fsaverage/*.txt",
<     "datasets/_infant/*.txt",
<     "datasets/_phantom/*.txt",
<     "html/*.js",
<     "html/*.css",
<     "html_templates/repr/*.jinja",
<     "html_templates/report/*.jinja",
<     "icons/*.svg",
<     "icons/*.png",
<     "io/artemis123/resources/*.csv",
<     "io/edf/gdf_encodes.txt",
---
> [tool.hatch.build]
> exclude = [
>     "/.*",
>     "/*.yml",
>     "/*.yaml",
>     "/*.toml",
>     "/*.txt",
>     "/mne/**/tests",
>     "/logo",
>     "/doc",
>     "/tools",
>     "/tutorials",
>     "/examples",
>     "/CITATION.cff",
>     "/codemeta.json",
>     "/ignore_words.txt",
>     "/Makefile",
>     "/CONTRIBUTING.md",
215a194,197
> 
> [tool.hatch.version]
> source = "vcs"
> raw-options = { version_scheme = "release-branch-semver" }
Only in setuptools/mne-1.7.0.dev26+g7bf1b4ab7: setup.cfg
Only in setuptools/mne-1.7.0.dev26+g7bf1b4ab7: tutorials

I think this all looks great and removes a few files from the sdist that we didn't ever deploy via the wheel anyway (in other words: this stuff never ended up in user installations).

cc @cbrnr @mscheltienne @sappelhoff @agramfort

@hoechenberger hoechenberger marked this pull request as ready for review December 6, 2023 13:10
@cbrnr
Copy link
Contributor

cbrnr commented Dec 6, 2023

Great work @hoechenberger! I tested an editable install in a workspace outside of the MNE source, and I get all completions in VS Code. On main, Pylance cannot even resolve import mne. So definitely a huge W!

To me, this looks good and could be merged. If we discover that something is missing, or that we want to exclude even more, we can always change it later.

@hoechenberger
Copy link
Member Author

tools/generate_codemeta.py needs to be adjusted, but I don't understand the code well enough to do it myself.

Copy link
Member

@larsoner larsoner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried the sdist, wheel, and an editable install on this branch and all seemed to work fine 👍

tools/generate_codemeta.py needs to be adjusted, but I don't understand the code well enough to do it myself.

What needs to be adjusted there? It just ran without errors for me on your branch and generated a diff that looked okay with a quick glance.

@larsoner larsoner added this to the 1.7 milestone Dec 6, 2023
@hoechenberger
Copy link
Member Author

hoechenberger commented Dec 6, 2023

I tried the sdist, wheel, and an editable install on this branch and all seemed to work fine 👍

tools/generate_codemeta.py needs to be adjusted, but I don't understand the code well enough to do it myself.

What needs to be adjusted there? It just ran without errors for me on your branch and generated a diff that looked okay with a quick glance.

A sorry, I suppose I was mistaken, we already took care of those changes when we moved all deps into pyproject.toml recently. I got mixed up.

Copy link

@ofek ofek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! I'm very glad static analysis is working for you again in VSCode 🙂

@hoechenberger
Copy link
Member Author

@ofek Since you're around – do you know if there are any plans to integrate hatch-vcs into hatchling? Or to work toward a closer integration?

@ofek
Copy link

ofek commented Dec 6, 2023

Not at the moment as I don't have time for that. Also I know the maintainer of setuptools_scm which that package is backed by and it seems like a lot of work so I would much rather just outsource to Ronny.

Copy link
Member

@drammock drammock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one comment/question. LGTM

"io/edf/gdf_encodes.txt",
[tool.hatch.build]
exclude = [
"/.*",
Copy link
Member

@drammock drammock Dec 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like it excludes all files at the root level of the repo; if that's right I don't see why we're explicitly excluding things like /Makefile etc below. Am I misinterpreting /.* here?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is the same syntax as gitignore and matches any entry at the root that starts with a period

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I would recommend using inclusion overall rather than exclusion to account for new stuff in future

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is the same syntax as gitignore and matches any entry at the root that starts with a period

ah right, like glob not regex.

Copy link
Member Author

@hoechenberger hoechenberger Dec 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~everything that's tracked in the VCS (git) is included by default, hence I picked the exclusion-based approach to exclude what's tracked in git, but shouldn't be packaged

Do you think that's not a good idea, @ofek?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's perfectly valid but my personal preference for source distributions is to only ship the project and test directories. Totally up to you!

@hoechenberger
Copy link
Member Author

I think we're good to merge here!

Thanks everybody for helping test this and for your feedback. Special thanks to @ofek!

@hoechenberger hoechenberger enabled auto-merge (squash) December 7, 2023 08:54
@hoechenberger

This comment was marked as resolved.

@hoechenberger hoechenberger merged commit d00cbb1 into mne-tools:main Dec 8, 2023
27 checks passed
hoechenberger added a commit to hoechenberger/mne-python that referenced this pull request Dec 8, 2023
Fixes packaging bug introduced in mne-tools#12269

Addresses the issues raised in:
mne-tools/mne-bids-pipeline#825 (comment)

mne-tools/mne-bids#1206 (comment)

Ultimately, we should start testing the installed package(s), not simply
fire up the tests from the root of the checked-out repo.
hoechenberger added a commit to hoechenberger/mne-python that referenced this pull request Dec 8, 2023
Fixes packaging bug introduced in mne-tools#12269

Addresses the issues raised in:
mne-tools/mne-bids-pipeline#825 (comment)

mne-tools/mne-bids#1206 (comment)

Ultimately, we should start testing the installed package(s), not simply
fire up the tests from the root of the checked-out repo.
snwnde pushed a commit to snwnde/mne-python that referenced this pull request Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Editable installs with pip and VS Code
6 participants