Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic pyproject.toml for installing dependencies #13

Merged
merged 8 commits into from
Mar 16, 2023
+108 −140
Merged

Add basic pyproject.toml for installing dependencies #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ada433f
remove IDE metadata
raehik Feb 16, 2023
1f58747
update gitignore
raehik Mar 16, 2023
c0c31d6
mention license in readme; add old copyright info
raehik Mar 8, 2023
1d980b6
add initial pyproject.toml
raehik Feb 16, 2023
857a92d
add Nix devshell file to help developing with Nix
raehik Mar 8, 2023
049d69f
cmip26: use regular tmpdir
raehik Mar 16, 2023
f9979f3
cmip26: fix logging config
raehik Mar 8, 2023
6f22e36
cmip26: log tmpdir
raehik Mar 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
mlruns/*
# general Python build files
__pycache__/*
*.pyc
/*.egg-info/
raehik marked this conversation as resolved.
Show resolved Hide resolved

# IDE project directories
/.idea/

# MLflow output
/mlruns/*
3 changes: 0 additions & 3 deletions .idea/.gitignore
View file
Open in desktop

This file was deleted.

93 changes: 0 additions & 93 deletions .idea/inspectionProfiles/Project_Default.xml
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions .idea/inspectionProfiles/profiles_settings.xml
View file
Open in desktop

This file was deleted.

4 changes: 0 additions & 4 deletions .idea/misc.xml
View file
Open in desktop

This file was deleted.

8 changes: 0 additions & 8 deletions .idea/modules.xml
View file
Open in desktop

This file was deleted.

12 changes: 0 additions & 12 deletions .idea/subgrid.iml
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions .idea/vcs.xml
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions LICENSE
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
MIT License

Copyright (c) 202?-2023 Arthur Guillaumin
Copyright (c) 2023 m2lines

Permission is hereby granted, free of charge, to any person obtaining a copy
Expand Down
3 changes: 3 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,3 +145,6 @@ Update analysis code so that it produces the nice figures in the paper


Add something about how to load the NN independently.

# License
Provided under the MIT license. See `LICENSE` for license text.
13 changes: 6 additions & 7 deletions cmip26.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,14 @@
import logging
import tempfile



# logging config
# obtain logging config from LOGGING_LEVEL environment variable
# e.g. `LOGGING_LEVEL=20 python cmip26.py ...`
# common numeric values: https://docs.python.org/3/library/logging.html#levels
logging_level = os.environ.get('LOGGING_LEVEL')
if logging_level is not None:
logging_level = getattr(logging, logging_level)
logging.basicConfig(level=logging_level)
logging.basicConfig(level=int(logging_level))
logger = logging.getLogger(__name__)


# Script parameters
CATALOG_URL = 'https://raw.githubusercontent.com/pangeo-data/pangeo-datastore\
/master/intake-catalogs/master.yaml'
Expand All @@ -41,7 +39,8 @@
apply coarse graining. Stores the resulting dataset into an MLFLOW \
experiment within a specific run.'

data_location = tempfile.mkdtemp(dir='/scratch/ag7531/temp/')
data_location = tempfile.mkdtemp()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that stuff gets stored here after data is downloaded.
We will almost certainly change this when tackling #8 but is it worth setting to a specified local location for now?
Given the data seems to be stored after this script is run I'd also question whether tempfile.mkdtemp() is the correct command to use given the docs seem to imply it is for temp dirs that should be closed and removed before completion.
This probably isn't the place to deal with this, however.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand, MLflow handles saving output:

mlflow.log_artifact(join(data_location, 'forcing'))

Which currently goes to ./mlruns/0/<hash>/artifacts/forcing. I think this is a tempdir and used correctly. Still useful to enable overriding (it was done so here), but I have a feeling tempfile can read envvars to do that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used correctly or incorrectly?
I didn't like the use of a tempfile to save important data when I saw it, and really tmpfiles should be cleared before ending the script.

I think you can probably just commit this as it should be dealt with in #4 and doesn't really matter where it creates it for now?

logger.info(f"working dir: {data_location}")

# Parse the command-line parameters
parser = argparse.ArgumentParser(description=DESCRIPTION)
Expand Down
65 changes: 65 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
[build-system]
requires = ["setuptools >= 61"]
build-backend = "setuptools.build_meta"

[project]
version = "0.2.0"

name = "gz21_ocean_momentum_cnn"
description = "TODO"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably write something tentative here, and/or as Arthur to provide.

readme = "README.md"
license = {file = "LICENSE"}
requires-python = ">=3.9"
classifiers = [
"Topic :: Scientific/Engineering :: Atmospheric Science",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
]

dependencies = [
# general, likely used all over
"scipy",
"xarray",
"dask",
"mlflow-skinny",

# data download
"intake",
"requests",
"aiohttp",
"intake-xarray",
"gcsfs",

# "torch>=1.13.1",
# "progressbar>=2.5"
]

authors = [
{ name="Arthur Guillaumin", email="[email protected]" },
]

maintainers = [
{ name="Ben Orchard", email="[email protected]" },
{ name="Jack Atkinson", email="[email protected]" },
{ name="Jim Denholm", email="[email protected]" },
]

[project.urls]
"Homepage" = "https://github.com/m2lines/GZ21_ocean_momentum_CNN"
"Bug Tracker" = "https://github.com/m2lines/GZ21_ocean_momentum_CNN/issues"

[project.optional-dependencies]
TEST = ["pytest"]

[tool.setuptools]
# By default, include-package-data is true in pyproject.toml, so you do
# NOT have to specify this line.
include-package-data = true

#[tool.setuptools.packages]
#find = {}
[tool.setuptools.packages.find]
where = ["."] # list of folders that contain the packages (["."] by default)
include = ["gz21_ocean_momentum_cnn", "gz21_ocean_momentum_cnn.*"] # package names should match these glob patterns (["*"] by default)
#exclude = ["gz21_ocean_momentum_cnn.tests*", "examples.py"] # exclude packages matching these glob patterns (empty by default)
namespaces = false # to disable scanning PEP 420 namespaces (true by default)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are imposing black we can add some of this into the file as follows:

[tool.black]
line-length = 88
target-version = ['py37']
include = '\.pyi?$'
exclude = '''
(
  /(
      \.eggs         # exclude a few common directories in the
    | \.git          # root of the project
    | \.hg
    | \.mypy_cache
    | \.tox
    | \.venv
    | _build
    | buck-out
    | build
    | dist
  )/
  | foo.py           # also separately exclude a file named foo.py in
                     # the root of the project
)
'''

This is optional though and I'm not sure I fully appreciate what it does at the moment.
Similar tools exist for linting etc.

25 changes: 25 additions & 0 deletions shell.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# This file is to help Nix users develop the package.
# With Nix installed, run the following commands in this directory:
#
# $ nix-shell
# $ virtualenv venv
# $ source venv/bin/activate
#
# Now you can use `pip install -e .` and generally do regular Python development
# things.
#
# Note that we can't make this a flake easily because of FHS.

{ pkgs ? import <nixpkgs> {} }:

(pkgs.buildFHSUserEnv {
name = "gz21-ocean-momentum-cnn-devshell";
runScript = "zsh";
targetPkgs = pkgs: (with pkgs; [
pkgs.python3
pkgs.python3Packages.pip
pkgs.python3Packages.virtualenv

pkgs.zlib # numpy
]);
}).env