Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create Compilation records where Missions are used for Figures and Projects #80

Open
MBARIMike opened this issue Sep 30, 2021 · 20 comments
Open

Create Compilation records where Missions are used for Figures and Projects #80

MBARIMike opened this issue Sep 30, 2021 · 20 comments
Labels
original use case Translated from Use Case document

Comments

@MBARIMike
Copy link
Contributor

From the Use Case document:

If the mission is part of a regional compilation , find where on SeafloorMapping the most current working directory is (e.g., where navadjust work is being done…may be years later).
@MBARIMike MBARIMike added the original use case Translated from Use Case document label Sep 30, 2021
@jbpaduan
Copy link
Collaborator

jbpaduan commented Dec 2, 2021

This can't be found in the Notes files unfortunately. The working directory will initially be in the same year's directory as the mission files, but as new missions are conducted for a project, the old missions will be used again and again, with the working directories migrating upwards, to new year directories.
e.g., MAO2016/20160722m1 will be worked up in MAO2016/NorthernExpedition/Figures, but gets used in 2018, 2019, 2020, and 2021 (currently in 2021/AxialSeamount/Figures/

@jbpaduan
Copy link
Collaborator

jbpaduan commented Dec 2, 2021

And in thinking about it, a mission might be utilized in many working directories even in the same year, as time goes by. Dave is interested in being able to navigate to the projects where a mission is used. Perhaps the way to find the working directories is to search for a mission's name in all of the datalist*.mb-1 files in the entire SeafloorMapping share and display each unique path where the mission has been used, sorted so most recent use is at the top of the list. Then the user can click on the path that suits their purpose.

@MBARIMike
Copy link
Contributor Author

Just what I was thinking:

search for a mission's name in all of the datalist*.mb-1 files in the entire SeafloorMapping share

Some kind of tree structure will result where we can map nodes onto Compilation records.

@MBARIMike MBARIMike mentioned this issue Dec 2, 2021
Open
@jbpaduan
Copy link
Collaborator

With this change of schema, and the clearest way to search for "projects" being to look for datalist*.mb-1 files that use a mission (and there can be more than one in a given directory, BTW), I will need to change what is in the exclude.list file. I had been focused on listing directories that were being picked up due to presence of ZTopo.grd files, but were not mission directories. However, if a "project" directory is in that list because it had a ZTopo.grd in it (there are many), will it now will be overlooked by any script? If so, they'll need to be deleted from the exclude.list and the ZTopo.grd files in them renamed.

@MBARIMike MBARIMike changed the title Find where on SeafloorMapping the most current working directory is Create Compilation records where Missions have been used for Figures and Projects Jan 31, 2022
@MBARIMike MBARIMike changed the title Create Compilation records where Missions have been used for Figures and Projects Create Compilation records where Missions are used for Figures and Projects Jan 31, 2022
@MBARIMike
Copy link
Contributor Author

MBARIMike commented Jan 31, 2022 •
edited
Loading

About to start making schema changes, so I want to capture the current schema diagram. This is generated on my development system with the command:

docker-compose exec -e DATABASE_URL=postgis://<user>:<pw>@localhost:5432/default django python smdb/manage.py graph_models smdb -o smdb/docs/smdb_schema.png

where <user> and <pw> are the user and password found in smdb/.envs/.production/.postgres.
smdb_schema

This image will now be kept up to date in the repository at:

https://github.com/mbari-org/SeafloorMappingDB/blob/main/smdb/docs/smdb_schema.png

@jbpaduan
Copy link
Collaborator

jbpaduan commented Feb 1, 2022 via email

@MBARIMike MBARIMike mentioned this issue Feb 4, 2022
Merged
@MBARIMike
Copy link
Contributor Author

I had futzed around with searching all the datalist*.mb-1 files for Missions and then looking up the mbgrid commands in the .cmd files in the Compilation directories, but it was quickly becoming completely overwhelming with many, many many-to-many relationships. The current PR now looks in just the Figures.cmd file found int a Compilation directory for the mbgrid commands, then navigates the input datalist and the output .grd file for metadata to link Missions to Compilations. In early tests on development systems this scheme is finding 500+ Compilations. The PR contains a new compilations view to show the Missions that make up the Compilation.

@MBARIMike MBARIMike mentioned this issue Feb 24, 2022
Merged
@jbpaduan
Copy link
Collaborator

Please modify this search to Figure*.cmd, as we I also have command files named like Figures_Jenny.cmd and Figure_Caress.cmd to separate our respective work. Note there are many files named like Figs_temp.cmd that don't need to be included in the search (they are subsets of the code in the Figures_Jenny.cmd-like files, so the whole command file isn't executed.)

@jbpaduan
Copy link
Collaborator

On the Compilation View page, the "Missions making up the Compilation" list is not being populated. As noted on the page, the link to the folder where the compilation grid is does not open the file system on Titan.

@jbpaduan
Copy link
Collaborator

jbpaduan commented Mar 11, 2024 •
edited
Loading

(Also regards issue #78)
Re link to the folder on the Compilations web pages,

  1. "Open this folder on Titan" hyperlink produces an error. It should not include the root of the grid name, just the path, as it is trying to mount the grid name rather than the folder (for example, the folder here is Figures and the grid name is Monterey2020_Topo1m.grd and the link is to cifs://titan.shore.mbari.org/SeafloorMapping/2020/Monterey/Figures/Monterey2020_Topo1m .)
  2. Maybe the answer to above is here: There is a comment that needs to be addressed and/or commented out: e.g., on the page,
    https://smdb.shore.mbari.org/compilations/2020-monterey-figures-monterey2020_topo1m/
    “{# I think this would work if SeafloorMapping/2020/Monterey/Figures/Monterey2020_Topo1m were set up as a share.”

@jbpaduan
Copy link
Collaborator

jbpaduan commented Mar 11, 2024 •
edited
Loading

Re Compilations not finding all Missions

  1. An example is on this page, https://smdb.shore.mbari.org/missions/2023-20230402m1/
    where “Used in Compilations“ shows nothing.
  2. In the 2023 Mid-Atlantic Ridge mission pages, e.g., https://smdb.shore.mbari.org/missions/2023-20230310m2/
    https://smdb.shore.mbari.org/missions/2023-20230318m1/
    the only compilations shown are in the cruise MissionPlanning directories, and the compilations in the cruise Figures directory is not being found/displayed, /Volumes/SeafloorMapping/2023/MAR_FKt230303/Figures/

Another, https://smdb.shore.mbari.org/compilations/2019-agu-20191126aukamappingaguposter-cruisereportfigures-auka_mb_topo5cm/
has the grid name and jpg thumbnail but finds no missions making up the compilation.

@jbpaduan
Copy link
Collaborator

jbpaduan commented Mar 11, 2024 •
edited
Loading

(MM noted this in Issue #76) Many thumbnails for compilations are not being found. On Compilations pages, e.g., from 2020/Monterey/Figures/...
https://smdb.shore.mbari.org/compilations/2020-monterey-figures-monterey2020_topo1m/ ,
It says there is no thumbnail "ZTopoSlopeNav.jpg". A JPG in that folder on Titan is Monterey2020_Topo1m_SlopeNav.jpg, which is what ought to be used (/Volumes/SeafloorMapping/2020/Monterey/Figures.)
For a test, I renamed Monterey2020_Topo1m_SlopeNav.jpg to Monterey2020_Topo1m_slopeNav.jpg on 1/1/24, but it hasn’t changed the behavior). What is the lookup script looking for?
 
Another missing thumbnail example:
https://smdb.shore.mbari.org/compilations/2022-montereycanyon-figures-tubewormslump_topo_2mutm/
indeed has no corresponding GIS products or jpgs (Dave must have made this and the others in this directory for planning purposes?)
 
On the other hand, this one works:
https://smdb.shore.mbari.org/compilations/2020-surridge-figures-surridge_mauv_topo1m/
(finds Figures, and a jpg thumbnail)
 
This one works too:
https://smdb.shore.mbari.org/compilations/2023-pacnw-cascadia-axial-figures-roguefan_mauv_topo1m/
but we never made a JPG for this UTM version compilation:
https://smdb.shore.mbari.org/compilations/2023-pacnw-cascadia-axial-figures-roguefan_mauv_topo1m_utm/
 
Usually plots made by our processing script in the mission directories are SlopeNav_Plot.jpg, or slopeNav.jpg. The JPGs in the various /Figures directories, or in our post-cruise (AGU, publications) directories don't have that nomenclature, and they haven’t always been part of our workflow. Can we use the tifs made from compilations if there is no JPG?

@MBARIMike
Copy link
Contributor Author

MBARIMike commented Mar 18, 2024 •
edited
Loading

The relevant code I'm stepping through in debug mode is here...

Actually, the relevant line seems to be the pattern that's used to locate Figure*.cmd files. It finds only the Figure.cmd or Figures.cmd file in a compilation directory. Should it be relaxed to find more files? For example, the directory mentioned in 2 of #80 (comment) does not have a Figures.cmd file, but these:

➜  ~ cd /Volumes/SeafloorMapping/2023/MAR_FKt230303/Figures
➜  Figures ls -l Figure*.cmd
-rwx------+ 1 seafloor  SHORE\Domain Users  18490 Dec  9 15:06 Figures_EMARK.cmd
-rwx------+ 1 seafloor  SHORE\Domain Users   7991 Nov 16 11:48 Figures_GrappeDeux.cmd
-rwx------+ 1 caress    SHORE\Domain Users   2573 Apr  5  2023 Figures_OLD_not_EMARK_or_Grappe2.cmd
-rwx------+ 1 seafloor  SHORE\Domain Users  23257 Dec  7 12:28 Figures_PuyDesFolles.cmd
-rwx------+ 1 caress    SHORE\Domain Users   3747 Apr  5  2023 Figures_make_datalists_EM124.cmd
-rwx------@ 1 paje      SHORE\Domain Users   5837 Jun 28  2023 Figures_not_EMARK_or_Grappe2.cmd
-rwx------+ 1 seafloor  SHORE\Domain Users   3429 Dec  8 16:14 Figures_prior_data_and_transits.cmd

@jbpaduan
Copy link
Collaborator

jbpaduan commented Mar 18, 2024 via email

@MBARIMike
Copy link
Contributor Author

I finally got the Mac Arm development environment working with this large PR. This also includes a more relaxed search for Figures*.cmd files. The database was reloaded last night and a lot more Compilations are shown in the Mission detail views. I see several niggling problems now that I'm going to start addressing...

@MBARIMike
Copy link
Contributor Author

One such detail is that many requests like this one are returning 500 errors:

https://smdb.shore.mbari.org/compilations/2023-mar_fkt230303-figures-puydesfolles_mauv_topo1m/

This is because two identically named Compilations were loaded from two different .cmd files as can be seen in this query:

docker_user@smdb:/opt/SeafloorMappingDB$ docker-compose run --rm django python manage.py shell_plus
...
In [1]: cs = Compilation.objects.filter(slug='2023-mar_fkt230303-figures-puydesfolles_mauv_topo1m')

In [2]: cs
Out[2]: <QuerySet [<Compilation: 2023/MAR_FKt230303/Figures/PuyDesFolles_MAUV_Topo1m>, <Compilation: 2023/MAR_FKt230303/Figures/PuyDesFolles_MAUV_Topo1m>]>

In [3]: cs[0].cmd_filename
Out[3]: <FieldFile: /mbari/SeafloorMapping/2023/MAR_FKt230303/Figures/Figures_MGDS.cmd>

In [4]: cs[1].cmd_filename
Out[4]: <FieldFile: /mbari/SeafloorMapping/2023/MAR_FKt230303/Figures/Figures_PuyDesFolles.cmd>

How should the loader or query logic deal with situations like this where files named like Figures_MGDS.cmd and Figures_PuyDesFolles.cmd are producing the same figure?

@MBARIMike
Copy link
Contributor Author

The database got reloaded yesterday with several fixes for finding and displaying Compilations. The Compilations detail page now shows warning(s) if multiple names are found, e.g. for https://smdb.shore.mbari.org/compilations/2023-mar_fkt230303-figures-puydesfolles_mauv_topo1m/. (The warnings don't behave properly due to my lack of CSS understanding, but this is a temporary fix for now.)

@jbpaduan
Copy link
Collaborator

jbpaduan commented Nov 6, 2024

Compilations page is not reliably finding *SlopeNav.jpgs; is it looking only for ZTopoSlopeNav.jpg in filenames? Can it display any JPG with the root of the grid name? Interestingly, in the list of compilations that 20241017m1 is used in, it finds the name of the thumbnail that isn't rendering on the page for the compilation:
https://smdb.shore.mbari.org/compilations/2024-montereycanyon-figures-seismosands_2024_topo2m/

@MBARIMike
Copy link
Contributor Author

The _thumbnail_filename() method should find that SeismoSands_2024_Topo2m_SlopeNav.jpg file because it uses a "*" in the glob(). A message got written to last weekend's load log:

INFO 2024-11-03 20:58:16,942 load.py _thumbnail_filename():1457 Searched /mbari/SeafloorMapping/2024/MontereyCanyon/Figures/SeismoSands_2024_Topo2m* and found no files with extensions = ('jpg', 'jpeg', 'png', 'tif')

Yet the file has a date 2 days later:

docker_user@smdb:/opt/SeafloorMappingDB$ ls -l /mbari/SeafloorMapping/2024/MontereyCanyon/Figures/SeismoSands_2024_Topo2m*.jpg
-rwx------ 1 paje 4294967294 1720550 Nov  5 09:45 /mbari/SeafloorMapping/2024/MontereyCanyon/Figures/SeismoSands_2024_Topo2m_SlopeNav.jpg

Is it possible that the file didn't exist last weekend when the load ran?

@jbpaduan
Copy link
Collaborator

jbpaduan commented Nov 6, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
original use case Translated from Use Case document
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@MBARIMike @jbpaduan