Merge branch 'master' into itb

* master: (68 commits)
  Fix tests (SOFTWARE-5862)
  Document "production" boolean (SOFTWARE-5862)
  Add "production" boolean to resources in namespaces JSON (SOFTWARE-5862)
  Add scheduled downtime for GLOW T2_US_Wisconsin for HDFS upgrade
  changing DN
  Add downtime for SDSC_NRP_OSDF_S3_ORIGIN due to HW issue
  Add downtime for SDSC_NRP_OSDF_CACHE due to HW issue
  Add downtime for Stashcache-UofA due to overloaded
  Add downtime for CARDIFF_UK_OSDF_CACHE due to overloaded
  Remove extraneous space
  Add UNL institution ID
  Create UNL_Yin.yaml
  Create UW_Loverde.yaml
  Add facility and site for Riga Technical University CMS Tier-2 (FD #76435)
  Retire old resources
  Create UCSD_Sfiligoi
  Adds downtime for uw it ce due to maintenance
  Add downtime for CINCINNATI_INTERNET2_OSDF_CACHE due to #toPelican
  Add downtime for BOISE_INTERNET2_OSDF_CACHE due to #toPelican
  Add downtime for MGHPCC_NRP_OSDF_CACHE due to HW issue
  Update SPRACE_downtime.yaml
  Update SPRACE_downtime.yaml
  Removing downtime - #toPelican
  Update SPRACE_downtime.yaml
  Add Institution ID to projects/SJSU_IT.yaml
  Update FANDM-ITS_downtime.yaml
  Adding scitoken issuer for fdp/gyrokinetic
  New project: SJSU_IT
  Add downtime for UKI-SCOTGRID-ECDF-STASHCACHE due to offline - checking with the admin
  Changing downtime time
  Update UERJ_downtime.yaml
  Add downtime for DENVER_INTERNET2_OSDF_CACHE due to '#toPelican'
  Add downtime for SUT-STASHCACHE due to Hardware migration
  Update topology/University of Wisconsin/CHTC/CHTC.yaml
  Use the A record until we transition
  Avoid switching the /ospool/PROTECTED writeback until we transition
  Make sure to set AllowedVOs (INF-1796)
  Make sure the issuer matches
  First attempt at registering new AP40 Pelican origin (INF-1796)
  Fix hostname in SDSC NRP FDP OSDF cache
  Update BNL-ATLAS.yaml
  Add downtime for NCAR_OSDF_ORIGIN due to Firewall issues
  Changing S3 port to 8444
  Downtime for Georgia Tech PACE OSG 2 due to Site Maintenance
  Add downtime for  Stashcache-Chicago due to Hardware issues RAID problem
  Update projects/IGWN_Staff.yaml
  Create IGWN_Staff.yaml
  Create PSU_Lynch.yaml
  Upgrade Werkzeug to 3.0.3+
  adding endpoint and auth_endpoint #toPelican
  ...

Dockerfile

@@ -30,7 +30,9 @@ RUN pip3 install --no-cache-dir -r requirements-apache.txt
 # Create data directory, and gather SSH keys for git
 RUN mkdir                  /data && \
     chown -v apache:apache /data && \
-    ssh-keyscan github.com bitbucket.org >> /etc/ssh/ssh_known_hosts
+    ssh-keyscan github.com bitbucket.org >> /etc/ssh/ssh_known_hosts && \
+    git config --global --add safe.directory /data/app/topology && \
+    git config --global --add safe.directory /data/app/contact
 
 # Add fetch-crl cronjob
 # Add daily restart of httpd to load renewed certificates

mappings/project_institution.yaml

@@ -58,6 +58,7 @@ OHSU: "Oregon Health & Science University"
 OSU: "The Ohio State University"
 Pitt: "University of Pittsburgh"
 PortlandState: "Portland State University"
+PFW: "Purdue University Fort Wayne"
 PSI: "Planetary Science Institute"
 PSU: "Pennsylvania State University"
 Repertoire: "Repertoire Immune Medicines"

projects/CUBoulder_Piper.yaml

@@ -0,0 +1,10 @@
+Department: INSTAAR
+Description: My role is to expose the community of scientists at CSDMS (https://csdms.colorado.edu)
+  to high-throughput computing. I plan to make a series of demonstration cases, run
+  them through OSG, and write up my experiences to highlight how HTC could be useful
+  for research in this community.
+FieldOfScience: Earth Sciences
+FieldOfScienceID: 40.0601b
+InstitutionID: https://osg-htc.org/iid/gpeckuwpdnrs
+Organization: University of Colorado Boulder
+PIName: Mark Piper

projects/IGWN_Staff.yaml

@@ -0,0 +1,7 @@
+Department: Physics
+Description: IGWN staff
+FieldOfScience: Gravitational Physics
+FieldOfScienceID: '40.0202'
+InstitutionID: https://osg-htc.org/iid/uvf22j6xjbtv
+Organization: Georgia Institute of Technology
+PIName: James Clark

projects/NCAR_Schuster.yaml

@@ -0,0 +1,10 @@
+Department: Computational and Information System Lab
+Description: >-
+  We plan to test the capabilities of the OSPool to run climate science
+  research related data analytics workflows using datasets made accessible through
+  NSF NCAR's integration with the Open Science Data Federation.
+FieldOfScience: Atmospheric Sciences
+FieldOfScienceID: '40.0401'
+InstitutionID: https://osg-htc.org/iid/hfo8pge14iwg
+Organization: National Center for Atmospheric Research
+PIName: Douglas Schuster

projects/PFW_Sharma.yaml

@@ -0,0 +1,9 @@
+Department: Department of Chemistry and Biochemistry
+Description: 'Investigating protein-protein interactions related to the human immune
+  system. Other projects include immune protein-drug interactions. We mainly use molecular
+  dynamics simulations and molecular coking methodologies for our study. '
+FieldOfScience: Chemistry
+FieldOfScienceID: '40.0511'
+InstitutionID: 'https://osg-htc.org/iid/dgsgv9qbvgce'
+Organization: Purdue University Fort Wayne
+PIName: Arjun Sharma

projects/PSU_Lynch.yaml

@@ -0,0 +1,17 @@
+Department: Plant Science
+Description: "OpenSimRoot is an open-source, 3D functional\u2013structural plant model\
+  \ that simulates root system architecture 3 dimensionally, accounting for interactions\
+  \ between soil physical properties, root growth and associated metabolic costs thus\
+  \ enabling study of the value of root traits for resource acquisition and plant\
+  \ growth. As a research tool it supports experimental designs and mechanistic understanding\
+  \ of underlying processes. We aim to understand plant\u2013soil interactions in\
+  \ a wide range of soil taxa, especially with respect to degraded soils in developing\
+  \ countries, in order to address the knowledge gap that is critical for global food\
+  \ security. https://plantscience.psu.edu/research/labs/roots/methods/computer-analysis-tools\
+  \ https://plantscience.psu.edu/research/labs/roots/publications/overviews/opensimroot-widening-the-scope-and-application-of-root-architectural-models\
+  \ https://rootmodels.gitlab.io/ "
+FieldOfScience: Plant Biology
+FieldOfScienceID: '01.1101'
+InstitutionID: https://osg-htc.org/iid/f1tlj6c19ppg
+Organization: Pennsylvania State University
+PIName: Jonathan  Lynch

projects/SJSU_IT.yaml

@@ -0,0 +1,10 @@
+Department: IT/HPC
+Description: Our goals are to explore and learn about the OSG computing environment
+  so we can help and share the knowledge with SJSU researchers to enable them to utilize
+  OSG HPC for their research projects.
+FieldOfScience: Computer and Information Services
+FieldOfScienceID: '11.0101'
+InstitutionID: https://osg-htc.org/iid/0r71ijbk5mbz
+Organization: San Jose State University
+PIName: Atul Pala
+

projects/UCSD_Sfiligoi

@@ -0,0 +1,8 @@
+Department: San Diego Supercomputer Center
+Description: The mission of the Global Infrastructure Lab (GIL) is to test and evaluate
+  infrastructure software. This project is used to exercise the OSPool services.
+FieldOfScience: Computer Science
+FieldOfScienceID: '11.0501'
+InstitutionID: https://osg-htc.org/iid/06wup3aye2t7
+Organization: University of California, San Diego
+PIName: Igor Sfiligoi

projects/UMontana_Hansen.yaml

@@ -0,0 +1,9 @@
+Department: Division of Biological Sciences
+Description: Study the structure, function, and pharmacology of ligand-gated ion channels.
+  We are interested in running virtual screenings to identify novel ligands with therapeutic
+  potential in brain disorders. Se more at https://hansen-neurolab.com/
+FieldOfScience: Biological and Biomedical Sciences
+FieldOfScienceID: '26.9999'
+InstitutionID: https://osg-htc.org/iid/sdmbw89obfoi
+Organization: University of Montana
+PIName: Kasper Hansen

projects/UNL_Yin.yaml

@@ -0,0 +1,10 @@
+Department: Food Science and Technology
+Description: "Our lab is a Bioinformatics and Computational Biology Lab. We have the\
+  \ following major research interests:1. Plant and microbial bioinformatics 2. Genome\
+  \ biology and evolutionary genomics\r\n3. Carbohydrate metabolism and secondary\
+  \ metabolism "
+FieldOfScience: Biological and Biomedical Sciences
+FieldOfScienceID: '26.9999'
+InstitutionID:  'https://osg-htc.org/iid/q9k1b8dfrw25'
+Organization: University of Nebraska - Lincoln
+PIName: Yanbin Yin

projects/UW_Loverde.yaml

@@ -0,0 +1,12 @@
+Department: Physics
+Description: "The University of Washington Dark Universe Science Center brings experts\
+  \ in dark matter, dark energy, inflation, and gravity together with those who study\
+  \ how these invisible ingredients drive the evolution of the Universe along with\
+  \ the stars, black holes, and galaxies within it. By combining observations of the\
+  \ cosmos with direct measurements of the Universe\u2019s invisible content, we hope\
+  \ to fundamentally change our understanding of Nature. (https://sites.google.com/uw.edu/dusc)"
+FieldOfScience: Physics
+FieldOfScienceID: '40.0202'
+InstitutionID: https://osg-htc.org/iid/8lpmoeouw66o
+Organization: University of Washington
+PIName: Marilena Loverde

requirements-rootless.txt

@@ -28,6 +28,6 @@ requests~=2.31.0
 six~=1.16.0
 smmap~=5.0.0
 urllib3~=2.0.2
-Werkzeug~=2.3.4
+Werkzeug~=3.0.3
 WTForms~=3.0.1
 xmltodict~=0.13.0

src/README.md

@@ -522,6 +522,7 @@ Each cache in the list contains the following attributes:
 - `endpoint`: The `<HOST>:<PORT>` of the public (`xrootd@stash-cache`) service
 - `auth_endpoint`: The `<HOST>:<PORT>` of the authenticated (`xrootd@stash-cache-auth`) service
 - `resource`: The resource name of the cache.
+- `production`: true if the resource is in "production" (as opposed to ITB)
 
 The JSON also contains an attribute `namespaces` that is a list of namespaces with the following attributes:
 - `path` is the path of the namespace
@@ -553,11 +554,13 @@ The final result looks like
     {
       "auth_endpoint": "osg-gftp.pace.gatech.edu:8443",
       "endpoint": "osg-gftp.pace.gatech.edu:8000",
+      "production": true,
       "resource": "Georgia_Tech_PACE_GridFTP"
     },
     {
       "auth_endpoint": "osg-gftp2.pace.gatech.edu:8443",
       "endpoint": "osg-gftp2.pace.gatech.edu:8000",
+      "production": true,
       "resource": "Georgia_Tech_PACE_GridFTP2"
     }
   ],
@@ -567,6 +570,7 @@ The final result looks like
         {
           "auth_endpoint": "rds-cache.sdsc.edu:8443",
           "endpoint": "rds-cache.sdsc.edu:8000",
+          "production": true,
           "resource": "RDS_AUTH_OSDF_CACHE"
         }
       ],

src/stashcache.py

@@ -564,7 +564,17 @@ def _service_resource_dict(
                 if not is_null(svc, "Details", "auth_endpoint_override"):
                     auth_endpoint = svc["Details"]["auth_endpoint_override"]
                 break
-        return {"endpoint": endpoint, "auth_endpoint": auth_endpoint, "resource": r.name}
+        production = None
+        try:
+            production = bool(r.rg.production)
+        except AttributeError:
+            pass
+        return {
+            "endpoint": endpoint,
+            "auth_endpoint": auth_endpoint,
+            "resource": r.name,
+            "production": production,
+        }
 
     def _cache_resource_dict(r: Resource):
         return _service_resource_dict(r=r, service_name=XROOTD_CACHE_SERVER, auth_port_default=8443, unauth_port_default=8000)

src/tests/test_api.py

@@ -769,7 +769,8 @@ class TestEndpointContent:
         },
         'Tags': ['CC*']
     }
-    mock_resource = Resource("AMNH-ARES", mock_resource_information, global_data.get_topology().common_data)
+    mock_resource = Resource("AMNH-ARES", mock_resource_information, global_data.get_topology().common_data,
+                             mock_resource_group)
 
     mock_facility.add_site(mock_site)
     mock_site.add_resource_group(mock_resource_group)

src/tests/test_stashcache.py

@@ -1,5 +1,6 @@
 from configparser import ConfigParser
 import copy
+
 import flask
 import pytest
 import re
@@ -279,6 +280,7 @@ def validate_cache_schema(cc):
         assert HOST_PORT_RE.match(cc["auth_endpoint"])
         assert HOST_PORT_RE.match(cc["endpoint"])
         assert cc["resource"] and isinstance(cc["resource"], str)
+        assert "production" in cc and isinstance(cc["production"], (type(None), bool))
 
     @staticmethod
     def validate_namespace_schema(ns):

src/webapp/common.py

@@ -222,6 +222,13 @@ def trim_space(s: str) -> str:
     ret = re.sub(r"(?m)^[ \t]+", "", ret)
     return ret
 
+
+def fix_newlines(in_str: str) -> str:
+    """Replace Windows newlines with Unix newlines in a string;
+    other CR characters are replaced with a space"""
+    return in_str.replace("\r\n", "\n").replace("\r", " ")
+
+
 def run_git_cmd(cmd: List, dir=None, git_dir=None, ssh_key=None) -> bool:
     """
     Run git command, optionally specifying ssh key and/or git dirs

src/webapp/forms.py

@@ -8,6 +8,7 @@
     TextAreaField, SubmitField, TimeField, DateField
 from wtforms.validators import InputRequired, ValidationError
 
+from .common import fix_newlines, trim_space
 from . import models
 
 UTCOFFSET_CHOICES = [
@@ -350,6 +351,13 @@ def __init__(self, *args, **kwargs):
         self.field_of_science.data = kwargs.get("field_of_science", self.field_of_science.data)
         self.field_of_science_id.data = kwargs.get("field_of_science_id", self.field_of_science_id.data)
         self.description.data = kwargs.get("description", self.description.data)
+        try:
+            self.description.data = trim_space(
+                fix_newlines(
+                    self.description.data.strip())
+            )
+        except (TypeError, AttributeError):
+            pass
 
         self.infos = ""
 

src/webapp/topology.py

@@ -109,7 +109,7 @@ def is_ccstar(self):
         return self._is_ccstar
 
 class Resource(object):
-    def __init__(self, name: str, yaml_data: ParsedYaml, common_data: CommonData):
+    def __init__(self, name: str, yaml_data: ParsedYaml, common_data: CommonData, rg: "ResourceGroup"):
         self.name = name
         self.service_types = common_data.service_types
         self.common_data = common_data
@@ -124,6 +124,7 @@ def __init__(self, name: str, yaml_data: ParsedYaml, common_data: CommonData):
             raise ValueError(f"Resource {name} does not have an FQDN")
         self.fqdn = self.data["FQDN"]
         self.id = self.data["ID"]
+        self.rg = rg
 
     def get_stashcache_files(self, global_data, legacy):
         """Gets a resources Cache files as a dictionary"""
@@ -378,14 +379,14 @@ def __init__(self, name: str, yaml_data: ParsedYaml, site: Site, common_data: Co
         self.support_center = OrderedDict([("ID", scid), ("Name", scname)])
 
         self.resources_by_name = {}
-        for name, res in yaml_data["Resources"].items():
+        for res_name, res in yaml_data["Resources"].items():
             try:
                 if not isinstance(res, dict):
                     raise TypeError("expecting a dict")
-                res_obj = Resource(name, ParsedYaml(res), self.common_data)
-                self.resources_by_name[name] = res_obj
+                res_obj = Resource(res_name, ParsedYaml(res), self.common_data, rg=self)
+                self.resources_by_name[res_name] = res_obj
             except (AttributeError, KeyError, TypeError, ValueError) as err:
-                log.exception("Error with resource %s: %r", name, err)
+                log.exception("Error with resource %s: %r", res_name, err)
                 continue
 
         self.data = yaml_data

topology/Brookhaven National Laboratory/BNL ATLAS Tier1/BNL-ATLAS.yaml

@@ -220,14 +220,14 @@ Resources:
           ID: OSG1000212
     Description: An OSG CE resource at BNL, mainly serving USATLAS production jobs,
       not open to general OSG users.
-    FQDN: gridgk07.racf.bnl.gov
+    FQDN: gridgk07.sdcc.bnl.gov
     ID: 438
     Services:
       CE:
         Description: Compute Element
         Details:
           hidden: false
-          sam_uri: htcondor://gridgk07.racf.bnl.gov
+          sam_uri: htcondor://gridgk07.sdcc.bnl.gov
     VOOwnership:
       ATLAS: 100
     WLCGInformation:
@@ -261,14 +261,14 @@ Resources:
           ID: OSG1000212
     Description: This CE will serve grid jobs submitted to the *local* USATLAS resources,
       e.g. for panda queue ANALY_BNL_LOCAL, at BNL.
-    FQDN: gridgk08.racf.bnl.gov
+    FQDN: gridgk08.sdcc.bnl.gov
     ID: 545
     Services:
       CE:
         Description: Compute Element
         Details:
           hidden: false
-          sam_uri: htcondor://gridgk08.racf.bnl.gov
+          sam_uri: htcondor://gridgk08.sdcc.bnl.gov
     VOOwnership:
       ATLAS: 100
     WLCGInformation:

topology/Cardiff University/Cardiff Computing Cluster/CardiffPRPCachingInfrastructure.yaml

@@ -17,6 +17,10 @@ Resources:
         Primary:
           Name: Fabio Andrijauskas
           ID: OSG1000162
+      Site Contact:
+        Primary:
+          Name: Duncan MacLeod
+          ID: OSG1000688
     FQDN: stashcache.gravity.cf.ac.uk
     DN: /C=UK/O=eScience/OU=Cardiff/L=WeSC/CN=stashcache.gravity.cf.ac.uk
     Services:
@@ -38,6 +42,10 @@ Resources:
         Primary:
           Name: Fabio Andrijauskas
           ID: OSG1000162
+      Site Contact:
+        Primary:
+          Name: Duncan MacLeod
+          ID: OSG1000688
     FQDN: cf-ac-uk-cache.nationalresearchplatform.org
     DN: /DC=org/DC=incommon/C=US/ST=California/O=University of California, San Diego/CN=cf-ac-uk-cache.nationalresearchplatform.org
     Services:

topology/Cardiff University/Cardiff Computing Cluster/CardiffPRPCachingInfrastructure_downtime.yaml

@@ -53,3 +53,26 @@
   Services:
   - XRootD cache server
 # ---------------------------------------------------------
+- Class: SCHEDULED
+  ID: 1794218483
+  Description: 'Scheduled power outage'
+  Severity: Outage
+  StartTime: May 01, 2024 12:00 +0100
+  EndTime: May 02, 2024 17:00 +0000
+  CreatedTime: May 01, 2024 11:09 +0100
+  ResourceName: CARDIFF_UK_OSDF_CACHE
+  Services:
+  - XRootD cache server
+# ---------------------------------------------------------
+- Class: UNSCHEDULED
+  ID: 1808101910
+  Description: overloaded
+  Severity: Outage
+  StartTime: May 14, 2024 19:30 +0000
+  EndTime: May 31, 2024 19:30 +0000
+  CreatedTime: May 15, 2024 21:56 +0000
+  ResourceName: CARDIFF_UK_OSDF_CACHE
+  Services:
+  - XRootD cache server
+# ---------------------------------------------------------
+

topology/Clemson University/Clemson-Palmetto/Clemson-Palmetto.yaml

@@ -3,7 +3,7 @@ GroupID: 212
 Production: true
 Resources:
   Clemson-Connect:
-    Active: true
+    Active: false
     ContactLists:
       Administrative Contact:
         Primary:
@@ -30,7 +30,7 @@ Resources:
           hidden: false
           uri_override: [email protected]/pbs
   Clemson-Palmetto:
-    Active: true
+    Active: false
     ContactLists:
       Administrative Contact:
         Primary:

topology/Cybera/Cybera - Edmonton/CyberaEdmonton.yaml

@@ -4,6 +4,7 @@ GroupID: 519
 GroupDescription: Edmonton branch of Cybera
 Resources:
   CYBERA_EDMONTON:
+    Active: false
     ID: 1032
     Description: Cybera Edmonton CE
     ContactLists: