Upgrade to Solr 8.11.2 and remove vulnerabilities and unnecessary dependencies

[jordanpadams]

🗒️ Summary

Requires NASA-PDS/registry-legacy-solr#5

⚙️ Test Data and/or Report

Download some PDS4 test data. Here are some examples:

• Any of the sub-directories under here: https://pds.nasa.gov/data/pds4/releases/
  • e.g. https://pds.nasa.gov/data/pds4/releases/img/m2020_edlcam_raw-20220722/ or https://pds.nasa.gov/data/pds4/releases/ppi/maven.static.c-20230710/ or a few of those sub-directories
• Any of the test bundles here: https://pds.nasa.gov/data/pds4/test-data/registry/

Install registry:
Build and untar the registry-mgr-legacy package

% git clone [email protected]:NASA-PDS/registry-mgr-legacy.git
% git checkout vulnerability_fix
% mvn clean package
% tar -xzvf registry-mgr-legacy-2.3.0-SNAPSHOT-bin.tar.gz

Create environment variables:

% cd registry-mgr-legacy-2.3.0-SNAPSHOT
% export REGISTRY_HOME=$(pwd)

# If in production, add REGISTRY_HOME to .bash_profile
% echo "export REGISTRY_HOME=$(pwd)" >> $HOME/.bash_profile

# Create a DATA_HOME dir. This will get large over time so choose a location on disk with sufficient available space

% export DATA_HOME=$HOME/registry-data/
% mkdir -p $DATA_HOME

% export SOLR_DOC_HOME=$DATA_HOME/registry-legacy/data/solr-docs
% mkdir -p $SOLR_DOC_HOME

# If in production, add env vars to .bash_profile
% echo "export DATA_HOME=$DATA_HOME" >> $HOME/.bash_profile
% echo "export SOLR_DOC_HOME=$SOLR_DOC_HOME" >> $HOME/.bash_profile

Spin up a Docker container with Solr and our custom core configurations

% $REGISTRY_HOME/bin/registry_installer_docker.sh install

Verify installation:

% docker ps

CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                    NAMES
0ee274f1fa5f        registry-mgr-legacy:2.3.0-SNAPSHOT   "/bin/bash -c 'solr …"   5 minutes ago       Up 5 minutes        0.0.0.0:8983->8983/tcp   registry

% curl http://localhost:8983/solr/admin/cores?action=STATUS
{
  "responseHeader":{
    "status":0,
    "QTime":5},
  "initFailures":{},
  "status":{
    "data_shard1_replica_n1":{
      "name":"data_shard1_replica_n1",
      "instanceDir":"/var/solr/data/data_shard1_replica_n1",
      "dataDir":"/var/solr/data/data_shard1_replica_n1/data/",
      "config":"solrconfig.xml",
      "schema":"managed-schema",
      "startTime":"2023-09-11T22:21:20.716Z",
      "uptime":32698879,
      "lastPublished":"active",
      "configVersion":0,
      "cloud":{
        "collection":"data",
        "shard":"shard1",
        "replica":"core_node3",
        "replicaType":"NRT"},
      "index":{
        "numDocs":7,
        "maxDoc":7,
        "deletedDocs":0,
        "indexHeapUsageBytes":-1,
        "version":40,
        "segmentCount":1,
        "current":true,
        "hasDeletions":false,
...
...
...
        "segmentsFile":"segments_b",
        "segmentsFileSizeInBytes":268,
        "userData":{
          "commitTimeMSec":"1694474833706",
          "commitCommandVer":"1776785643228102656"},
        "lastModified":"2023-09-11T23:27:13.706Z",
        "sizeInBytes":17236,
        "size":"16.83 KB"}}}}

Install harvest-legacy:
Build and untar the harvest-legacy package

% git clone [email protected]:NASA-PDS/harvest-legacy.git
% git checkout vulnerability_fix
% mvn clean package
% tar -xzvf harvest-legacy-2.3.0-SNAPSHOT-bin.tar.gz

% cd harvest-legacy-2.3.0-SNAPSHOT
% export HARVEST_HOME=$(pwd)

# If in production, add HARVEST_HOME to .bash_profile
% echo "export HARVEST_HOME=$HARVEST_HOME" >> $HOME/.bash_profile

Update Harvest config:

# Copy master config to some place "safe"
% mkdir -p $DATA_HOME/conf
% cp $HARVEST_HOME/conf/harvest/examples/harvest-policy-master.xml $DATA_HOME/conf

# Open and Update harvest-policy-master.xml <directories> to include path to
# data you want to load

Run Harvest on Solr docs:

% $HARVEST_HOME/bin/harvest-legacy -c $DATA_HOME/conf/harvest-policy-master.xml \
     -C $HARVEST_HOME/conf/search/defaults/ \
     -o $SOLR_DOC_HOME

Load data into registry:

% $REGISTRY_HOME/bin/registry-mgr $SOLR_DOC_HOME

Go to Solr Admin interface and poke around at some queries: http://localhost:8983/solr/#/data/query

♻️ Related Issues

Resolves NASA-PDS/operations#19
Resolves NASA-PDS/tracking-service#3
Resolves NASA-PDS/software-issues-repo#2
Refs https://github.com/NASA-PDS/registry-pds3-catalog/issues/4
Refs NASA-PDS/ds-view#3
Refs NASA-PDS/search-ui-legacy#1

[nutjob4life]

Two tiny changes!

[c-suh]

From the changes in the README.md: replace "The core functionality for this service is satisfied by Apache Solr." with "Future implementations are already underway for the PDS Search API." (or append this "Future implementations..." sentence to the existing paragraph)

[jordanpadams]

@c-suh feel free to update. I do not think this holds up this PR.

[c-suh]

• github.io URL
• description (in pom vs README)
• docker_image name
• Solr/Lucene versions 8.3.1 vs 8.11.2

[nutjob4life]

👍