load_data_proc.sh not working #3356

lauragails · 2023-04-28T21:10:45Z

lauragails
Apr 28, 2023

Describe the bug
I am trying to load and annotate my VCF using load_data_dataproc.sh and SeqrVCFToMTTask keeps failing. I posted this in a discussion, but the post is not properly labeled. @hanars has been extremely helpful debugging thus far, answering my other posts!

Link to page(s) where bug is occurring
Command analagous to below is not working. Below is copied from the tutorial:

run the loading command in the pipeline-runner container. Adjust the arguments as needed

BUILD_VERSION=38                 # can be 37 or 38
SAMPLE_TYPE=WES                  # can be WES or WGS
INDEX_NAME=your-dataset-name     # the desired index name to output. Will be used later to link the data to the corresponding seqr project
 
INPUT_FILE_PATH=/${GS_FILE_PATH}/${FILENAME}  
 
docker-compose exec pipeline-runner load_data_dataproc.sh $BUILD_VERSION $SAMPLE_TYPE $INDEX_NAME $GS_BUCKET $INPUT_FILE_PATH

Scope of the bug
I am experiencing this when running my entire vcf file. The bucket does not have an extra slash RE location (I carried this slash throughout). But maybe it's possible that it's messing up the code downstream too? All of the gs:// filepaths exists as they should.

Screenshots
Here is my entire log:

sudo docker-compose exec pipeline-runner /usr/local/bin/laura_loading.sh 
+ GS_FILE_PATH=data/GRCh38
+ FILENAME=AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz
+ BUILD_VERSION=38
+ SAMPLE_TYPE=WES
+ INDEX_NAME=peru_test
+ GS_BUCKET=gs://gala-mssm/seqr_docker/
+ INPUT_FILE_PATH=/data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz
+ case ${BUILD_VERSION} in
+ FULL_BUILD_VERSION=GRCh38
+ SOURCE_FILE=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz
+ DEST_FILE=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt
+ REFERENCE_DATA_BUCKET=gs://gala-mssm/seqr_docker//reference_data/GRCh38
+ cd /seqr-loading-pipelines/luigi_pipeline
+ NETWORK=seaver
+ hailctl dataproc submit seqr-loading-cluster seqr_loading.py --pyfiles lib,../hail_scripts SeqrVCFToMTTask --local-scheduler --source-paths gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz --dest-path gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt --genome-version 38 --sample-type WES --vep-config-json-path gs://gala-mssm/seqr_docker//reference_data/GRCh38/vep-GRCh38-loftee-dataproc.json --reference-ht-path gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht --clinvar-ht-path gs://gala-mssm/seqr_docker//reference_data/GRCh38/clinvar.GRCh38.ht --dont-validate
Submitting to cluster 'seqr-loading-cluster'...
gcloud command:
gcloud dataproc jobs submit pyspark seqr_loading.py \
    --files= \
    --py-files=/tmp/pyscripts_nswva85u.zip \
    --properties= \
    -- \
    SeqrVCFToMTTask \
    --local-scheduler \
    --source-paths \
    gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz \
    --dest-path \
    gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt \
    --genome-version \
    38 \
    --sample-type \
    WES \
    --vep-config-json-path \
    gs://gala-mssm/seqr_docker//reference_data/GRCh38/vep-GRCh38-loftee-dataproc.json \
    --reference-ht-path \
    gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht \
    --clinvar-ht-path \
    gs://gala-mssm/seqr_docker//reference_data/GRCh38/clinvar.GRCh38.ht \
    --dont-validate
Job [c65cd7ddc50c471a94752be28a0d3ee1] submitted.
Waiting for job output...
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
SLF4J: Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
SLF4J: Ignoring binding found at [jar:file:/usr/lib/spark/jars/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
/opt/conda/default/lib/python3.10/site-packages/luigi/parameter.py:296: UserWarning: Parameter "hgmd_ht_path" with value "None" is not of type string.
  warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if SeqrVCFToMTTask(source_paths=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz, dest_path=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt, genome_version=38, vep_runner=VEP, reference_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=, clinvar_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=True, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/vep-GRCh38-loftee-dataproc.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz, hail_temp_dir=) is complete
DEBUG: Checking if VcfFile(filename=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz) is complete
INFO: Informed scheduler that task   SeqrVCFToMTTask_gs___gala_mssm_s_VARIANTS_gs___gala_mssm_s_09a2bb5724   has status   PENDING
INFO: Informed scheduler that task   VcfFile_gs___gala_mssm_s_b6c539a3a5   has status   DONE
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 14608] Worker Worker(salt=2168684944, workers=1, host=seqr-loading-cluster-m, username=root, pid=14608) running   SeqrVCFToMTTask(source_paths=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz, dest_path=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt, genome_version=38, vep_runner=VEP, reference_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=, clinvar_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=True, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/vep-GRCh38-loftee-dataproc.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz, hail_temp_dir=)
Initializing Hail with default parameters...
Running on Apache Spark version 3.3.0
SparkUI available at http://seqr-loading-cluster-m.c.seaver-wes.internal:45579
Welcome to
     __  __     <>__
    / /_/ /__  __/ /
   / __  / _ `/ / /
  /_/ /_/\_,_/_/_/   version 0.2.114-cc8d36408b36
LOGGING: writing to /home/hail/hail-20230428-2038-0.2.114-cc8d36408b36.log
ERROR: [pid 14608] Worker Worker(salt=2168684944, workers=1, host=seqr-loading-cluster-m, username=root, pid=14608) failed    SeqrVCFToMTTask(source_paths=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz, dest_path=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt, genome_version=38, vep_runner=VEP, reference_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=, clinvar_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=True, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/vep-GRCh38-loftee-dataproc.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz, hail_temp_dir=)
Traceback (most recent call last):
  File "/opt/conda/default/lib/python3.10/site-packages/luigi/worker.py", line 203, in run
    new_deps = self._run_get_new_deps()
  File "/opt/conda/default/lib/python3.10/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
    task_gen = self.task.run()
  File "/tmp/c65cd7ddc50c471a94752be28a0d3ee1/seqr_loading.py", line 81, in run
    check_if_path_exists(source_path, "source_path")
  File "/tmp/c65cd7ddc50c471a94752be28a0d3ee1/seqr_loading.py", line 23, in check_if_path_exists
    raise ValueError(f"{label} path not found: {path}")
ValueError: source_path path not found: gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   SeqrVCFToMTTask_gs___gala_mssm_s_VARIANTS_gs___gala_mssm_s_09a2bb5724   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 1 pending tasks possibly being run by other workers
DEBUG: There are 1 pending tasks unique to this worker
DEBUG: There are 1 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=2168684944, workers=1, host=seqr-loading-cluster-m, username=root, pid=14608) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 2 tasks of which:
* 1 complete ones were encountered:
    - 1 VcfFile(filename=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz)
* 1 failed:
    - 1 SeqrVCFToMTTask(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

ERROR: (gcloud.dataproc.jobs.submit.pyspark) Job [c65cd7ddc50c471a94752be28a0d3ee1] failed with error:
Google Cloud Dataproc Agent reports job failure. If logs are available, they can be found at:
https://console.cloud.google.com/dataproc/jobs/c65cd7ddc50c471a94752be28a0d3ee1?project=seaver-wes&region=us-east1
gcloud dataproc jobs wait 'c65cd7ddc50c471a94752be28a0d3ee1' --region 'us-east1' --project 'seaver-wes'
https://console.cloud.google.com/storage/browser/dataproc-staging-us-east1-1094890130017-19cojukm/google-cloud-dataproc-metainfo/2ae02e04-f1ca-4c80-85d6-6ebd6c66bd0f/jobs/c65cd7ddc50c471a94752be28a0d3ee1/
gs://dataproc-staging-us-east1-1094890130017-19cojukm/google-cloud-dataproc-metainfo/2ae02e04-f1ca-4c80-85d6-6ebd6c66bd0f/jobs/c65cd7ddc50c471a94752be28a0d3ee1/driveroutput
Traceback (most recent call last):
  File "/usr/local/bin/hailctl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/__main__.py", line 112, in main
    dataproc_cli.main(args)
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/cli.py", line 124, in main
    jmp[args.module].main(args, pass_through_args))
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/submit.py", line 88, in main
    gcloud.run(cmd)
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/gcloud.py", line 9, in run
    return subprocess.check_call(["gcloud"] + command)
  File "/usr/local/lib/python3.7/subprocess.py", line 341, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['gcloud', 'dataproc', 'jobs', 'submit', 'pyspark', 'seqr_loading.py', '--cluster=seqr-loading-cluster', '--files=', '--py-files=/tmp/pyscripts_nswva85u.zip', '--properties=', '--', 'SeqrVCFToMTTask', '--local-scheduler', '--source-paths', 'gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz', '--dest-path', 'gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt', '--genome-version', '38', '--sample-type', 'WES', '--vep-config-json-path', 'gs://gala-mssm/seqr_docker//reference_data/GRCh38/vep-GRCh38-loftee-dataproc.json', '--reference-ht-path', 'gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht', '--clinvar-ht-path', 'gs://gala-mssm/seqr_docker//reference_data/GRCh38/clinvar.GRCh38.ht', '--dont-validate']' returned non-zero exit status 1.

Note that I did have to upgrade my hail (per a different issue here)
And I also see in the hail logfile:

2023-04-28 20:00:57.312 Hail: WARN: This Hail JAR was compiled for Spark 3.3.0, running with Spark 3.3.2.
  Compatibility is not guaranteed.
2023-04-28 20:00:59.736 Hail: INFO: SparkUI: http://0522db40d9c8:4040
2023-04-28 20:01:00.130 Hail: INFO: Running Hail version 0.2.114-cc8d36408b36

hanars · 2023-05-01T15:12:23Z

hanars
May 1, 2023
Maintainer

Your GS_BUCKET has an trailing slash - it should be gs://gala-mssm/seqr_docker not gs://gala-mssm/seqr_docker/

0 replies

lauragails · 2023-05-01T15:26:17Z

lauragails
May 1, 2023
Author

Except this is the path to my copied data, which match what the script is reading (see below).

To get rid of the slash I would have to move everything "up" a bucket. I've been hesitant to do this, but would you recommend me doing that? Thank you!

gsutil ls gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht

gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht/README.txt
gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht/_SUCCESS
gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht/metadata.json.gz
gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht/globals/
gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht/index/
gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht/rows/

gsutil ls gs://gala-mssm/seqr_docker/reference_data/GRCh38/combined_reference_data_grch38.ht
CommandException: One or more URLs matched no objects.

gsutil ls gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz
gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz

gsutil ls gs://gala-mssm/seqr_docker/data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz
CommandException: One or more URLs matched no objects.

1 reply

hanars May 1, 2023
Maintainer

The issue is when you run the pipeline you are getting the error ValueError: source_path path not found: gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz

I am not sure that moving everything up a level will fix your problem, but the fact that your gsutil ls command is working but the pipeline is reporting it can't find the file makes me think that google does not handle the double slash in data paths the way you would want. I might be wrong and it is in fact a different issue, but to me that seems the most likely.

The gsutil mv command should be very fast, so I think just running the following should fix the issue:
gsutil mv gs://gala-mssm/seqr_docker//data gs://gala-mssm/seqr_docker/data

lauragails · 2023-05-01T19:24:07Z

lauragails
May 1, 2023
Author

Unfortunately that didn't get rid of the error. But I also have the same issue with the reference data. I ran the same move command with that directory, and because each file is being copied and moved, I suspect it will take a while.

But I did notice that when this command was generated above (and when I re-ran) there was an empty flag. Do you think this would cause problems? Or can you think of anything else? I will let you know when I try again after everything gets moved.

Thank you again!

gcloud dataproc jobs submit pyspark seqr_loading.py \
    --files= \
    --py-files=/tmp/pyscripts_nswva85u.zip \
    --properties= \
    -- \
    SeqrVCFToMTTask \
    --local-scheduler \
    --source-paths \
    gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz \
    --dest-path \
    gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt \
    --genome-version \
    38 \
    --sample-type \
    WES \
    --vep-config-json-path \
    gs://gala-mssm/seqr_docker//reference_data/GRCh38/vep-GRCh38-loftee-dataproc.json \
    --reference-ht-path \
    gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht \
    --clinvar-ht-path \
    gs://gala-mssm/seqr_docker//reference_data/GRCh38/clinvar.GRCh38.ht \
    --dont-validate

0 replies

lauragails · 2023-05-09T21:00:36Z

lauragails
May 9, 2023
Author

It does look like that did the trick for the VCF file reading, because I could see all of the checks that were done on the VCF on the screen, that weren't done before.

I now hit a different error, that I suspect has more to do with the permissions I have for the cloud, but posting it here just too in case you have any suggestions:

  VEP Error output:
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.

I emailed our administrator about this, but will follow up if a seqr software specific question comes up. Thank you again!

edit I have already tried this https://www.digitalocean.com/community/questions/how-to-fix-docker-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socket

1 reply

hanars May 11, 2023
Maintainer

this sounds like an issue with docker not running/ not being correctly configured, but let me know if it turns out to be a seqr-specific issue

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

load_data_proc.sh not working #3356

{{title}}

Replies: 4 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

load_data_proc.sh not working #3356

lauragails Apr 28, 2023

Replies: 4 comments · 2 replies

hanars May 1, 2023 Maintainer

lauragails May 1, 2023 Author

hanars May 1, 2023 Maintainer

lauragails May 1, 2023 Author

lauragails May 9, 2023 Author

hanars May 11, 2023 Maintainer

lauragails
Apr 28, 2023

Replies: 4 comments 2 replies

hanars
May 1, 2023
Maintainer

lauragails
May 1, 2023
Author

hanars May 1, 2023
Maintainer

lauragails
May 1, 2023
Author

lauragails
May 9, 2023
Author

hanars May 11, 2023
Maintainer