Error summary: NoRouteToHostException: No route to host (Host unreachable)

[anjalijain22]

Hi Hana,

Hope you're well!

I'm encountering a NoRouteToHostException: No route to host (Host unreachable) error while running the seqr pipeline on an hg38 dataset. Below are more details:

Java stack trace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 147 in stage 6.0 failed 1 times, most recent failure: Lost task 147.0 in stage 6.0 (TID 3147) (88f03545b3d2 executor driver): java.io.IOException: Error accessing gs://hail-common/references/grch38_to_grch37.over.chain.gz

Hail version: 0.2.95-513139587f57
Error summary: NoRouteToHostException: No route to host (Host unreachable)
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   SeqrVCFToMTTask__seqr_reference__VARIANTS__input_vcfs_GRCh_ad3dee1934   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 2 pending tasks possibly being run by other workers
DEBUG: There are 2 pending tasks unique to this worker
DEBUG: There are 2 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=439025064, workers=1, host=88f03545b3d2, username=root, pid=410857) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 3 tasks of which:
* 1 complete ones were encountered:
    - 1 VcfFile(filename=/input_vcfs/GRCh38/Muise/allSamples.hc.vqsr.vt.annovar.filters.passing.subset.vcf.gz)
* 1 failed:
    - 1 SeqrVCFToMTTask(...)
* 1 were left pending, among these:
    * 1 had failed dependencies:
        - 1 SeqrMTToESTask(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

When I run the following command: docker-compose exec pipeline-runner gsutil ls gs://hail-common/references/grch38_to_grch37.over.chain.gz it works fine to list the file in google storage. I also re-ran the pipeline thinking it may be a one off error due to connectivity issues but unfortunately ran into the same issue again.

Would appreciate your insight on this!

Thanks,
Anjali

[hanars]

Can you please include the full logs/ stacktrace

[anjalijain22]

Sure, here it is - err_logs_WES_38.txt

[hanars]

This is probably not the only issue you are having, but one problem seems to be that there are a lot of unanticipated annotations on your input VCF. The input VCF for seqr loading should be the minimal output of GATK joint calling, and can not have additional annotations, particularly vep. Please make sure to delete any intermediate files that may have been output by the pipeline, and run the loading again using an unannotated VCF

MT using schema class SeqrVariantsAndGenotypesSchema already has "filters" annotation.
MT using schema class SeqrVariantsAndGenotypesSchema already has "rg37_locus" annotation.
MT using schema class SeqrVariantsAndGenotypesSchema already has "rsid" annotation.
MT using schema class SeqrVariantsAndGenotypesSchema already has "vep" annotation.

It also looks like you are using the dont-validate flag, which we discourage. Unless there is a compelling reason why you need to load data that is failing validation, this should be avoided. Can you provide the command you are running?

[anjalijain22]

I see, I think there are a few annotations in the VCF but I'm fairly certain VEP annotations are not present. I checked the VCF again and there is no CSQ field.

Also, I had to use the dont-validate flag as seqr was running into the below validation error and we are certain that the dataset is WES:

SeqrValidationError: Sample type validation error: dataset sample-type is specified as WES but appears to be WES because it contains many common non-coding variants

Below is the command I'm using:

docker-compose exec pipeline-runner python3 -m seqr_loading SeqrMTToESTask --local-scheduler --reference-ht-path "/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht" --clinvar-ht-path "/seqr-reference-data/GRCh38/clinvar.GRCh38.ht" --vep-config-json-path "/vep_configs/vep-GRCh38-loftee.json" --es-host elasticsearch --es-index-min-num-shards 1 --sample-type "WES" --es-index "muise_wes_cohort" --genome-version "38" --source-paths "/input_vcfs/GRCh38/Muise/allSamples.hc.vqsr.vt.annovar.filters.passing.subset.vcf.gz" --dest-path "/input_vcfs/GRCh38/Muise/Muise_allSamples.mt" --dont-validate > /mnt/data/seqr/pipeline-logs/err_file_muise_wes_cohort-38_5.txt

[hanars]

The file path .hc.vqsr.vt.annovar.filters generally is not the name of a VCF that is unannotated, it implies you have run your VCF through a bunch of different tools. While I highlighted VEP as a definite problem, in general the seqr pipeline is not able to handle VCFs that have been run through any other annotation pipeline. If this is a GATK joint called VCF you can send me the header of your VCF and I can verify for you which if any fields are causing issues, but if you generated this VCF in any way that is not GATK joint calling it is unlikely that it will be possible to get it to load.

Alrternatively, this might be an issue with intermediate files. If, for example, there exists /input_vcfs/GRCh38/Muise/Muise_allSamples.mt that might be a cached output file that is being reused and causing issues

[anjalijain22]

The VCF does have some annotations apart from VEP which could be causing an issue. I started off with easier solution of removing the /input_vcfs/GRCh38/Muise/Muise_allSamples.mt and re-running the pipeline-runner and it worked! Thanks for your help with this!