Can't load Upload New Callset via Edit Datasets in project

[atb1r21]

Hi,

I have recently upgraded seqr and data is being ingested correctly and I can see that everything is working. There is an issue on the seqr web interface, where when I create a project and attempt to use the Edit Datasets function it doesn't work.

Previously I could just pull the .tsv files from the ID Mapping File Path from /vep_data on our local install and it would work correctly. I can see that the files are available in the correct area of the pipeline-runner and that our seqr instance can see the files that are in /vep_data or /input_vcfs when I look in the Data Management -> Elasticsearch status page.

The issue when attempting to upload the .tsv is that I get the following error: [Errno 2] No such file or directory: '/vep_data/map_newtest.tsv'

Any advice would be greatly appreciated as I can see that the file is in the correct place, and additionally I have given 777 permissions to the directory and this file in /vep_data to make sure that there are no permissions issues here.

Thanks,
Alister

[hanars]

Hi Alister,

Can you please explain what you are using the ID mapping file for? The recommended way of using seqr is that Individual IDs in seqr should match the IDs in the VCF, and if this is really not possible then a mapping file should be provided to the load data command so they are remapped during the loading process. Providing a mapping file at the "Edit Dataset" step is really not the recommended way of using seqr, so I would recommend against doing this at all

[atb1r21]

Hi,

I passed that to the users of our seqr instance and they have managed to resolve that issue:

'I’ve fixed it! There was a mismatch with the ids in the vcf and I’ve I’ve updated the family and pedigree files and now I can load the data in without a mapping file!'

This is allowing the data to be loaded in, which is good progress, so thank you very much for the advice the users found it very helpful.

Whilst we're discussing I am seeing the following when attempting to use the Variant Search:

relation "reference_data_genecopynumbersensitivity" does not exist LINE 1: ...erence_data_genecopynumbersensitivity"."pTS" FROM "reference... ^

I have loaded in the reference data from the provided steps for an on prem installation, is there something I have possibly missed here?

Thanks again,
Alister

[hanars]

It's possible that after the update there was an error updating the database. Try running the following:

docker-compose exec seqr python manage.py migrate --database=reference_data
docker-compose exec seqr python manage.py update_gene_cn_sensitivity

[atb1r21]

After going into the container and running the above I've got the following outputs:

6c0371d6a7bb:/seqr]$ python manage.py migrate --database=reference_data
Operations to perform:
Apply all migrations: admin, auth, contenttypes, guardian, matchmaker, panelapp, reference_data, seqr, sessions, social_django
Running migrations:
Applying reference_data.0001_initial...Traceback (most recent call last):
File "/opt/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 82, in _execute
return self.cursor.execute(sql)
psycopg2.errors.DuplicateTable: relation "reference_data_dbnsfpgene" already exists

So it seems the database already exists.

The second command regarding sensitivity appears to show the more verbose error of what I am seeing when doing a search:

6c0371d6a7bb:/seqr]$ python manage.py update_gene_cn_sensitivity
{"timestamp": "2022-06-15 14:47:12,472", "severity": "INFO", "message": "Updating <reference_data.management.commands.update_gene_cn_sensitivity.CNSensitivityReferenceDataHandler object at 0x7f9f8ea4f550>"}
{"timestamp": "2022-06-15 14:47:12,648", "severity": "INFO", "message": "Re-using /tmp/pHI_pTS_scores.Jan21.txt previously downloaded from https://storage.googleapis.com/seqr-reference-data/cn_sensitivity/pHI_pTS_scores.Jan21.txt"}
{"timestamp": "2022-06-15 14:47:12,648", "severity": "INFO", "message": "Parsing file"}
17263 records [00:00, 36189.08 records/s]
{"timestamp": "2022-06-15 14:47:13,133", "severity": "ERROR", "message": "relation "reference_data_genecopynumbersensitivity" does not exist\nLINE 1: SELECT COUNT() AS "__count" FROM "reference_data_genecopynu...\n ^\n", "traceback": "Traceback (most recent call last):\n File "/opt/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute\n return self.cursor.execute(sql, params)\npsycopg2.errors.UndefinedTable: relation "reference_data_genecopynumbersensitivity" does not exist\nLINE 1: SELECT COUNT() AS "__count" FROM "reference_data_genecopynu...\n ^\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/seqr/reference_data/management/commands/utils/update_utils.py", line 111, in update_records\n logger.info("Deleting {} existing {} records".format(model_objects.count(), model_name))\n File "/opt/venv/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method\n return getattr(self.get_queryset(), name)(args, **kwargs)\n File "/opt/venv/lib/python3.7/site-packages/django/db/models/query.py", line 412, in count\n return self.query.get_count(using=self.db)\n File "/opt/venv/lib/python3.7/site-packages/django/db/models/sql/query.py", line 519, in get_count\n number = obj.get_aggregation(using, ['__count'])['__count']\n File "/opt/venv/lib/python3.7/site-packages/django/db/models/sql/query.py", line 504, in get_aggregation\n result = compiler.execute_sql(SINGLE)\n File "/opt/venv/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql\n cursor.execute(sql, params)\n File "/opt/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 98, in execute\n return super().execute(sql, params)\n File "/opt/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 66, in execute\n return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n File "/opt/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers\n return executor(sql, params, many, context)\n File "/opt/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute\n return self.cursor.execute(sql, params)\n File "/opt/venv/lib/python3.7/site-packages/django/db/utils.py", line 90, in exit\n raise dj_exc_value.with_traceback(traceback) from exc_value\n File "/opt/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute\n return self.cursor.execute(sql, params)\ndjango.db.utils.ProgrammingError: relation "reference_data_genecopynumbersensitivity" does not exist\nLINE 1: SELECT COUNT() AS "__count" FROM "reference_data_genecopynu...\n ^\n\n", "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent"}

It seems there's a database issue here?

Thanks,
Alister

[hanars]

It looks like your reference database got into a completely broken state. I'm not sure how, but the easiest way to fix it is to drop it an repopulate it completely (this is a separate database from the one that stores user-generated data in seqr so there will be no data loss):

docker-compose exec seqr psql --host postgres -U postgres -c 'DROP DATABASE reference_data_db';
docker-compose exec seqr psql --host postgres -U postgres -c 'CREATE DATABASE reference_data_db';
docker-compose exec seqr python manage.py migrate --database=reference_data
docker-compose exec seqr python manage.py update_all_reference_data --use-cached-omim

[atb1r21]

That has worked and everything is working fine now. Thank you very much for your help.

We've had a few issues, so I am not sure where that came from, I inherited managing this system and documentation about it all has been lacking from previous admins.

Thanks again,
Alister