Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

URGENT: two of the bdb databases does not download with action #45

Open
marvinm2 opened this issue Aug 29, 2023 · 6 comments
Open

URGENT: two of the bdb databases does not download with action #45

marvinm2 opened this issue Aug 29, 2023 · 6 comments
Assignees

Comments

@marvinm2
Copy link
Collaborator

marvinm2 commented Aug 29, 2023

From the gene.json file (https://data.bridgedb.org/gene.json), the following two .bridge files do not download:

  • Ec_Derby_Ensembl_91.bridge
  • Mx_Derby_Ensembl_85.bridge

Because of this, the Docker image does not work.

@egonw egonw removed their assignment Sep 10, 2023
@egonw
Copy link
Member

egonw commented Sep 10, 2023

@tabbassidaloii, can you help out here? What is the status of these files?

@marvinm2, the JSON files have fields to indicate compatibility. I think we should add there the "tested with Docker" option, and just exclude these two files from that for now.

@tabbassidaloii
Copy link
Member

@marvinm2 both Ec_Derby_Ensembl_91 and Mx_Derby_Ensembl_85.bridge are on Zenodo (https://zenodo.org/record/3667670 and https://zenodo.org/record/5638771/, respectively), and the links are correctly provided in gene.json file. I just tried and both are working.

@marvinm2
Copy link
Collaborator Author

marvinm2 commented Nov 8, 2023

I experience these issues when starting the service

org.bridgedb.IDMapperException: Could not connect to database: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.bridgedb.rdb.SimpleGdbFactory.createInstance(SimpleGdbFactory.java:64)
	at org.bridgedb.rdb.IDMapperRdb$DriverPgdb.connect(IDMapperRdb.java:57)
	at org.bridgedb.BridgeDb.connect(BridgeDb.java:54)
	at org.bridgedb.rdb.GdbProvider.fromConfigFile(GdbProvider.java:164)
	at org.bridgedb.webservicetesting.BridgeDbWebservice.RestletService.connectGdbs(RestletService.java:216)
	at org.bridgedb.webservicetesting.BridgeDbWebservice.RestletService.start(RestletService.java:116)
	at org.restlet.routing.Filter.start(Filter.java:265)
	at org.restlet.routing.Router.start(Router.java:814)
	at org.restlet.Component.startRouters(Component.java:626)
	at org.restlet.Component.start(Component.java:568)
	at org.restlet.Restlet.handle(Restlet.java:306)
	at org.restlet.Component.handle(Component.java:405)
	at org.restlet.Server.handle(Server.java:507)
	at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
	at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
	at org.restlet.engine.connector.HttpServerHelper$1.handle(HttpServerHelper.java:64)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:717)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:689)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:115)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:141)
	at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:252)
	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2855)
	at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:411)
	at org.apache.derby.iapi.jdbc.InternalDriver$1.run(InternalDriver.java:660)
	at org.apache.derby.iapi.jdbc.InternalDriver$1.run(InternalDriver.java:656)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.iapi.jdbc.InternalDriver.getNewEmbedConnection(InternalDriver.java:654)
	at org.apache.derby.iapi.jdbc.InternalDriver.connect(InternalDriver.java:300)
	at org.apache.derby.iapi.jdbc.InternalDriver.connect(InternalDriver.java:967)
	at org.apache.derby.iapi.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:136)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251)
	at org.bridgedb.rdb.SimpleGdbFactory.createInstance(SimpleGdbFactory.java:59)
	... 24 more
Caused by: ERROR XJ040: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:300)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory.java:170)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:75)
	... 38 more
Caused by: java.nio.file.NoSuchFileException: /opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1259)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:733)
	at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:850)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:248)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:177)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:191)
	at org.apache.derby.impl.io.JarStorageFactory.doInit(JarStorageFactory.java:111)
	at org.apache.derby.impl.io.BaseStorageFactory.init(BaseStorageFactory.java:88)
	at org.apache.derby.impl.io.JarStorageFactory.init(JarStorageFactory.java:35)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.privGetStorageFactoryInstance(StorageFactoryService.java:222)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.access$400(StorageFactoryService.java:73)
	at org.apache.derby.impl.services.monitor.StorageFactoryService$12.run(StorageFactoryService.java:977)
	at org.apache.derby.impl.services.monitor.StorageFactoryService$12.run(StorageFactoryService.java:971)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.getCanonicalServiceName(StorageFactoryService.java:969)
	at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1515)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:999)
	at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:598)
	at org.apache.derby.impl.jdbc.EmbedConnection$4.run(EmbedConnection.java:4042)
	at org.apache.derby.impl.jdbc.EmbedConnection$4.run(EmbedConnection.java:4038)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.jdbc.EmbedConnection.startPersistentService(EmbedConnection.java:4036)
	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2816)
	... 35 more
Unable to start the Restlet
The org.restlet.Component class has no Restlet defined to process calls. Maybe it wasn't properly started.
Unable to start the Restlet

org.bridgedb.IDMapperException: Could not connect to database: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.bridgedb.rdb.SimpleGdbFactory.createInstance(SimpleGdbFactory.java:64)
	at org.bridgedb.rdb.IDMapperRdb$DriverPgdb.connect(IDMapperRdb.java:57)
	at org.bridgedb.BridgeDb.connect(BridgeDb.java:54)
	at org.bridgedb.rdb.GdbProvider.fromConfigFile(GdbProvider.java:164)
	at org.bridgedb.webservicetesting.BridgeDbWebservice.RestletService.connectGdbs(RestletService.java:216)
	at org.bridgedb.webservicetesting.BridgeDbWebservice.RestletService.start(RestletService.java:116)
	at org.restlet.routing.Filter.start(Filter.java:265)
	at org.restlet.routing.Router.start(Router.java:814)
	at org.restlet.Component.startRouters(Component.java:626)
	at org.restlet.Component.start(Component.java:568)
	at org.restlet.Restlet.handle(Restlet.java:306)
	at org.restlet.Component.handle(Component.java:405)
	at org.restlet.Server.handle(Server.java:507)
	at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
	at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
	at org.restlet.engine.connector.HttpServerHelper$1.handle(HttpServerHelper.java:64)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:717)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:689)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:115)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:141)
	at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:252)
	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2855)
	at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:411)
	at org.apache.derby.iapi.jdbc.InternalDriver$1.run(InternalDriver.java:660)
	at org.apache.derby.iapi.jdbc.InternalDriver$1.run(InternalDriver.java:656)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.iapi.jdbc.InternalDriver.getNewEmbedConnection(InternalDriver.java:654)
	at org.apache.derby.iapi.jdbc.InternalDriver.connect(InternalDriver.java:300)
	at org.apache.derby.iapi.jdbc.InternalDriver.connect(InternalDriver.java:967)
	at org.apache.derby.iapi.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:136)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251)
	at org.bridgedb.rdb.SimpleGdbFactory.createInstance(SimpleGdbFactory.java:59)
	... 24 more
Caused by: ERROR XJ040: Failed to start database 'jar:(/opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge)database' with class loader jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, see the next exception for details.
	at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:300)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory.java:170)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:75)
	... 38 more
Caused by: java.nio.file.NoSuchFileException: /opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1259)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:733)
	at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:850)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:248)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:177)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:191)
	at org.apache.derby.impl.io.JarStorageFactory.doInit(JarStorageFactory.java:111)
	at org.apache.derby.impl.io.BaseStorageFactory.init(BaseStorageFactory.java:88)
	at org.apache.derby.impl.io.JarStorageFactory.init(JarStorageFactory.java:35)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.privGetStorageFactoryInstance(StorageFactoryService.java:222)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.access$400(StorageFactoryService.java:73)
	at org.apache.derby.impl.services.monitor.StorageFactoryService$12.run(StorageFactoryService.java:977)
	at org.apache.derby.impl.services.monitor.StorageFactoryService$12.run(StorageFactoryService.java:971)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.getCanonicalServiceName(StorageFactoryService.java:969)
	at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1515)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:999)
	at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:598)
	at org.apache.derby.impl.jdbc.EmbedConnection$4.run(EmbedConnection.java:4042)
	at org.apache.derby.impl.jdbc.EmbedConnection$4.run(EmbedConnection.java:4038)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.jdbc.EmbedConnection.startPersistentService(EmbedConnection.java:4036)
	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2816)
	... 35 more
Unable to start the Restlet
The org.restlet.Component class has no Restlet defined to process calls. Maybe it wasn't properly started.

@marvinm2
Copy link
Collaborator Author

marvinm2 commented Nov 8, 2023

Are these files valid? They do not have QC notes

@egonw
Copy link
Member

egonw commented Nov 10, 2023

For Ec (which seems valid):

$ bash qc.sh  ~/Downloads/Ec_Derby_Ensembl_91.bridge ~/Downloads/Ec_Derby_Ensembl_91.bridge
INFO: old database is Ensembl 91 (build: 20180509)
INFO: new database is Ensembl 91 (build: 20180509)
INFO: Number of ids in T (GeneOntology): 3660 (unchanged)
INFO: Number of ids in En (Ensembl): 4140 (unchanged)
INFO: Number of ids in S (UniProtKB): 8241 (unchanged)
INFO: Attribute provided: Type
INFO: Attribute provided: Description
INFO: Attribute provided: Symbol
INFO: Attribute provided: Chromosome
INFO: new size is 12 Mb (changed +0.0%)
INFO: OLD database has a total number of identifiers of 16041
INFO: OLD database has a total number of mappings of 31933
INFO: OLD database has Schema Version is less than 4, and we cannot calculate Primary and Secondary identifier counts
INFO: NEW database has a total number of identifiers of 16041
INFO: NEW database has a total number of mappings of 31933
INFO: NEW database has Schema Version is less than 4, and we cannot calculate Primary and Secondary identifier counts
ERROR: 4140/4140 (100%) ids do not match expected pattern for Ensembl
ERROR: expected pattern is '^ENS[A-Z]*[FPTG]\d{11}$'
ERROR: aberrant ids are e.g. 'b4471', 'b1227', 'b4470', 'b1223', 'b1224', 'b1225', 'b1226', 'b1220', 'b1221', 'b1222'
ERROR: 4103/8241 (49%) ids do not match expected pattern for UniProtKB
ERROR: expected pattern is '^([A-N,R-Z][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9])|([O,P,Q][0-9][A-Z, 0-9][A-Z, 0-9][A-Z, 0-9][0-9])(\.\d+)?|([A-N,R-Z][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9])$'
ERROR: aberrant ids are e.g. 'UPI000012FD85', 'UPI000012FD86', 'UPI0000168087', 'UPI000012FD87', 'UPI00003B25FD', 'UPI000012FD88', 'UPI000016825D', 'UPI000012FD89', 'UPI000012FD90', 'UPI0001505B8D'

And Mx (also seems valid):

$ bash qc.sh  ~/Downloads/Mx_Derby_Ensembl_85.bridge ~/Downloads/Mx_Derby_Ensembl_85.bridge 
INFO: old database is Ensembl 1 (build: 20161018)
INFO: new database is Ensembl 1 (build: 20161018)
INFO: Number of ids in T (GeneOntology): 2106 (unchanged)
INFO: Number of ids in En (Ensembl): 4018 (unchanged)
INFO: Number of ids in S (UniProtKB): 7955 (unchanged)
INFO: Attribute provided: Type
INFO: Attribute provided: Description
INFO: Attribute provided: Symbol
INFO: Attribute provided: Chromosome
INFO: new size is 11 Mb (changed +0.0%)
INFO: OLD database has a total number of identifiers of 14079
INFO: OLD database has a total number of mappings of 26457
INFO: OLD database has Schema Version is less than 4, and we cannot calculate Primary and Secondary identifier counts
INFO: NEW database has a total number of identifiers of 14079
INFO: NEW database has a total number of mappings of 26457
INFO: NEW database has Schema Version is less than 4, and we cannot calculate Primary and Secondary identifier counts
ERROR: 4018/4018 (100%) ids do not match expected pattern for Ensembl
ERROR: expected pattern is '^ENS[A-Z]*[FPTG]\d{11}$'
ERROR: aberrant ids are e.g. 'Rv0753c', 'Rv1176c', 'Rv3716c', 'Rv2919c', 'Rv2918c', 'Rv1103c', 'Rv2349c', 'Rv2129c', 'Rv3432c', 'Rv3047c'
ERROR: 3978/7955 (50%) ids do not match expected pattern for UniProtKB
ERROR: expected pattern is '^([A-N,R-Z][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9])|([O,P,Q][0-9][A-Z, 0-9][A-Z, 0-9][A-Z, 0-9][0-9])(\.\d+)?|([A-N,R-Z][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9][A-Z][A-Z, 0-9][A-Z, 0-9][0-9])$'
ERROR: aberrant ids are e.g. 'UPI00000D0FEC', 'UPI000012B714', 'UPI00000C1538', 'UPI00001650EB', 'UPI000012B72A', 'UPI000016527E', 'UPI000013944B', 'UPI00001652CE', 'UPI00000D4FCB', 'UPI00000D1117'

@egonw
Copy link
Member

egonw commented Nov 10, 2023

@marvinm2, I also not the Caused by: java.nio.file.NoSuchFileException: /opt/bridgedb-databases/Ec_Derby_Ensembl_91.bridge in the stacktrace. That suggests something else. Is this the correct stacktrace, or after you removed downloading it, but not before you removed it from the gdb.config?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants