Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error when setting up ModelDB via Docker Compose #993

Open
RaduStoicescu opened this issue Jul 14, 2020 · 6 comments
Open

Error when setting up ModelDB via Docker Compose #993

RaduStoicescu opened this issue Jul 14, 2020 · 6 comments
Assignees
@ravishetye @anandjakhaniya

Comments

@RaduStoicescu
Copy link

On Windows 10 when running docker-compose -f docker-compose-all.yaml up I get the following:

modeldb-backend | {"thread":"main","level":"DEBUG","loggerName":"ai.verta.modeldb.utils.ModelDBHibernateUtil","message":"ModelDBHibernateUtil getSessionFactory() retrying for DB connection after 2560 millisecond ","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1594737612,"nanoOfSecond":72000000},"threadId":1,"threadPriority":5,"hostName":"03fd2e8e7248","kubernetes.podIP":""} modeldb-backend | {"thread":"main","level":"WARN","loggerName":"ai.verta.modeldb.utils.ModelDBHibernateUtil","message":"ModelDBHibernateUtil checkDBConnection() got error ","thrown":{"commonElementCount":0,"localizedMessage":"The connection attempt failed.","message":"The connection attempt failed.","name":"org.postgresql.util.PSQLException","cause":{"commonElementCount":19,"localizedMessage":"Connect timed out","message":"Connect timed out","name":"java.net.SocketTimeoutException","extendedStackTrace":"java.net.SocketTimeoutException: Connect timed out\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]\n\tat java.net.Socket.connect(Socket.java:648) ~[?:?]\n\tat org.postgresql.core.PGStream.<init>(PGStream.java:75) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ~[postgresql-42.2.6.jar!/:42.2.6]\n"},"extendedStackTrace":"org.postgresql.util.PSQLException: The connection attempt failed.\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:292) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.Driver.makeConnection(Driver.java:458) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.Driver.connect(Driver.java:260) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat java.sql.DriverManager.getConnection(DriverManager.java:677) ~[java.sql:?]\n\tat java.sql.DriverManager.getConnection(DriverManager.java:228) ~[java.sql:?]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.checkDBConnection(ModelDBHibernateUtil.java:483) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.checkDBConnectionInLoop(ModelDBHibernateUtil.java:325) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.createOrGetSessionFactory(ModelDBHibernateUtil.java:240) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.App.initializeServicesBaseOnDataBase(App.java:363) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.App.main(App.java:260) [classes!/:1.0-SNAPSHOT]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]\n\tat jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]\n\tat java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\nCaused by: java.net.SocketTimeoutException: Connect timed out\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]\n\tat java.net.Socket.connect(Socket.java:648) ~[?:?]\n\tat org.postgresql.core.PGStream.<init>(PGStream.java:75) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ~[postgresql-42.2.6.jar!/:42.2.6]\n\t... 19 more\n"},"endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1594737622,"nanoOfSecond":77000000},"threadId":1,"threadPriority":5,"hostName":"03fd2e8e7248","kubernetes.podIP":""}
@ravishetye
Copy link

ravishetye commented Jul 14, 2020 •
edited
Loading

@RaduStoicescu : modeldb backend is unable to connect to postgres , can you verify it postgres container is up?

I am not sure how this is done on windows but this is a list of container which needs to be up for modeldb to work correctly.

 (⎈) ~/w/c/o/k/l/e/p/m/backend> docker ps
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS              PORTS                              NAMES
5d8ad539fcf0        vertaaiofficial/modeldb-frontend:2.0.7   "docker-entrypoint.s…"   8 seconds ago       Up 7 seconds        0.0.0.0:3000->3000/tcp             modeldb-frontend
cecb257725c5        postgres                                 "docker-entrypoint.s…"   11 seconds ago      Up 10 seconds       5432/tcp                           modeldb-postgres
1d02914e811a        vertaaiofficial/modeldb-proxy:2.0.7      "/app/bin/proxy"         11 days ago         Up 8 seconds        0.0.0.0:8080->8080/tcp             modeldb-proxy
84abdfb07132        vertaaiofficial/modeldb-graphql:2.0.7    "/app/bin/graphql"       11 days ago         Up 8 seconds        0.0.0.0:4000->4000/tcp             modeldb-graphql
271c82d3f177        vertaaiofficial/modeldb-backend:2.0.7    "java -jar modeldb-1…"   11 days ago         Up 9 seconds        0.0.0.0:8085-8086->8085-8086/tcp   modeldb-backend

If postgres had problems coming up it should also show up earlier in the logs you shared.

@RaduStoicescu
Copy link
Author

RaduStoicescu commented Jul 14, 2020 •
edited
Loading

Looks like it has trouble having access to "/var/lib/postgresql/data/pgdata". I'm not sure how to create or how to set permissions for that directory inside the Docker


This user must also own the server process.


The database cluster will be initialized with locale "en_US.utf8".

The default database encoding has accordingly been set to "UTF8".

The default text search configuration will be set to "english".


Data page checksums are disabled.


fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok

creating subdirectories ... ok

selecting dynamic shared memory implementation ... posix

selecting default max_connections ... 20

selecting default shared_buffers ... 400kB

selecting default time zone ... Etc/UTC

creating configuration files ... ok

2020-07-14 19:43:17.163 UTC [82] FATAL: data directory "/var/lib/postgresql/data/pgdata" has wrong ownership

2020-07-14 19:43:17.163 UTC [82] HINT: The server must be started by the user that owns the data directory.

child process exited with exit code 1

initdb: removing contents of data directory "/var/lib/postgresql/data/pgdata"```

@anandjakhaniya
Copy link
Contributor

Hey @RaduStoicescu,
I am a contributor to modeldb backend and have docker-compose working in my setup on windows.
My set up is

Windows 10 Home single language
I have used docker toolbox (https://docs.docker.com/toolbox/toolbox_install_windows/) setup for the dockerization of applications.

this is my docker-compose-all.yaml file

version: "3.5"
services:
    modeldb-backend:
        image: vertaaiofficial/modeldb-backend:2.0.7
        container_name: "modeldb-backend"
        expose:
            - "8085"
            - "8086"
        ports:
            - "8085:8085"
            - "8086:8086"
        networks:
            - modeldb_network
        environment:
            - VERTA_MODELDB_CONFIG:/config/config.yaml
        volumes:
            - /c/Users/config:/config/
            - /c/Users/artifact-store:/artifact-store/
    modeldb-proxy:
        image: vertaaiofficial/modeldb-proxy:2.0.7
        container_name: "modeldb-proxy"
        environment:
            MDB_ADDRESS: "modeldb-backend:8085"
            SERVER_HTTP_PORT: "8080"
        expose:
            - "8080"
        ports:
            - "8080:8080"
        depends_on:
            - modeldb-backend
        networks:
            - modeldb_network
    modeldb-graphql:
        image: vertaaiofficial/modeldb-graphql:2.0.7
        container_name: "modeldb-graphql"
        environment:
            MDB_ADDRESS: "modeldb-backend:8085"
            SERVER_HTTP_PORT: "4000"
            QUERY_PATH: "/api/v1/graphql/query"
        expose:
            - "4000"
        ports:
            - "4000:4000"
        depends_on:
            - modeldb-backend
        networks:
            - modeldb_network
    modeldb-frontend:
        image: vertaaiofficial/modeldb-frontend:2.0.7
        container_name: "modeldb-frontend"
        expose:
            - "3000"
        ports:
            - "3000:3000"
        depends_on:
            - modeldb-proxy
            - modeldb-graphql
        environment:
            DEPLOYED: "yes"
            BACKEND_API_PROTOCOL: "http"
            BACKEND_API_DOMAIN: "modeldb-webapp:3000"
            MDB_ADDRESS: http://modeldb-proxy:8080
            ARTIFACTORY_ADDRESS: http://modeldb-backend:8086
            GQL_ADDRESS: http://modeldb-graphql:4000
        networks:
            - modeldb_network
    modeldb-postgres:
        image: postgres
        container_name: "modeldb-postgres"
        restart: always
        environment:
            POSTGRES_PASSWORD: root
            PGDATA: /var/lib/postgresql/data/pgdata
        networks:
            - modeldb_network
        volumes:
            - /e/Users/data:/var/lib/postgresql/data/pgdata
networks:
  modeldb_network:

where I have used shared /c/Users volume for config & artifact-store and for the Postgres data I am using other drive location like /e/Users/data .
I could not use /c/Users as mapped volume for the Postgres because it is by default read-only access by the windows and you can not do any write operation on it. Postgres will create backup volume on shared volume so you needs write permission on the shared volume.
Can you try to use other drive location instead of c: as a mapped volume for Postgres as shown in docker-compose-all.yaml above.

I hope you can replicate my setup and that unblocks you. If not, then let me know your Windows version and about the docker setup, and then I can help you better.

@RaduStoicescu
Copy link
Author

RaduStoicescu commented Jul 16, 2020 •
edited
Loading

I changed it like this and I have the same problem

I have Windows 10 Business, version 1903

I have found this on docker forums, it looks like a common problem: https://forums.docker.com/t/data-directory-var-lib-postgresql-data-pgdata-has-wrong-ownership/17963

        image: postgres
        container_name: "modeldb-postgres"
        restart: always
        environment:
            POSTGRES_PASSWORD: root
            PGDATA: /var/lib/postgresql/data/pgdata
        networks:
            - modeldb_network
        volumes:
            - /d/users/data:/var/lib/postgresql/data/pgdata

This is the error:

modeldb-postgres    | This user must also own the server process.
modeldb-postgres    |
modeldb-postgres    | The database cluster will be initialized with locale "en_US.utf8".
modeldb-postgres    | The default database encoding has accordingly been set to "UTF8".
modeldb-postgres    | The default text search configuration will be set to "english".
modeldb-postgres    |
modeldb-postgres    | Data page checksums are disabled.
modeldb-postgres    |
modeldb-postgres    | fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
modeldb-postgres    | creating subdirectories ... ok
modeldb-postgres    | selecting dynamic shared memory implementation ... posix
modeldb-postgres    | selecting default max_connections ... 20
modeldb-postgres    | selecting default shared_buffers ... 400kB
modeldb-postgres    | selecting default time zone ... Etc/UTC
modeldb-postgres    | creating configuration files ... ok
modeldb-postgres    | running bootstrap script ... 2020-07-16 17:56:19.080 UTC [82] FATAL:  data directory "/var/lib/postgresql/data/pgdata" has wrong ownership
modeldb-postgres    | 2020-07-16 17:56:19.080 UTC [82] HINT:  The server must be started by the user that owns the data directory.
modeldb-postgres    | child process exited with exit code 1
modeldb-postgres    | initdb: removing contents of data directory "/var/lib/postgresql/data/pgdata"```

@ravishetye
Copy link

@RaduStoicescu : were you able to bring up Postgres? if so could you share the high-level steps so that we could add it to the modeldb documentation.
If you can point us to the steps which unblocked you at a high level, someone from our team could update the documents.
If you rather prefer to contribute a PR you could add a section for Windows setup under https://github.com/VertaAI/modeldb/blob/master/DEPLOY.md#Deploy-pre-published-images.

@daviddwlee84
Copy link

Basically found a similar issue on Ubuntu 22.04 with latest docker installed

docker compose -f docker-compose-all.yaml up

modeldb-postgres  | PostgreSQL Database directory appears to contain a database; Skipping initialization                                                                                                        
modeldb-postgres  |
modeldb-postgres  | 2024-04-17 06:35:33.234 UTC [1] LOG:  starting PostgreSQL 16.2 (Debian 16.2-1.pgdg120+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit 
modeldb-postgres  | 2024-04-17 06:35:33.235 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432                                                                                                        
modeldb-postgres  | 2024-04-17 06:35:33.235 UTC [1] LOG:  listening on IPv6 address "::", port 5432
modeldb-postgres  | 2024-04-17 06:35:33.236 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"                                                                                          
modeldb-postgres  | 2024-04-17 06:35:33.240 UTC [29] LOG:  database system was shut down at 2024-04-17 06:24:56 UTC                                                                                             
modeldb-postgres  | 2024-04-17 06:35:33.248 UTC [1] LOG:  database system is ready to accept connections                                                                                                        
modeldb-graphql   | 2024/04/17 06:35:33 SERVER_HTTP_PORT : 4000                                                                                                                                                 
modeldb-proxy     | 2024/04/17 06:35:33 MDB_ADDRESS : modeldb-backend:8085                                                                                                                                      
modeldb-proxy     | 2024/04/17 06:35:33 SERVER_HTTP_PORT : 8080
modeldb-proxy     | 2024/04/17 06:35:33 Starting verta-backend proxy on port : 8080                                                                                                                             
modeldb-frontend  | yarn run v1.22.4
modeldb-frontend  | warning package.json: No license field                                                                                                                                                      
modeldb-frontend  | $ nodemon server.js
modeldb-frontend  | [nodemon] 1.19.4                                                                                                                                                                            
modeldb-frontend  | [nodemon] to restart at any time, enter `rs`
modeldb-frontend  | [nodemon] watching dir(s): *.*
modeldb-frontend  | [nodemon] watching extensions: js,mjs,json
modeldb-frontend  | [nodemon] starting `node server.js`
modeldb-frontend  | [HPM] Proxy created: /  ->  http://modeldb-proxy:8080
modeldb-frontend  | [HPM] Proxy rewrite rule created: "^/api/v1/modeldb" ~> "/v1"
modeldb-frontend  | [HPM] Proxy created: /  ->  http://modeldb-backend:8086
modeldb-frontend  | [HPM] Proxy created: /  ->  http://modeldb-graphql:4000
modeldb-frontend  | [HPM] Proxy rewrite rule created: "^/api/v1/graphql/" ~> "/" 


Repeating...

modeldb-backend   | {"thread":"main","level":"DEBUG","loggerName":"ai.verta.modeldb.utils.ModelDBHibernateUtil","message":"ModelDBHibernateUtil getSessionFactory() retrying for DB connection after 2560 millisecond ","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1713335805,"nanoOfSecond":539000000},"threadId":1,"threadPriority":5,"hostName":"9c2449b11b85","kubernetes.podIP":""}
modeldb-backend   | {"thread":"main","level":"WARN","loggerName":"ai.verta.modeldb.utils.ModelDBHibernateUtil","message":"ModelDBHibernateUtil checkDBConnection() got error ","thrown":{"commonElementCount":0,"localizedMessage":"org.mariadb.jdbc.Driver","message":"org.mariadb.jdbc.Driver","name":"java.lang.ClassNotFoundException","extendedStackTrace":"java.lang.ClassNotFoundException: org.mariadb.jdbc.Driver\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:435) ~[?:?]\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[?:?]\n\tat org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:89) ~[modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]\n\tat java.lang.Class.forName0(Native Method) ~[?:?]\n\tat java.lang.Class.forName(Class.java:340) ~[?:?]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.checkDBConnection(ModelDBHibernateUtil.java:477) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.checkDBConnectionInLoop(ModelDBHibernateUtil.java:325) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.createOrGetSessionFactory(ModelDBHibernateUtil.java:240) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.App.initializeServicesBaseOnDataBase(App.java:368) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.App.main(App.java:263) [classes!/:1.0-SNAPSHOT]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]\n\tat jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]\n\tat java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n"},"endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1713335805,"nanoOfSecond":541000000},"threadId":1,"threadPriority":5,"hostName":"9c2449b11b85","kubernetes.podIP":""}


Try adding a Project on UI

modeldb-frontend  | /api/v1/modeldb/hydratedData/findHydratedProjects
modeldb-frontend  | Requesting /api/v1/modeldb/hydratedData/findHydratedProjects
modeldb-frontend  | /static/media/Verta_logo.b2461278.svg
modeldb-frontend  | Returning 503 Service Unavailable; 261b sent                                                                                                                                                 modeldb-frontend  | /favicon.png                                                                                                                                                                                 modeldb-frontend  | /api/v1/modeldb/hydratedData/findHydratedProjects
modeldb-frontend  | Requesting /api/v1/modeldb/hydratedData/findHydratedProjects
modeldb-frontend  | Returning 503 Service Unavailable; 261b sent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ravishetye ravishetye

@anandjakhaniya anandjakhaniya

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@daviddwlee84 @ravishetye @RaduStoicescu @anandjakhaniya