Connection Issue with RDBMS Configuration in Server Mode #4700

ravamo · 2024-12-11T20:05:18Z

ravamo
Dec 11, 2024

I have an Apache Hop service configured like this: /hop/hop-server.sh /data/hop/config/hop-server-config.xml

<hop-server-config>
  
  <hop-server>
    <name>Apache Hop Coetl GobCan Demo</name>
    <hostname>0.0.0.0</hostname>
    <port>9595</port>
  </hop-server>

  <!-- Join the web server thread and wait until it's finished.
  The default is true
  -->
  <joining>true</joining>

  <!-- The maximum number of log lines kept in memory by the server.
  The default is 0 which means: keep all lines
  -->
  <max_log_lines>0</max_log_lines>

  <!-- The time (in minutes) it takes for a log line to be cleaned up in memory.
  The default is 0 which means: never clean up log lines
  -->
  <max_log_timeout_minutes>1440</max_log_timeout_minutes>

  <!-- The time (in minutes) it takes for a pipeline or workflow execution to be removed from the server status.
  The default is 0 which means: never clean executions
  -->
  <object_timeout_minutes>1440</object_timeout_minutes>

  <!-- The folder to read metadata objects from so that web services and database connections for sequences can be found.
  The default is that no metadata is configured: remotely executed pipelines and workflows will have their own metadata.
  -->
  <metadata_folder>path/projects/default/metadata/</metadata_folder>

</hop-server-config>

And I have this configuration:
The problem is that when I run an ETL, I get this error, and I don't understand why, as the JSON file with the connection definition is present inside the metadata folder within rdbms.

Pipeline has allocated a new transform: [Write to log].0
Initialising 5 transforms...
ERROR: Error initializing transform [Salida Tabla]
ERROR: java.lang.NullPointerException: Cannot invoke "org.apache.hop.core.database.DatabaseMeta.getIDatabase()" because "this.data.databaseMeta" is null
        at org.apache.hop.pipeline.transforms.tableoutput.TableOutput.init(TableOutput.java:482)
        at org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66)
        at java.base/java.lang.Thread.run(Thread.java:833)
Transform [Información de sistema.0] initialized flawlessly.
Transform [Añadir constantes.0] initialized flawlessly.
ERROR: Transform [Salida Tabla.0] failed to initialize!
Transform [Establecer ejecucion.0] initialized flawlessly.
Transform [Write to log.0] initialized flawlessly.
ERROR: Unable to prepare for execution of the pipeline
ERROR: org.apache.hop.core.exception.HopException:
We failed to initialize at least one transform.  Execution can not begin!
        at org.apache.hop.pipeline.Pipeline.prepareExecution(Pipeline.java:1087)
        at org.apache.hop.pipeline.engines.local.LocalPipelineEngine.prepareExecution(LocalPipelineEngine.java:231)
        at org.apache.hop.pipeline.Pipeline.execute(Pipeline.java:527)
        at org.apache.hop.workflow.actions.pipeline.ActionPipeline.execute(ActionPipeline.java:542)
        at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:745)
        at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:888)
        at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:447)
        at org.apache.hop.workflow.Workflow.startExecution(Workflow.java:307)
        at org.apache.hop.workflow.engines.local.LocalWorkflowEngine.startExecution(LocalWorkflowEngine.java:248)
        at java.base/java.lang.Thread.run(Thread.java:833)
 Finished action [Set Execution time] (result=[false])
 Finished action [Borrar ficheros temporales] (result=[false])
 Finished action [Eliminar registros erroneos] (result=[false])
 Finished action [Set variables 2] (result=[false])
 Workflow execution finished
 Workflow duration : 0.42 seconds [  0.420" ]

This issue is related to the following topics:
1.- #4685
2.- #4634

ravamo · 2024-12-16T11:05:46Z

ravamo
Dec 16, 2024
Author

Any idea? I've recreated everything, and I still get that error even when running it with hop-run.

6 replies

ravamo Dec 16, 2024
Author

Json File : /servers/hop/hop/resources/proyect/metadata/rdbms/DB_Dl_SAMPLE.json, this config file in my local works ..

{
  "rdbms": {
    "POSTGRESQL": {
      "databaseName": "${DATABASE_NAME}",
      "pluginId": "POSTGRESQL",
      "accessType": 0,
      "hostname": "${DATABASE_HOST}",
      "password": "${DATABASE_PASSWORD}",
      "pluginName": "PostgreSQL",
      "port": "${DATABASE_PORT}",
      "attributes": {
        "SUPPORTS_TIMESTAMP_DATA_TYPE": "Y",
        "QUOTE_ALL_FIELDS": "N",
        "SUPPORTS_BOOLEAN_DATA_TYPE": "Y",
        "FORCE_IDENTIFIERS_TO_LOWERCASE": "N",
        "PRESERVE_RESERVED_WORD_CASE": "Y",
        "SQL_CONNECT": "",
        "FORCE_IDENTIFIERS_TO_UPPERCASE": "N",
        "PREFERRED_SCHEMA_NAME": ""
      },
      "manualUrl": "",
      "username": "${DATABASE_USER}"
    }
  },
  "name": "DB_Dl_SAMPLE"
}

This is prohect part of config/hop-config.json

  "projectsConfig" : {
    "enabled" : true,
    "projectMandatory" : false,
    "environmentMandatory" : false,
    "environmentsForActiveProject" : false,
    "defaultProject" : null,
    "defaultEnvironment" : null,
    "standardParentProject" : null,
    "standardProjectsFolder" : null,
    "defaultProjectConfigFile" : "project-config.json",
    "projectConfigurations" : [ {
      "projectName" : "load-datalake",
      "projectHome" : "/servers/hop/hop/resources/proyect",
      "configFilename" : "project-config.json"
    } ],
    "lifecycleEnvironments" : [ {
      "name" : "datalake-local",
      "purpose" : "Development",
      "projectName" : "load-datalake",
      "configurationFiles" : [ ]
    } ],
    "projectLifecycles" : [ ]
  }

root@coetl:/servers/hop/hop# sh hop-run.sh -j redmine-load-datalake -f/servers/hop/hop/resources/proyect/load-datalake/repository/main.hwf -r local
2024/12/16 13:27:33 - HopRun - Enabling project 'load-datalake'
2024/12/16 13:27:33 - HopRun - Starting workflow: /servers/hop/hop/resources/load-datalake/repository/main.hwf
2024/12/16 13:27:33 - mainRedmineLoadDatalake - Start of workflow execution
2024/12/16 13:27:33 - mainRedmineLoadDatalake - Starting action [Set variables 2]
2024/12/16 13:27:33 - mainRedmineLoadDatalake - Starting action [Eliminar registros erroneos]
2024/12/16 13:27:34 - mainRedmineLoadDatalake - Starting action [Borrar ficheros temporales]
2024/12/16 13:27:34 - Borrar ficheros temporales - File [/servers/hop/data-integration/resources/redmine-load-datalake/repository/tmp/totales.csv] already deleted or did not exist.
2024/12/16 13:27:34 - Borrar ficheros temporales - File [/servers/hop/hop/resources/load-datalake/repository/tmp/urls.csv] already deleted or did not exist.
2024/12/16 13:27:34 - mainRedmineLoadDatalake - Starting action [Set Execution time]
2024/12/16 13:27:34 - Set Execution time - Using run configuration [local]
2024/12/16 13:27:34 - repository\setExecutionTime - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2024/12/16 13:27:34 - repository\setExecutionTime - Execution started for pipeline [repository\setExecutionTime]
2024/12/16 13:27:35 - Salida Tabla.0 - ERROR: Error initializing transform [Salida Tabla]
2024/12/16 13:27:35 - Salida Tabla.0 - ERROR: java.lang.NullPointerException: Cannot invoke "org.apache.hop.core.database.DatabaseMeta.getIDatabase()" because "this.data.databaseMeta" is null
2024/12/16 13:27:35 - Salida Tabla.0 - 	at org.apache.hop.pipeline.transforms.tableoutput.TableOutput.init(TableOutput.java:482)
2024/12/16 13:27:35 - Salida Tabla.0 - 	at org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66)
2024/12/16 13:27:35 - Salida Tabla.0 - 	at java.base/java.lang.Thread.run(Thread.java:833)
2024/12/16 13:27:35 - repository\setExecutionTime - ERROR: Transform [Salida Tabla.0] failed to initialize!
2024/12/16 13:27:35 - Set Execution time - ERROR: Unable to prepare for execution of the pipeline
2024/12/16 13:27:35 - Set Execution time - ERROR: org.apache.hop.core.exception.HopException: 
2024/12/16 13:27:35 - Set Execution time - We failed to initialize at least one transform.  Execution can not begin!
2024/12/16 13:27:35 - Set Execution time - 
2024/12/16 13:27:35 - Set Execution time - 
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.pipeline.Pipeline.prepareExecution(Pipeline.java:1087)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.pipeline.engines.local.LocalPipelineEngine.prepareExecution(LocalPipelineEngine.java:231)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.pipeline.Pipeline.execute(Pipeline.java:527)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.workflow.actions.pipeline.ActionPipeline.execute(ActionPipeline.java:542)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:745)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:888)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:888)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:888)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:888)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:447)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.workflow.Workflow.startExecution(Workflow.java:307)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.workflow.engines.local.LocalWorkflowEngine.startExecution(LocalWorkflowEngine.java:248)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.run.HopRun.runWorkflow(HopRun.java:438)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.run.HopRun.runWorkflow(HopRun.java:389)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.run.HopRun.run(HopRun.java:206)
2024/12/16 13:27:35 - Set Execution time - 	at org.apache.hop.run.HopRun.main(HopRun.java:930)
2024/12/16 13:27:35 - mainRedmineLoadDatalake - Finished action [Set Execution time] (result=[false])
2024/12/16 13:27:35 - mainRedmineLoadDatalake - Finished action [Borrar ficheros temporales] (result=[false])
2024/12/16 13:27:35 - mainRedmineLoadDatalake - Finished action [Eliminar registros erroneos] (result=[false])
2024/12/16 13:27:35 - mainRedmineLoadDatalake - Finished action [Set variables 2] (result=[false])
2024/12/16 13:27:35 - mainRedmineLoadDatalake - Workflow execution finished
2024/12/16 13:27:35 - mainRedmineLoadDatalake - Workflow duration : 1.512 seconds [  1.512" ]
HopRun exit.

i don't know if you need anything else ..

bamaer Dec 16, 2024
Collaborator

your hop-run command specifies a project (-j), but no environment.

ravamo Dec 16, 2024
Author

thats right , but I’m following this article to build a solid approach: [Running Apache Hop Pipelines and Workflows Using Hop-Run](https://www.know.bi/blog/our-blog-1/build-deploy-6-running-apache-hop-pipelines-and-workflows-using-hop-run-12).
I add -e load-datalake-local in a command and the same error ..

bamaer Dec 16, 2024
Collaborator

I add -e load-datalake-local in a command and the same error ..

You don't specify any configuration files in your environment ("configurationFiles" : [ ]).

The guide you refer to explains the basics of using hop-run, those example workflows/pipelines don't use any metadata items, so there's no need for an environment that contains variables.

ravamo Dec 17, 2024
Author

This is my ETL, and I store all the variables in the set (even though I know it's not the best option). I’ve hardcoded everything: I created a new project that contains the metadata folder, and inside rdbs is the JSON file with the database configuration, all hardcoded. I also created an environment with an empty file because I don’t need anything specific, but I included it to keep the workflow complete.

I uploaded this folder to the server via SCP and manually added the project by modifying the hop-config.json file to adjust the paths. Locally, everything worked perfectly with zero issues, but on the server, I’m encountering this error.

command : ./hop-run.sh -j TestBaseDeDatos -f /home/hop/TestBaseDeDatos/mainRedmineLoadDatalake.hwf -r local -e server-mode

ERROR: Error initializing transform [Salida Tabla]
2024/12/17 10:39:47 - Salida Tabla.0 - ERROR: java.lang.NullPointerException: Cannot invoke "org.apache.hop.core.database.DatabaseMeta.getIDatabase()" because "this.data.databaseMeta" is null
2024/12/17 10:39:47 - Salida Tabla.0 - 	at org.apache.hop.pipeline.transforms.tableoutput.TableOutput.init(TableOutput.java:482)
2024/12/17 10:39:47 - Salida Tabla.0 - 	at org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66)
2024/12/17 10:39:47 - Salida Tabla.0 - 	at java.base/java.lang.Thread.run(Thread.java:833)

Does this issue sound familiar to anyone? I can’t seem to pinpoint the problem overall.

bamaer · 2024-12-17T12:47:19Z

bamaer
Dec 17, 2024
Collaborator

your screenshot does not show the relevant parts of your problem, for example: what does the database connection in Salida Tabla look like?

1 reply

bamaer Dec 17, 2024
Collaborator

FYI: if you run ./hop-run.sh -j samples -f ${PROJECT_HOME}/transforms/tableinput-basic.hpl from your hop installation folder, you should have a correct run.
If you remove or rename config/projects/samples/metadata/rdbms/hop-samples.json in your hop installation folder, you'll have the exact same error as the one you showed earlier, so it definitely looks like there either is something wrong with your metadata, or your hop-run is not looking in the metadata folder you expect it to.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connection Issue with RDBMS Configuration in Server Mode #4700

{{title}}

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Connection Issue with RDBMS Configuration in Server Mode #4700

ravamo Dec 11, 2024

Replies: 2 comments · 7 replies

ravamo Dec 16, 2024 Author

ravamo Dec 16, 2024 Author

bamaer Dec 16, 2024 Collaborator

ravamo Dec 16, 2024 Author

bamaer Dec 16, 2024 Collaborator

ravamo Dec 17, 2024 Author

bamaer Dec 17, 2024 Collaborator

bamaer Dec 17, 2024 Collaborator

ravamo
Dec 11, 2024

Replies: 2 comments 7 replies

ravamo
Dec 16, 2024
Author

ravamo Dec 16, 2024
Author

bamaer Dec 16, 2024
Collaborator

ravamo Dec 16, 2024
Author

bamaer Dec 16, 2024
Collaborator

ravamo Dec 17, 2024
Author

bamaer
Dec 17, 2024
Collaborator

bamaer Dec 17, 2024
Collaborator