Upgraded Cronicle to 0.9.7 and Live Job Event Log no longer is in real time

[aric89]

Summary

Updated Cronicle to the latest version. When jobs are run the Live Job Event Log window says Connecting to server: http://IP then says Connected Successfully then nothing else until the job finishes.

Steps to reproduce the problem

Upgrade Cronicle from v8. Something might not have upgraded properly

Your Setup

Running on 2 servers, one is a primary and one is a worker.

Operating system and version?

Primary is Oracle Linux 7.9
Worker is Fedora 35

Node.js version?

Primary is 10.23.3-1.0.1.el7
Work is 1:16.15.0-3.module_f35+14429+b5c8f482

Cronicle software version?

0.9.7

Are you using a multi-server setup, or just a single server?

Multi

Are you using the filesystem as back-end storage, or S3/Couchbase?

File system

Can you reproduce the crash consistently?

Every Job that runs, no crash log

Log Excerpts

[jhuckaby]

You gotta upgrade Node.js. Version 10 is very far out of date and unsupported. Once you upgrade Node.js, you will probably need to reinstall the Cronicle dependencies. Do this via:

sudo bash
cd /opt/cronicle
bin/control.sh stop
npm install
bin/control.sh stop

[aric89]

Awww upgrade to Node.js 16 LTS then?

Thanks!

[jhuckaby]

Yup!

[aric89]

I upgraded nodejs to version 16.15 and did the npm install. Same issue :(

[jhuckaby]

How very bizarre. Did you upgrade Cronicle on your worker server as well? They both have to be v0.97 for the new socket.io client to work (I was forced to upgrade that dependency due to a vulnerability discovered in it).

If you run a job on your primary server, can you watch the live log then?

[aric89]

both are running v0.9.7. They both have this issue when a job is ran on either of them. The only thing I can think of is when I did the upgrade the normal npm registry was blocked on our firewall so I had to change the registry url. Would the upgrade might not have updated the new socket.io files properly?

[aric89]

[email protected] /opt/cronicle
└── [email protected]

[email protected] /opt/cronicle
└── [email protected]

[email protected] /opt/cronicle
└─┬ [email protected]
└── [email protected]

[email protected] /opt/cronicle
└─┬ [email protected]
└── [email protected]

[jhuckaby]

Well, I guess I've been barking up the wrong tree. Your version of socket.io is correct. So that's not the problem.

The real kicker here for me is that you said that even if you run a job on your primary server (not the worker server), the live log ALSO doesn't work. Did I hear that correctly? If so, then this problem is truly bizarre -- because Cronicle opens a websocket connection to the primary server right on page load, and the app won't even work without it. So if the UI is working, and you can see the schedule and launch a job, then the primary websocket is working. But then the live log view somehow DOESN'T work, even if the job is on the primary??? That is just mind-boggling to me. I literally cannot explain it.

Are you sure nothing else has changed in your environment recently? Different job, different script, different settings, different configuration, some kind of architecture or security change perhaps? Are you sure you are flushing your output buffers when logging?

You could try downgrading back to v0.8.x I guess 🤷🏻‍♂️

sudo bash
cd /opt/cronicle
bin/control.sh upgrade 0.8.61

I'm sorry I don't have a better answer. This issue really has me confused.

[aric89]

We've done standard Linux upgrades. I put an Apache proxy in front of it but I've verified that it doesn't work with and without the proxy.

We haven't done anything with output buffers so I guess no we aren't flushing the buffers.

[jhuckaby]

That is very strange. Also weird that it "connects" properly, but then there is no output. I assume once the job completes then the log appears?

I can't think of anything else to try. This is very odd.

[aric89]

Correct, once it finishes then it spits out the log info like it can't read the log output in real time until its done.

[aric89]

I downgraded to 0.8.61 and it worked immediately.

[jhuckaby]

Very concerning, thank you. I guess v0.9 isn't fully baked yet 😞

[aric89]

I tried to downgrade to 0.9.0 first but i got the error in the screenshot. I went to the next version up and no dice, downgrade to 0.8.61 and it worked.

[jhuckaby]

Ok thanks. I cannot explain, but when I have some time, I'll dig into this. For now, stay on v0.8.