Batch run of a time-consuming script in the background

[jkozlowicz]

I am trying to batch-run a time-consuming script and I am running into timeout problems.

I am either getting request timeouts from the HTTP library I am using to make the requests, or I am getting a timeout error from CrowdStrike saying Exceeded maximum read timeout: 29.36s.

I checked on the target hosts and the script is properly started and runs till the end, which is the desired behaviour, but how can I make the runscript command return sooner and then check the progress of the running script, rather than run the command and hang till it timeouts?

The script that I first create and then run is this:
my_executable

#!/bin/bash

./<my-executable> --<my-arguments>

and I start it as follows:
runscript -CloudFile=\"my_executable\" -CommandLine=\"-arg1 val1 ...\"

The script and the arguments are working fine. I tried to run the bash script in the background by adding & either to the script content or at the end of -CommandLine=\"-arg1 val1 ... & \", but then I wasn't able to start the script. (either something with the syntax didn't work or perhaps CrowdStrike doesn't allow to run processes in the background?)

How can I start a script on a remote host, keep it running, and just receive the information if it was properly started or not, without having to wait until it finishes?

[jshcodes]

Hi @jkozlowicz -

We could add -Timeout=120 to your runscript call and extend this timeout. (This may not solve the issue depending.)

We could also upload a script, then call it with runscript. That script could then execute, "do work" (like executing my_executable) and then report back in a manner that can be reviewed with a follow up check.

The cloud_request_id can be used later to check status, but depending on the implementation it might be easier to check status using another regular runscript call.

[jkozlowicz]

Timeout=120 will not help. The run time will vary depending the work to be done on a given host. It may take minutes or even an hour,.

I do upload a script and then run it with runscript (this is exactly what I wrote in my initial write up). It does exactly what you mentioned, it execute an executable:

my-script

#!/bin/bash

./executable.sh --output out.csv

But the problem is that it doesn't return right after starting the script, but instead wait till it finishes, which ends in a timeout.

[jshcodes]

You don't return even when the executable sends the spawned process to the background (using &)?

[jkozlowicz]

That is correct. I just double checked that adding a & at the end, doesn't return immediately after spawning a process, but waits till it returns something and eventually times out.

This is the error message coming as stderr from the runscript batch command (I get back these attributes per each host affected by the batch command):

host=XYZ
stderr=The command was terminated as it has not produced any output for a long time,
stdout='',
complete=True,
errors=[]

So I have to treat timeouts like those as successful attempts to execute a script. What I'd prefer instead, would be to immediately receive a response containing something only in the stdout attribute, e.g. "started" or "PID = X".

I can workaround this (by treating timeouts as potential successes and then checking the script progress with ps or checking for an output that the script is supposed to create), but it is less ideal. ;)

Similar story on Windows. Even Microsoft's documentation states that Start-Process cannot be run in a remote session with the expectation of the spawned process to outlive the session:
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/start-process?view=powershell-7.4#notes

By default, Start-Process launches a process asynchronously. Control is instantly returned to PowerShell even if the new process is still running.

On the local system, the launched process lives on independent from the calling process.
On a remote system, the new process is terminated when the remote session ends, immediately following the Start-Process command. Therefore, you can't use Start-Process in a remote session expecting the launched process to outlive the session.

If that you could confirm that this is indeed a constraint, then I'd just go ahead and proceed with the assumption that the execution of a remote script will not return and implement a workaround for that. Otherwise, I am happy to any advice on how I could improve that.

[jkozlowicz]

I also found this link: CrowdStrike/psfalcon#249 (comment) saying that there is a way to launch a Secondary Process whenever there as a need to run a script that might need more than 5 minutes to complete, which I'd hoped could help, but the link doesn't work

[jkozlowicz]

@jshcodes Looks like I was wrong...

I think I wanted to recreate a script, but forgot the give it a new name and thus I thought instructions to make a process run in background were added, but in fact an old script without these was being used. I also ran into problems managing remote processes as in some cases they were dying after closing the connection with the host, which only added to the confusion. I think I have solved all the issues related to that so we can close this discussion.

It seems that I am now capable of running process in background with:

• nohup ./<my-script.sh> & disown on linux machines
• Start-Process -WindowStyle hidden "<exectuable>" on windows machines