Is it possible to run fio in the background and ask for status

[tjodork]

We have an exerciser we use, where we start up a daemon or server.
We the issue a command to the server to start up a specific workload from a script and the call to start the workload returns right away,
We can query status to the server, both general run status and then also more detailed status on reads/writes,etc
We can then issue a shutdown to stop the workload

This is helpful since I can run a script that will issue the command to start the exer and then loop waiting a specific time, while also issuing downloads and other commands to the drives, checking the system for errors and requesting status from the running workload and when the desired time is hit, then end the workload.

I'm trying to accomplish the same thing with fio. If I use the client-server option, the client is hung while the workload it going.

Is what I'm asking currently possible or any suggestions?

Thanks
Tim

[vincentkfu]

You can run fio in the background by placing an ampersand at the end of the command. You will see the process' PID.

if you want to query the status of the job send the USR1 singnal to the fio process with kill. See the example below:

vincent@localhost:~$ fio --name=test --filesize=1T --ioengine=null --time_based --runtime=1h --output=test &> test.output &
[1] 324056
vincent@localhost:~$ cat test
vincent@localhost:~$ kill -USR1 324056
vincent@localhost:~$ cat test
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.35
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=324086: Fri Jan 19 09:27:58 2024
  read: IOPS=9106k, BW=34.7GiB/s (37.3GB/s)(1051GiB/30267msec)
    clat (nsec): min=10, max=35527, avg=19.68, stdev=14.35
     lat (nsec): min=30, max=35547, avg=38.81, stdev=19.94
    clat percentiles (nsec):
     |  1.00th=[   10],  5.00th=[   10], 10.00th=[   20], 20.00th=[   20],
     | 30.00th=[   20], 40.00th=[   20], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   20], 80.00th=[   20], 90.00th=[   20], 95.00th=[   20],
     | 99.00th=[   21], 99.50th=[   30], 99.90th=[   30], 99.95th=[   40],
     | 99.99th=[   41]
   bw (  MiB/s): min=34085, max=36576, per=99.99%, avg=35567.24, stdev=864.81, samples=60
   iops        : min=8725846, max=9363546, avg=9105213.27, stdev=221392.11, samples=60
  lat (nsec)   : 20=5.11%, 50=94.88%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=99.73%, sys=0.01%, ctx=551, majf=0, minf=6
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=275614775,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=34.7GiB/s (37.3GB/s), 34.7GiB/s-34.7GiB/s (37.3GB/s-37.3GB/s), io=1051GiB (1129GB), run=30267-30267msec

[tjodork]

Hi thanks for that information.
I have a perl script that starts fio in the background and returns the PID
I'm having trouble getting FIO to end with the kill -USR1 pid
Is there any way to debug this ? i.e. how would I know if fio saw the kill command
Any other suggestions

[tjodork]

note the jobfile has a lot of drives
cat /tmp/fio.sdc_plus178.jobfile
[global]
bs=4k
direct=1
buffered=0
iodepth=1
numjobs=1
group_reporting=1
rw=randwrite
percentage_random=30
rwmixread=0
rwmixwrite=0
ioengine=libaio
runtime=60
time_based=1
norandommap
randrepeat=0
ramp_time=0
random_generator=tausworthe64
nice=19
prio=0
prioclass=1
unified_rw_reporting=1
log_hist_msec=100000
percentile_list=1:25:50:75:90:99:99.5:99.9:99.99:99.999:99.9999:99.99999
clat_percentiles=1
lat_percentiles=0
slat_percentiles=0
size=24%
[job1]
filename=/dev/sdc
new_group
[job2]
filename=/dev/sda
new_group
[job3]
filename=/dev/sdb
new_group
[job4]
filename=/dev/sde
new_group
[job5]
filename=/dev/sdd
new_group
[job6]
... . . . . . . .

and the process i sent the USR1 to was 11104

ps -ef | grep fio
root 7626 2 0 14:33 ? 00:00:00 [vfio-irqfd-clea]
root 11104 1 98 14:37 pts/0 00:03:10 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 11223 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 11255 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12010 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12441 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12442 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12443 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12444 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12445 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12446 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12447 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12448 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12449 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12450 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12451 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12453 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12454 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12455 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12456 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12457 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12458 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12459 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12460 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12461 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12462 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12463 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12464 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12465 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12466 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 12467 11104 0 14:37 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults

[vincentkfu]

Every time you run kill -USR1 $pid fio will dump full output to the file specified with --output=. If you want fio to stop running just use kill $pid.

[tjodork]

thanks for your expertise and quick reply
I misspoke .... I WAS trying to get the dump of data to the file
for some reason when I have multiple drives in the jobfile (and note I do specify new_group) and I do the kill -USR1 pid, no data is dumped...see below

root 27165 25800 0 15:27 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27166 25800 0 15:27 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27167 25800 0 15:27 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27168 25800 0 15:27 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27169 25800 0 15:27 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27171 25800 0 15:27 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27172 25800 0 15:27 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27174 25800 0 15:27 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27175 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27176 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27177 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27178 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27179 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27180 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27181 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27182 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27188 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27189 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27190 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27191 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27193 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27194 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27195 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27196 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27197 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27198 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27199 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27200 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27201 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27202 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27203 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27204 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27205 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27206 25800 0 15:28 ? 00:00:00 fio /tmp/fio.sdc_plus178.jobfile --output-format=json --output=/tmp/bgresults
root 27208 12775 0 15:28 pts/1 00:00:00 grep --color=auto fio
root@ess6000tos [/] kill -USR1 25800
root@ess6000tos [/] cat /tmp/bgresults
root@ess6000tos [/] cat /tmp/bgresults
root@ess6000tos [/] cat /tmp/bgresults
root@ess6000tos [/] kill -USR1 25800
root@ess6000tos [/] cat /tmp/bgresults
root@ess6000tos [/]

[vincentkfu]

Having multiple jobs shouldn't affect this feature.

I ran this in one window:

vincent@fio-dev:~/fio-dev/fio-canonical$ ./fio --name=test --ioengine=null --time_based --runtime=600s --filesize=1T --numjobs=1024 --group_reporting

Then in another window I ran:

vincent@fio-dev:~$ ps -ef | grep fio
avahi        634       1  0 03:02 ?        00:00:00 avahi-daemon: running [fio-dev.local]
vincent     1727    1377 24 17:17 pts/0    00:00:02 ./fio --name=test --ioengine=null --time_based --runtime=600s --filesize=1T --numjobs=1024 --group_reporting
vincent     1729    1727  0 17:17 ?        00:00:00 ./fio --name=test --ioengine=null --time_based --runtime=600s --filesize=1T --numjobs=1024 --group_reporting
vincent     1730    1727  0 17:17 ?        00:00:00 ./fio --name=test --ioengine=null --time_based --runtime=600s --filesize=1T --numjobs=1024 --group_reporting
vincent     1731    1727  0 17:17 ?        00:00:00 ./fio --name=test --ioengine=null --time_based --runtime=600s --filesize=1T --numjobs=1024 --group_reporting
vincent     1732    1727  0 17:17 ?        00:00:00 ./fio --name=test --ioengine=null --time_based --runtime=600s --filesize=1T --numjobs=1024 --group_reporting
vincent     1733    1727  0 17:17 ?        00:00:00 ./fio --name=test --ioengine=null --time_based --runtime=600s --filesize=1T --numjobs=1024 --group_reporting
...
vincent     2750    1727  0 17:17 ?        00:00:00 ./fio --name=test --ioengine=null --time_based --runtime=600s --filesize=1T --numjobs=1024 --group_reporting
vincent     2751    1727  0 17:17 ?        00:00:00 ./fio --name=test --ioengine=null --time_based --runtime=600s --filesize=1T --numjobs=1024 --group_reporting
vincent     2752    1727  0 17:17 ?        00:00:00 ./fio --name=test --ioengine=null --time_based --runtime=600s --filesize=1T --numjobs=1024 --group_reporting
vincent     2754    1698  0 17:17 pts/1    00:00:00 grep fio
vincent@fio-dev:~$ kill -USR1 1727

And then I saw a full output dump in the original window.
Based on the information you have shared so far I don't see anything that seems problematic. Just start with a very simple job for troubleshooting. Then slowly build it up to its final form with the full set of options that you want.

[tjodork]

wow, thanks again for your help.
turns out with running 179 drives at the same time, when I request that output it took almost 3 minutes for the data to go to the file.
So it DOES work,...just a bit slow : )