Autocue: How can I send JSON metadata to stdin of a process?

[Moonbase59]

For AzuraCast’s Autocue (my autocue.cue_file integration), to avoid unneeded multiple runs on remote or read-only file systems, I need to pass JSON metadata to the external process cue_file, into its stdin.

How can I do that?

I have (Liquidsoap) metadata in meta, a list of key-value tuples, and this code:

    res = ref("")
    try
      res :=
        list.hd(
          default="",
          process.read.lines(
            timeout=timeout,
            process.quote.command(settings.autocue.cue_file.path(), args=args())
          )
        )
    catch err do
      log(
        level=2,
        label=label,
        'cue_file error: #{err}'
      )
      res := ""
    end

How can I expand that to send meta as well-formed JSON to cue_file’s stdin?

Help from the knowledgeable people here much appreciated—thanks in advance!

Background

cue_file (as a pre-processor) can take overriding metadata as JSON, which helps it avoiding unnecessary re-analysis runs. This is especially useful when certain file tags need to be overridden, or when we cannot write back tags, because a filesystem is remote or read-only. It will also help reducing bandwidth on connections like S3 or SFTP.

For AzuraCast, we plan to use its database as an intermediate storage for these values, which should boost performance a lot. Standalone Liquidsoap and pre-processing scripts can also greatly benefit from this feature.

[toots]

Does this help: https://www.liquidsoap.info/doc-2.2.4/json.html#exporting-json-values ?

[Moonbase59]

Sure, thanks. But my main problem is still how to pass it to cue_file’s stdin in the above code. Really don’t know how to do that.

I’m sure Liquidsoap must have some means to do that, apart from writing files and using stdin on process.quote.command. I found process.stdin which seems to support a socket write, but how to implement that in above?

[Moonbase59]

Spent a few hours with "The Book" and LS source code, but couldn’t make out anything helpful.

It can’t be that complicated to write a string to a process stdin, or can it?

[vitoyucepi]

I don't think you can just do that. If I were you, I would either use /tmp, /dev/shm or create a custom tmpfs mount to host the temporary json file.

As an alternative, you can try pipe in shell:

a = { a = "1" }
b = json.stringify(a)                    
c = process.quote.command("echo", args = [b])
print(process.read.lines("#{c} | jq"))

I think liquidsoap also supports

1. named pipes;
2. unix sockets;
3. tcp sockets;
4. http.

[Moonbase59]

Thanks! Feels a littly clumsy, though, having to echo all the metadata. I’ll definitely have to remove the cover images then, I guess. Hope we don’t run into any input buffer length problems with that construct…

Could we be sure /dev/shm was available and writable in all cases?

[vitoyucepi]

Could we be sure /dev/shm was available and writable in all cases?

You may never know this:

1. It's Linux-specific.
2. It depends on the kernel configuration.

Use file.temp if you're not sure.
By the way, images should not be in JSON without base64 or something similar.

[Moonbase59]

images should not be in JSON without base64 or something similar

Sure, I was just hesitant because of their size.

Thanks for all the pointers, much appreciated!

[toots]

I'm gonna see if I can add that soon it shouldn't be too hard.

…

On Mon, Jun 10, 2024, 21:56 Matthias C. Hormann ***@***.***> wrote: images should not be in JSON without base64 or something similar Sure, I was just hesitant because of their *size*. Thanks for all the pointers, much appreciated! — Reply to this email directly, view it on GitHub <#3958 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGUVFEINROD2HPVQFUZBLDZGYAGXAVCNFSM6AAAAABJBRHULKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOMZRGI3DQ> . You are receiving this because you commented.Message ID: ***@***.***>

[Moonbase59]

That would be fantastic. I mean, we can read a process stdout in different ways, just not write (easily). Thanks for considering!

In the meantime, I’ll go with the temp file approach.

[toots]

The reason is that most process terminate when you close their stdin so retaining control over it and closing it is the best way to ensure things will go as expected.

[toots]

That was fairly easy: #3962

So you think we'll need a backport to 2.2.x? I'd like to avoid pushing a new release on this branch, I'd rather focus on having 2.3.x out soon.

[Moonbase59]

Not necessarily, I’d say. We can live with the tempfiles for now. 2.3.0 could have it, though ;-)

[Moonbase59]

I’m using this now, for the time being. @vitoyucepi Any objections/errors you see?

    # write metadata to temp file for cue_file to pick up
    tempfile = file.temp("cue_file", ".json")
    json_meta = meta_json_stringify(compact=true, meta)
    log(level=4, label=label, "#{tempfile}: #{json_meta}")
    file.write(
      data=json_meta,
      append=true,
      tempfile
    )
    args := ['-j', tempfile, ...args()]

    res = ref("")
    try
      res :=
        list.hd(
          default="",
          process.read.lines(
            timeout=timeout,
            process.quote.command(settings.autocue.cue_file.path(), args=args())
          )
        )
    catch err do
      log(
        level=2,
        label=label,
        'cue_file error: #{err}'
      )
      res := ""
    end
    file.remove(tempfile)

[vitoyucepi]

Why do you need res = ref("")

res =
  try
    list.hd(
      default="",
      process.read.lines(
        timeout=timeout,
        process.quote.command(settings.autocue.cue_file.path(), args=args())
      )
    )
  catch err do
    log(
      level=2,
      label=label,
      'cue_file error: #{err}'
    )
    ""
  end

I would also like to understand what we are actually catching here.

command = process.quote.command(settings.autocue.cue_file.path(), args=args)

process = process.run(command)
  
output = 
  if process.status == "exit" and process.status.code == 0 then
    process.stdout
  else
    # log?
    null()
  end
lines = null.map(fun(output) -> r/\r?\n/.split(output), output)
result = 
  try
    null.map(list.hd, lines)
  catch _ do
    # log?
    null()
  end

Or maybe I'm just overcomplicating things...

[Moonbase59]

@vitoyucepi I’m somehow a ref() fan, as toots already noticed. 😄 Probably due to me always messing up with LS variable scoping (more used to Python). But truly, your first block is much more elegant and I’ll change that!

LS is not my cup of tea… Have to check what null.map() does. Your second suggestion looks much cleaner (for diagnosis), but tbh, I only want to see if we got a result or not and don’t care much about anything but the one line of compact JSON I hope to get from process.stdout. That’s why I simply put them all into one try. Should I still dissect it like you did?

There’s certainly a lot in autocue.cue_file.liq that can be coded much cleanlier. I’m just trying to realize my ideas with the very little LS knowledge I have… Eventually, when all is running well, I should probably ask one of you for a code review/cleanup. ;-)

[vitoyucepi]

This is how I try to implement railways and show my passion for Result<T,E>.