Add arbitary server metadata field to servers

[ann0see]

Lately there have been some discussions about privacy related topics and issues for bot opt-in/opt-out features.

I believe that we should find a standard for bots similar to e.g. robots.txt or metadata in webpages for (non) human readable data the server admin can set. Ideally the content would be arbitrary to be extensible.

I mainly see two ways to achieve this:

1. Provide support for metadata - or an url to e.g. a JSON metadata document in Jamulus itself as a new protocol message or just JSON itself. This could be an extension to the server info (cleanest solution, however not backwards compatible)
2. Provide a standard for HTML comments in the welcome message (needs server admins to set a welcome message - will have fragmentation issues, needs parsing of the whole welcome message, but should be fairly easy to implement)

I think the Jamulus guidelines would tell us to go with 2. Whereas I think that we should at least provide an user-friendly JS page + documentation for the standard to let users "click together" their options.

As initial flags I'd suggest:

requestNoBots: [will request that any kind of bot doesn't connect], can be a wildcard or some specific name
isPrivate: [states that this server should not be used publicly. Maily an information]
adminContact: [gives contact info like E-Mail of server admin]
website: [gives link to website of server]

[mcfnord]

I think adminContact may be the most valuable metadata for solving mysteries that have arisen now and then.

[pljones]

With something like half of all servers not being maintained, by the look of the version number displayed in Jamulus Explorer, you'd have to allow for the fact this information will all be opt-in by the maintainer - so it's probably only going to be those interested in being trawled who supply any info at all.

[ann0see]

True. It's not even enforceable.

[mcfnord]

I've wanted to respond to this for a long time. It is enforceable. If you want an admin contact, for example, just don't list a server that omits it. The thing I hadn't figured out is: If you want operators to publish information like this for users, why wouldn't you First advocate for it in coverage about welcome messages? If there's no infrastructure for using some bit of metadata, and if the goal is to empower users, then guidance about admin email or website address goes as cleartext into the welcome message. And if you don't enforce its presence in a strict, enforcing directory system, then really, it's a bad idea to invite operators to specify metadata in a less transparent way than cleartext html. But the case for a bot metadata standard is opposite: It's helpful from day 1 and isn't hiding useful information from people who need it. (I guess someone might care what the bot policy is there, but in that instance, a thoughtful operator should mention it in the cleartext html as well is the metadata.) I might be out on a limb here, but if you don't enforce metadata compliance, than the case for metadata other than bot policy is really weak to me. Furthermore, I don't think you gain much by enforcing metadata inclusion to be publicly listed. Metadata needs to be actionable information that does not benefit users to know. For information users should see, we will advocate for cleartext HTML in a welcome message or file, perhaps even to the point of providing a template /etc/jamulus/welc.html where a user might modify default contact information and change a default bot policy. (Previously, I heard some people don't want a welcome message, and a default welcome message should probably render as nothing to assure that user experience is unchanged.)

[ann0see]

Ok. Valid points.

Initially the idea was that arbitary computer readable metadata (including server admin contact as example). They are readable by external tools.

You think bottom up (Specific metadata like Server admin, bot policies and how they should be handled) while I think top down (a general standard for metadata of any type but with these examples).

Probably we need to divide concepts.

So:

1. Encourage Server Admins to add Human readable contact information to their welcome message
2. Provide a general standard for metadata. This can also contain contact info but is mainly used for machines.

[ann0see]

Ok. So how do we agree on a Format? Suggestions? HTML Meta tags? HTML comments? A separate protocol message?

[pljones]

I wish for a world without friction.
Google maintains this -- but it's fiction.
I want to cast music from my phone to my hub.
But it's stored as FLAC, locally, there's the rub. (Ugh...)

I've trawled for years without finding a solution - can't do it from Windows, can't do it from the Linux server where the files live, can't do it from the hub itself, it won't see the file server.

Setting up a server isn't easy, whether it's a cloud VM or real, local hardware and regardless of OS. It's hard regardless of what you're planning on running on it. As technology is continually evolving, it's a lifetime's work just to keep up with maintaining the platform. It's simply not necessary for the vast majority of people. Most people who run public Jamulus Servers have found it too hard even to update Jamulus, despite their being straightforward guidance on how to do it non-destructively on that.

It is not easy to make the above easy, let alone foolproof. Foolproof means incredibly tedious, checking each step in any process has been completed correctly by testing it and making sure there's a way backwards to an assured previously working position from a failed step forward. That's harder than most people want to bother with. So they break things instead and then complain. Or get bored and complain things don't work the way they'd like.

Then there are people who find the challenge is actually interesting. And like sharing that by contributing content explaining how to achieve a result or performing part of the process in a safe way.

[ann0see]

Thanks.

Setting up a server isn't easy, whether it's a cloud VM or real, local hardware and regardless of OS. It's hard regardless of what you're planning on running on it.

Also true. Still, it's our burden to make it easier. That was one of my initial goals when getting into contributing to Jamulus.

Concerning

The following minimum setup is required to [...]

True. The example works - and is/was intended for but the generic case. The headless .deb install shouldn't need to care about that as the systemd unit file should be installed by default.

Yes, one could consider this as outdated information but also as needed.

I'm still confused by your answer @mcfnord . Yes, getting User Experience right - including linux documentation is rough.

[mcfnord]

Is it hard for the .deb install to create a default /etc/jamulus/welc.html file, and include a -w to it in the default systemctl configuration?

[ann0see]

No. But it might be confusing as not everyone wants a welcome message. What is exactly unclear in the documentation? Should we extend the Point 5 here: https://jamulus.io/wiki/Running-a-Server#installation ?

[mcfnord]

The welc.html can be empty. Would that change any behavior vs. no welcome message at all?

Extending Point 5 was kind of my first approach. At minimum, -w needs specific guidance. If I learn to squash, we'll give hints about how to handle this trouble. But we still won't be explicit about a solution.

I guess we want to avoid ponderous "wording to all those (and any future) command docs," so instead we "assume a certain level of knowledge," because we wish to avoid "another case of deciding whether *nix users can use *nix." Once you embrace the fact that, "You don't need to run on a Debian-based Linux OS system to run Jamulus as a server. Any documentation needs to cover all cases or be generic enough to cover all cases," the resulting pull request is just vague puzzles (file accessible? user account? default? Literal path? I just wanna say welcome!).

So I'd love to leapfrog over that technical and ideological logjam and just fix it. And now you're saying it's easy to just fix it. So let's provide an /etc/jamulus/welc.html that's just empty, and fix it. Our genius Linux operator will see where they can find this file. The path is right there in their config. And maybe someday they'll edit it, using all their whiz-bang *nix skills. Or they might just change the -w to a string rather than a path. God speed, Linux operator. And with that infrastructure, it won't turn my stomach to ask operators to install metadata in the welcome file. In fact, someday we can ship a default welcome file with default metadata in it! So cool!

[mcfnord]

I like to think in simplest terms. I'd like to honor these keywords:

NO_BOTS
NO_SESSION_RECORDING

I'd honor it no matter where it appeared (even in cleartext), using any case.

x--tagname is more than I'd prefer. If there really is one use case here, introducing a generalized platform seems premature. That's why I'd also like to brainstorm other slam dunk metadata ideas beyond bot policy.

[ann0see]

I understand we are inlining with HTML and using hiding tricks. I'd rather write code that ignores anything about that and matches against well-known unique-enough and descriptive-enough single keywords we introduce.

To do it correctly, we would need to add a field to the serverinfo with metadata. Then it's independent of anything.

[ann0see]

Cleartext looks strange - in my opinion. Especially, if users don't care or aren't the target.

[mcfnord]

I don't mandate cleartext. I just don't care if the keyword is hidden or visible. Mandating hidden text would reduce operator success at this task. I'm afraid I don't quite believe there's a case for an elegant, flexible, machine-readable syntax.