UTF-8 issue with LiquidSoap

[SpeedyCharly]

From; AzuraCast/AzuraCast#4925

A while back I asked about why UTF-8 doesn't recognize standard French characters in the metadata as seen here;

---

2021/12/25 09:24:18 [camomile:3] Failed to convert "Fetes - Yvan- V\233ronique en ondes" from UTF-8 to UTF-8 (CamomileLibrary__CharEncoding.Configure(Config).Malformed_code)!
2021/12/25 09:24:18 [input_streamer:3] New metadata chunk ? -- Fetes - Yvan- V?ronique en ondes.

At the same time we poll the metadata from our Shoutcast stream [/7.html] to feed our chatroom bot which shows up properly:

Didn't really get a response the first time as it was embedded into another topic...

So, the question is; what does Shoutcast do that LS doesn't?

Could the problem reside with the Unicode library used?
[Camomile is a Unicode library for objective caml (a. k. a. OCaml or O'Caml) language. Camomile provides Unicode character type, UTF-8, UTF-16, UTF-32 strings, conversion to/from about 200 encodings, collation and locale-sensitive case mappings, and more. The library is currently designed for Unicode Standard 3.2.]

---

From; AzuraCast/AzuraCast#4937

Reported before on difficulties with UTF-8 and LS supporting French custom characters. At the time, I also noted that the very same metadata was transcribed correctly in ShoutCast. Yes, we're on Shout, not Ice; this to accommodate our djs.

Thus, I thought that switching autoDJ from UTF-8 to ISO-8859-1 (going full Legacy) would solve the problem, as AzuraCast documentation seems to indicate.

But, lo and behold, nothing happens quite as expected in the IT world! After doing the restart to apply the change, I immediately noticed something strange; the 'Profile' page modules were no longer reporting properly;

Public page display was also frozen but songs were still playing through normally;

Song queue was also weirding out;

But LS and the server were still running correctly as evidenced by ShoutCast page, player audio and our chatroom ;

Neither rebooting the server nor restarting the broadcast helped with this. The only way out of this mess was to reset back to UTF-8, restart AzuraCast, then clear the song queue. Normal operation resumed fine after that.

I'm assuming this bug was never reported on before because we're probably one of the rare stations working both in French and ShoutCast...

Can someone give me the lowdown on what is happening here?

[toots]

The last time I looked at this, liquidsoap was receiving already corrupted strings with the UTF8 replacement character, meaning that all information about this character was already lost and that something was off before the string reached liquidsoap.

I'd suggest to take a minute to analyze every step of your setup to better understand what strings are being used and in what encoding. Remember that encodings are not declared so each part of your system will assume a given encoding (latin1, ASCII, utf8) and process your strings as such. This means that if, for instance, a utf8 string hits a part of the system that assumes latin1 or ASCII, it will get corrupted.

• What are the encodings coming out of the file's metadata (if used)?
• What are the encoding coming out of the system's process call? What's the locale used in your server(s)?
• Etc...

Please also note that shoutcast and its ICY metadata protocol expects latin1, at least I checked. Typically, if you send UTF8 metadata to shoutcast without converting them to latin1 and then re-use the resulting string somewhere, your string will be corrupted.

[toots]

I cannot comment on the reason switching the encoding on azuracast freezes the application though I suppose that it is could be because of some crash when processing/parsing the input.

Character encoding can be either mass-converted ahead of time or else you can try to auto-detect. I understand that you have a lot of data but, to paraphrase someone, "with great amount of data comes great amount of conversion". 🕷

String encoding is unrelated to how you produce letters on the keyboard. It is an internal setting of computer systems. I can only, again, stress the need to document yourself a little bit more about the issue so that you are in a better position to understand and, potentially, fix it.

Again, I would suggest tracing the origin of the metadata that come into liquidsoap which, as I stated before, seem to be corrupted before. It could be because AzuraCast assumes UTF8 and you are feeding it non UTF8 data.

[toots]

Another thing: last I checked and, as stated above, it is likely the shoutcast server uses latin1 encoding for its strings, in which case, assuming this is true and your bot also expects latin1 characters, that would explain why it may be successful with shoutcast and not azuracast or liquidsoap which operate in UTF8.

[SpeedyCharly]

Called on the maintainer of our chatroom app and he assures me that chat and bot both run on UTF-8 protocol...

[SpeedyCharly]

FYI
ocaml/ocaml#5348

[SpeedyCharly]

Don't know if you guys did anything on this issue lately, but I haven't seen UTF-8 or Chamomile errors for 2-3 days now! Hosannas to the powers that be....