More aliases (for Cameroon, Sudan, Brazil, Congo)

[wu-lee]

Mainly inspired by an encounter with these in the wild:

• Brasil
• Cameroun
• The Sudan
• Democratic Republic of Congo

However, some extra permutations/variations added which seem valid, having consulted Wikipedia.

[sshaw]

We should keep Congo, The Democratic Republic Of The. It's valid: https://www.iso.org/obp/ui/#iso:code:3166:CD.

Version with parenthesis should be added too but this looks like a broader task for another day.

[wu-lee]

I think I meant to add the extra alias, and not delete the existing one. Will fix.

[sshaw]

Do you think we may as well add Republic of South Sudan and Republic of S. Sudan and similar entries for the others? In English Republic of South Sudan is how it would be formally written.

[wu-lee]

Sure. Although "S. Sudan" is a bit of a weird one, adding an extra permutation: abbreviations of "north", "south", etc. An "S." variation is included here and for a few other cases, but not for South Africa, South Georgia, North Macedonia etc.

[sshaw]

Cool thanks for this. Aside from inline comments: Brasil and République démocratique du Congo, etc... There are Portuguese and French respectively but this is en.yml. While I would definitely like to support other languages just having these sprinkled into the file for English normalizations would be a surprise. I can see the argument that one should have the country's name in its official language(s) but then what does that mean for the rest of the countries? Maybe that part is a work in progress?

[sshaw]

While I would definitely like to support other languages just having these sprinkled into the file for English normalizations would be a surprise. I can see the argument that one should have the country's name in its official language(s) but then what does that mean for the rest of the countries? Maybe that part is a work in progress?

There's was also #5

[wu-lee]

Brasil and République démocratique du Congo, etc... There are Portuguese and French respectively but this is en.yml. While I would definitely like to support other languages just having these sprinkled into the file for English normalizations would be a surprise.

I notice that "Côte D'Ivoire" is not English, and in fact many of the names contain other languages? For example, "São Tomé and Príncipe", "Sint Maarten", "Timor-Leste". There are surprises already!

I can see the argument that one should have the country's name in its official language(s) but then what does that mean for the rest of the countries? Maybe that part is a work in progress?

I'd probably argue that for a normalisation tool to be most effective, it needs to recognise all sorts of weirdness? So the "input" labels can't be categorised as any particular language, and in fact could be a mixture, like "Republic of Côte D'Ivoire", "Sint Maarten (Dutch Part)" etc. Outputs are another matter. The aliases seem mostly to be there for recognition, and the canonical names and identifiers are the outputs.

[edit] In my use case, I've been using this gem for normalising into an ISO code, and Carmen for generating names in various languages.

[sshaw]

I notice that "Côte D'Ivoire" is not English, and in fact many of the names contain other languages? > For example, "São Tomé and Príncipe", "Sint Maarten", "Timor-Leste". There are surprises already!

The difference seems to be as discussed in #8: names are often written as they are in their native language.

"Leste" may be Portuguese but "The Democratic Republic of" is not. "République démocratique du Congo" is all in French. "Republic of Côte D'Ivoire" only "Côte D'Ivoire" is French. This is the distinction. Maybe there are exceptions. "Curaçao" maybe since this is Portuguese (amongst others) but in Papiamento it's "Kòrsou".

I'd probably argue that for a normalisation tool to be most effective, it needs to recognise all sorts of weirdness?

Given this code/conversation I think there are some ways to normalize (in general not in the PR): St. => Saint, S. => South, etc...

like "Republic of Côte D'Ivoire", "Sint Maarten (Dutch Part)"

To me this is English with acceptable variants (aliases) of "Cote D'Ivoire" etc...

[wu-lee]

I've re-rolled this PR in the light of the above, see what you think now.

[sshaw]

Great thanks. This also gave me some things to think about and some ideas for code-level normalization.

[sshaw]

French! Is this intentional? 😅

[wu-lee]

Mouais?

[wu-lee]

Thanks, feel free to tweak things for consistency, although saying that I do need that damn Cameroun, which might be French...

Possibly helpful for you or others: I discover that the I18n gem's #transliterate can remove accents from text, and Rails' #parameterize can further strip out punctuation. The latter is small enough to steal wholesale, the former not so much. But processing text through those gets rid of a lot of the variation.

[sshaw]

Thanks, feel free to tweak things for consistency, although saying that I do need that damn Cameroun, which might be French...

You could always just use your own YAML file. This was my original ideal I just never coded it in but doing it would be trivial. Maybe less than trivial if we just go with an env var at first.

Possibly helpful for you or others: I discover that the I18n gem's #transliterate can remove accents from text...

Yes I see it's fairly compact will consider this thanks.