HTML character references &foo; are cut at the semi-colon

[jayvdb]

A URL containing an XML entity/HTML character reference, such as http://.../..?foo&bar;baz, will be cut at the semi-colon.

[jayvdb]

Another is ' in https://docs.red-dove.com/cfg/python.html results in a 404 at https://freeotp.github.io/&#39 (the ; is omitted, but it is the & which causes the 404 as it doesnt follow a ?)

[jayvdb]

It might useful to have the caller inform the parser what type of text is being provided, such as html, xml, md, rst, which give clues to the parser when it is trying to find the start and end of urls, and what decoding to perform on the url.

Or have a hook in the class which is given the location of the hostname, so the hook can decide the start and end of the url which surrounds the hostname. Then I could override the URLExtract class several times to implement this hook for various doctypes. _complete_url almost does this, but it would need to be a public member of the API.