fix missing error page and improper redirects

[lordlabuckdas]

As encountered in #230. some websites do not have a 404 template configured or redirect to the home page. To fix, this

1. An empty page is created
2. Or if the website redirects, it is caught by comparing the requested and returned URL (ref feature: redirects #302).

Other fixes:

• The get_headers(response_headers: CIMultiDict) -> headers: list method now returns the content type as well. This was done to optimize the working of Cloner and prevent a call to get_content_type which iterates through the list of headers to fetch the content type. From
  get_headers(response_headers: CIMultiDict) -> headers: list to
  get_headers(response_headers:CIMultiDict) -> list[headers: list, content_type: str]

• Moved the Content-Type header from headers key in meta info to a separate key
  From

  "/about": {
      "hash": "abcd",
      "headers": [{"Server": "Apache"}, {"Content-Type": "text/html"}]
  }

  to

  "/about": {
      "hash": "abcd",
      "headers": [{"Server": "Apache"}],
      "content_type": "text/html"
  }

  This was done because tanner_handler.py#L111 and snare_helpers.py#L96 already check for it and overwrite the Content-Type header.

[afeena]

I just spot we have initialization error in close function 🤔

[lordlabuckdas]

Strangely enough, that check is there just to be thorough and satisfy linter warnings. If you are referring to an instance where the error was actually raised, could you please share the steps to replicate it? I've tried it with multiple websites and it seems to work fine.

[afeena]

Do we really need an empty page? Why not letting aiohttp handle this?
https://docs.aiohttp.org/en/latest/web_exceptions.html

[mzfr]

Yeah I agree instead of generating a new empty page we should just let aiohttp handle that

[lordlabuckdas]

Special if-check for status code 404 because returning web.Response with status_code=404 did not work as intended.