[#7977] Update AttachmentController#show_as_html to do attachment masking in the background

[gbp]

Relevant issue(s)

Fixes #7977

What does this do?

This updates AttachmentController#show_as_html to do attachment masking in the background.

Why was this needed?

Our "view attachments in the browser" feature, or "view as HTML", is doing a lot of processing inline in the web processes and this sometimes causes memory issues on the server and results in oom-kill running.

Implementation notes

Uses the same logic as the #show action to check an attachment is masked before proceeding with the action. This means it will time outs after 5 seconds and then redirect the AttachmentMaskController to wait for the job to complete before rendering a view saying the attachment is now viewable.

[gbp]

Testing on staging and this works but I have changed the implementation slightly. Because we're not downloading files we don't need to render the done action at all, and can just redirect the to show as HTML action.

[gbp]

This is up on staging for testing, I have added a sleep 10 line to the job to simulate a slow job to show the redirect to the attachment mask controller.

In order to ensure attachments need to be masked, in the staging Rails console run: FoiAttachment.update_all(masked_at: nil)

[gbp]

The reason for this PR is that we saw Apache had been stopped on the WDTK production server. This was due to oom-kill running:
Oct 24 15:58:49 mstrwdtkhexa systemd[1]: apache2.service: Failed with result 'oom-kill'

Prior to this happening these processes were taking the most memory:

process, memory
1721111, 12%
1721824, 11%
1720565, 10%
1721559, 10%
1721971, 10%
1799539, 10%

Looking at each:

• 1721111 jumped from 896448 KB to 2926488 KB when processing an AttachmentController#show_as_html request at 15:57:23 but wasn't stuck processing when oom-kill ran
• 1721824 was at 787944 KB before starting processing an FoiAttachmentMaskJob job inline (not sidekiq) for an AttachmentController#show_as_html request at 15:53:39
• 1720565 jumped from 1199208 KB to 2639880 KB when processing an AttachmentController#show_as_html request at 15:57:14 but wasn't stuck processing when oom-kill ran
• 1721559 was at 856708 KB before starting processing an FoiAttachmentMaskJob job inline (not sidekiq) for an AttachmentController#show_as_html request at 15:53:34
• 1721971 was at 706764 KB before starting processing an FoiAttachmentMaskJob job inline (not sidekiq) for an AttachmentController#show_as_html request at 15:56:07
• 1799539 was at 316636 KB before starting processing an FoiAttachmentMaskJob job inline (not sidekiq) for an AttachmentController#show_as_html request at 15:53:37

[garethrees]

Reviewed on a call together. Looks great!