Investigate Production ZAP Search outtages

[TylerMatteo]

Starting approximately Nov 26th, we've been getting intermittent alerts from Updown that ZAP Search is down. The validity of these reports was backed up on 12/3 by Sarah Whitham sharing that Sara Avila was getting a 502 Bad Gateway error when trying to visit ZAP Search.

• Cross reference outage times with logs in Heroku to identify root cause
• Document research and share suggested next steps.

[horatiorosa]

@TangoYankee @TylerMatteo
I thought I was onto something with the alert noticing the description Client Request Interrupted below but it seems to be a red herring.

I searched the logs for more Client Request Interrupted incidents and they don't match up with any of the alerts in Uptown.

[horatiorosa]

Issue doesn't seem to related to timeouts either. The times don't match up.

[TangoYankee]

After Tyler increased the frequency of pings for Zola and Applicant Portal, we saw an Updown alert for them as well

[TangoYankee]

I checked the error logs in the reverse proxy. It seems there are several requests a minute that are unable to connect to IPv6 addresses at port 443. Multiple applications have their own connection failures- zola, applicant portal, community profiles, etc. Maybe there is a misconfiguration in Netlify? Either in the IPv6 addresses or their SSL certificates