x509: certificate signed by unknown authority"

[vikmeup]

I'm currently receiving this error when running an instance of gorush on Heroku.

{
    "counts": 1,
    "logs": [
        {
            "type": "failed-push",
            "platform": "ios",
            "token": "7027a0ec534ac263e1de4b0fdad50e7dba0b894c96ab51173958640e6ade049b",
            "message": "Hello World iOS!",
            "error": "Post https://api.push.apple.com/3/device/....: x509: certificate signed by unknown authority"
        }
    ],
    "success": "ok"
}

Attaching information from Heroku support team:

The certificate signed by unknown authority error means that the Certificate Authority being presented by the server is not recognised by the client/the OS root certificates store it's using.

Checking api.push.apple.com I see it's using the GeoTrust Global CA CA:

$ echo 'Q' | openssl s_client -connect api.push.apple.com:443
...
Certificate chain
0 s:CN = api.push.apple.com, OU = management:idms.group.533599, O = Apple Inc., ST = California, C = US
i:CN = Apple IST CA 2 - G1, OU = Certification Authority, O = Apple Inc., C = US
1 s:CN = Apple IST CA 2 - G1, OU = Certification Authority, O = Apple Inc., C = US
i:C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
Roughly twice a month we release an updated stack image, that pulls in any security/bug fixes from the upstream Ubuntu LTS releases on which the Heroku stack images are based. Yesterday's release included an updated ca-certificates package:
https://devcenter.heroku.com/changelog-items/2028

Looking at the Ubuntu changelog for this package, the "GeoTrust Global CA" entries have been removed (this link is for Ubuntu 18.04 aka "Bionic", but the same change was made to all Ubuntu LTSes):
https://ubuntuupdates.org/package/core/bionic/main/security/ca-certificates
http://launchpad.net/ubuntu/+archive/primary/+files/ca-certificates_20201027ubuntu0.18.04.1_20210119~18.04.1.diff.gz

This removal itself comes from the upstream Mozilla CA root certificates list:
https://bugzilla.mozilla.org/show_bug.cgi?id=1670769
https://hg.mozilla.org/projects/nss/rev/4c69d6d0cf210546bef1eed490712462b9296c62

The Mozilla CA program has been been progressively removing support for the Symantec/GeoTrust certificate authorities since 2017 due to serious shortcomings in the way the CA was operated:
https://wiki.mozilla.org/CA:Symantec_Issues
https://wiki.mozilla.org/CA/Additional_Trust_Changes#Symantec

That is to say, Apple are using a CA that's coming to the end of a multi-year phasing out/distrust process, for which large parts of the ecosystem have already dropped support. It appears that https://api.push.apple.com works in browsers only because they have given a special (presumably temporary) exemption to the Apple intermediate certificate in the certificate chain.

To demonstrate this issue isn't Heroku-specific, I can reproduce using the upstream Ubuntu docker image:

$ docker run --rm -it ubuntu:20.04 bash
root@1b98119cc104:/# apt-get update -qq
root@1b98119cc104:/# apt-get install -yqq curl
...
root@1b98119cc104:/# curl -I https://api.push.apple.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
And also other common Docker images, such as the official Go images (in the case of this tag, based on Debian Buster):

$ docker run --rm -it golang:1.15.8 curl -I https://api.push.apple.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
...
It appears Apple are aware of this potential incompatibility, since the Apple Push Notification service docs say:

To establish HTTP/2-based TLS sessions with APNs, you must ensure that a GeoTrust Global CA root certificate is installed on each of your providers. If a provider is running macOS, this root certificate is in the keychain by default. On other systems, this certificate might require explicit installation. You can download this certificate from the GeoTrust Root Certificates website. Here is a direct link to the certificate.

I'm presuming there must be backwards compatibility reasons (eg old clients with an outdated root cert stores or that have unfortunately hardcoded the CA) that has meant Apple hasn't updated the certificate sooner.

Searching for "GeoTrust api.push.apple.com" I came across other push clients experiencing the same issue:
jchambers/pushy#809

It seems like until Apple fix the CA they are using, push clients may need to bundle the necessary CA certs themselves, such as prototyped here:
jchambers/pushy#810

I would recommend filing an issue against https://github.com/appleboy/gorush or perhaps more likely the upstream package it uses for the Apple-specific parts (https://github.com/sideshow/apns2) describing the issue and asking that the client bundle the CA certs. Alternatively if there is a way to pass in a CA cert to gorush, you may be able to work around the issue before they fix it upstream.

[edmorley]

Cross-reference sideshow/apns2/issues/182.