Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bundling mozilla root CA store for when there are no default certs #45

Merged
merged 3 commits into from
Feb 14, 2024

Conversation

miki725
Copy link
Contributor

@miki725 miki725 commented Feb 8, 2024

not all machines have system certs such as in busybox containers and so by bundling certs from Mozilla we can fallback to them when system certs are missing

nimutils/net.nim Outdated Show resolved Hide resolved
nimutils/net.nim Outdated Show resolved Hide resolved
nimutils/net.nim Outdated Show resolved Hide resolved
Copy link

@ee7 ee7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bundling the certs sounds good to me, and I've double checked that the ones in this PR are identical to the ones at https://ccadb.my.salesforce-sites.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Websites (modulo whitespace changes).

Note that the certs are 218 KiB, not 18 KiB like I wrote before - I must have either misread/mistyped it. That's still fine by me. But after we merge this PR, we could consider opening a ticket to track the idea to reduce them.

Apologies: I've run out of time right now to review the other changes closely enough for me to approve.

nimutils/net.nim Outdated Show resolved Hide resolved
not all machines have system certs such as in busybox containers
and so by bundling certs from Mozilla we can fallback to them
when system certs are missing
@miki725
Copy link
Contributor Author

miki725 commented Feb 11, 2024

@viega updated to curl the certs on each compile and then using openssl command to validate download is not corrupted/etc and that its a valid PEM file

@ee7
Copy link

ee7 commented Feb 12, 2024

I think curling at build time is slightly better overall, partly because it avoids the certs being outdated when building an old version of chalk.

To answer my own question of "how often are these certs updated?", I had a vague memory of curl having a page for this. From here:

This bundle was generated at Tue Dec 12 04:12:04 2023 GMT.

This PEM file contains the datestamp of the conversion and we only make a new conversion if there's a change in either the script or the source file. This service checks for updates every day.

CA file revisions per date of appearance

Date Certificates
2023-12-12 145
2023-08-22 141
2023-05-30 137
2023-01-10 137
2022-10-11 142
2022-07-19 140
2022-04-26 135
2022-03-29 132
2022-03-18 132
2022-02-01 133

With this PR, building nimutils the output is:

Embedding Mozilla Root CA store with certificates total found: 147
For more information see https://wiki.mozilla.org/CA/Included_Certificates

I'm not sure why there's 2 more, unless the certs were updated today.

But maybe the curl URL is a slightly preferable upstream? Minor advantages:

  • it publishes a checksum
  • it publishes an update log
  • it has a comment for each cert, like everyone's favorite:
Hongkong Post Root CA 3
=======================
-----BEGIN CERTIFICATE-----
MIIFzzCCA7egAwIBAgIUCBZfikyl7ADJk0DfxMauI7gcWqQwDQYJKoZIhvcNAQELBQAwbzELMAkG
A1UEBhMCSEsxEjAQBgNVBAgTCUhvbmcgS29uZzESMBAGA1UEBxMJSG9uZyBLb25nMRYwFAYDVQQK
Ew1Ib25na29uZyBQb3N0MSAwHgYDVQQDExdIb25na29uZyBQb3N0IFJvb3QgQ0EgMzAeFw0xNzA2
MDMwMjI5NDZaFw00MjA2MDMwMjI5NDZaMG8xCzAJBgNVBAYTAkhLMRIwEAYDVQQIEwlIb25nIEtv
bmcxEjAQBgNVBAcTCUhvbmcgS29uZzEWMBQGA1UEChMNSG9uZ2tvbmcgUG9zdDEgMB4GA1UEAxMX
SG9uZ2tvbmcgUG9zdCBSb290IENBIDMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCz
iNfqzg8gTr7m1gNt7ln8wlffKWihgw4+aMdoWJwcYEuJQwy51BWy7sFOdem1p+/l6TWZ5Mwc50tf
jTMwIDNT2aa71T4Tjukfh0mtUC1Qyhi+AViiE3CWu4mIVoBc+L0sPOFMV4i707mV78vH9toxdCim
5lSJ9UExyuUmGs2C4HDaOym71QP1mbpV9WTRYA6ziUm4ii8F0oRFKHyPaFASePwLtVPLwpgchKOe
sL4jpNrcyCse2m5FHomY2vkALgbpDDtw1VAliJnLzXNg99X/NWfFobxeq81KuEXryGgeDQ0URhLj
0mRiikKYvLTGCAj4/ahMZJx2Ab0vqWwzD9g/KLg8aQFChn5pwckGyuV6RmXpwtZQQS4/t+TtbNe/
JgERohYpSms0BpDsE9K2+2p20jzt8NYt3eEV7KObLyzJPivkaTv/ciWxNoZbx39ri1UbSsUgYT2u
y1DhCDq+sI9jQVMwCFk8mB13umOResoQUGC/8Ne8lYePl8X+l2oBlKN8W4UdKjk60FSh0Tlxnf0h
+bV78OLgAo9uliQlLKAeLKjEiafv7ZkGL7YKTE/bosw3Gq9HhS2KX8Q0NEwA/RiTZxPRN+ZItIsG
xVd7GYYKecsAyVKvQv83j+GjHno9UKtjBucVtT+2RTeUN7F+8kjDf8V1/peNRY8apxpyKBpADwID
AQABo2MwYTAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAfBgNVHSMEGDAWgBQXnc0e
i9Y5K3DTXNSguB+wAPzFYTAdBgNVHQ4EFgQUF53NHovWOStw01zUoLgfsAD8xWEwDQYJKoZIhvcN
AQELBQADggIBAFbVe27mIgHSQpsY1Q7XZiNc4/6gx5LS6ZStS6LG7BJ8dNVI0lkUmcDrudHr9Egw
W62nV3OZqdPlt9EuWSRY3GguLmLYauRwCy0gUCCkMpXRAJi70/33MvJJrsZ64Ee+bs7Lo3I6LWld
y8joRTnU+kLBEUx3XZL7av9YROXrgZ6voJmtvqkBZss4HTzfQx/0TW60uhdG/H39h4F5ag0zD/ov
+BS5gLNdTaqX4fnkGMX41TiMJjz98iji7lpJiCzfeT2OnpA8vUFKOt1b9pq0zj8lMH8yfaIDlNDc
eqFS3m6TjRgm/VWsvY+b0s+v54Ysyx8Jb6NvqYTUc79NoXQbTiNg8swOqn+knEwlqLJmOzj/2ZQw
9nKEvmhVEA/GcywWaZMH/rFF7buiVWqw2rVKAiUnhde3t4ZEFolsgCs+l6mc1X5VTMbeRRAc6uk7
nwNT7u56AQIWeNTowr5GdogTPyK7SBIdUgC0An4hGh6cJfTzPV4e0hz5sy229zdcxsshTrD3mUcY
hcErulWuBurQB7Lcq9CClnXO0lD+mefPL5/ndtFhKvshuzHQqp9HpLIiyhY6UFfEW0NnxWViA0kB
60PZ2Pierc+xYw5F9KBaLJstxabArahH9CdMOA0uG0k7UvToiIMrVCjU8jVStDKDYmlkDJGcn5fq
dBb9HxEGmpv0
-----END CERTIFICATE-----

Copy link

@ee7 ee7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, except I think we need to invalidate the cache.

nimutils/net.nim Outdated Show resolved Hide resolved
nimutils/net.nim Outdated Show resolved Hide resolved
nimutils/net.nim Outdated Show resolved Hide resolved
nimutils/net.nim Show resolved Hide resolved
nimutils/net.nim Show resolved Hide resolved
const
opensslCmd = "openssl storeutl -noout -certs /dev/stdin"
(check, checkExitCode) = gorgeEx(opensslCmd, input=contents)
checkLines = check.splitLines()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking comment: this is a relatively costly split, since it allocates a string for each line (3500 ish of them) at compile time - it's calling the splitLines proc, not the iterator.

It probably doesn't have a significant impact on compile times, but it'd be better to get the index of the start of the last line, and slice from there to the end.

nimutils/net.nim Show resolved Hide resolved
nimutils/net.nim Show resolved Hide resolved
nimutils/net.nim Outdated Show resolved Hide resolved
@miki725 miki725 requested a review from ee7 February 12, 2024 20:31
Copy link

@ee7 ee7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, although I haven't written a book on OpenSSL :)

I'm happy to defer this question, but did we have any thoughts on using the certs published by curl instead? Aside from the reasons mentioned in #45 (comment), the salesforce URL looks weird to me. Perhaps the curl URL is more likely to stay working?

nimutils/net.nim Outdated Show resolved Hide resolved
nimutils/net.nim Outdated Show resolved Hide resolved
@miki725
Copy link
Contributor Author

miki725 commented Feb 12, 2024

not sure about grabbing it from curls site. thought to get it from the original link in case curl does any massaging but happy to change. cc @MyNameIsMeerkat @viega thoughts?

@MyNameIsMeerkat
Copy link

not sure about grabbing it from curls site. thought to get it from the original link in case curl does any massaging but happy to change. cc @MyNameIsMeerkat @viega thoughts?

I feel like we are splitting hairs at this point, the salesforce/microsoft link looks weird for sure but is one of the official sources cited by CCADB (https://www.ccadb.org/resources) so I have no concerns using it and would tend to have more trust in that as a source than using curl's website tbh.

Let's just get this done and merged

@ee7
Copy link

ee7 commented Feb 14, 2024

I did approve, so I'm happy for us to merge this. From my side, I think we can defer any further discussion to a follow-up issue/PR.

@miki725 miki725 merged commit 4ab1d36 into dev Feb 14, 2024
1 check passed
@miki725 miki725 deleted the ms/tls branch February 14, 2024 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants