botocore fails to import with older versions of pyopenssl installed due to an AttributeError

[gibsondan]

Describe the bug

if you have pyopenssl < 22.1.0 installed and try to import botocore.httpsession, you hit the following error on import:

>>> import botocore.httpsession
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/botocore/httpsession.py", line 45, in <module>
    from urllib3.contrib.pyopenssl import (
  File "/usr/local/lib/python3.11/site-packages/urllib3/contrib/pyopenssl.py", line 43, in <module>
    import OpenSSL.SSL  # type: ignore[import-untyped]
    ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/local/lib/python3.11/site-packages/OpenSSL/crypto.py", line 1570, in <module>
    class X509StoreFlags(object):
  File "/usr/local/lib/python3.11/site-packages/OpenSSL/crypto.py", line 1589, in X509StoreFlags
    NOTIFY_POLICY = _lib.X509_V_FLAG_NOTIFY_POLICY
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'. Did you mean: 'X509_V_FLAG_EXPLICIT_POLICY'?

The callsite guards against ImportErrors (i.e. what you would see if the package was not installed at all) but does not guard against AttributeErrors, so the import fails:

botocore/botocore/httpsession.py

Lines 39 to 49 in 1ad3285

	try:
	# pyopenssl will be removed in urllib3 2.0, we'll fall back to ssl_ at that point.
	# This can be removed once our urllib3 floor is raised to >= 2.0.
	with warnings.catch_warnings():
	warnings.simplefilter("ignore", category=DeprecationWarning)
	# Always import the original SSLContext, even if it has been patched
	from urllib3.contrib.pyopenssl import (
	orig_util_SSLContext as SSLContext,
	)
	except ImportError:
	from urllib3.util.ssl_ import SSLContext

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

botocore to still import on older versions of pyopenssl

Current Behavior

An error is raised on import with the above stack traced

Reproduction Steps

In a new venv (on a Macbook M2):

pip install -U botocore 'pyopenssl<22.1.0'
python -c 'import botocore.httpsession'

Possible Solution

Catch AttributeError as well as ImportError in the linked import callsite above

Additional Information/Context

No response

SDK version used

1.35.71

Environment details (OS name and version, etc.)

Mac OS M2

[tim-finnigan]

Thanks for reaching out — this appears to be a duplicate of boto/boto3#3585, where a maintainer replied:

Thanks for the report! Boto3 doesn't actually use pyOpenSSL. In fact, we'd actively discourage for most use cases in favor of the ssl module that's present in all supported versions of Python.

What we're hitting here is urllib3 accessing code within pyopenssl that may not have bindings available. Depending on which version of urllib3 you're using, it typically supports pyOpenSSL>=0.14.0. It looks like that may not be accurate on every system (or PyOpenSSL has released some breaking changes in some versions).

From Boto3's side, we can expand our exception handling to deal with the AttributeError that will hopefully help prevent this edge case. I'll leave this marked as a bug while we're working on a patch.

I'm going to close this as a duplicate and we can continue tracking this in boto/boto3#3585.