Python requests package interferes with keyvault

[lungj]

SUMMARY

Using an inventory plug-in that uses the requests package causes Azure keyvault to fail.

ISSUE TYPE

• Bug Report

COMPONENT NAME

Lookup module azure.azcollection.azure_keyvault_secret

ANSIBLE VERSION

ansible [core 2.16.0]
  config file = /Users/jlung/Documents/checkout/proxmox/ansible.cfg
  configured module search path = ['/Users/jlung/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/jlung/Library/Caches/pypoetry/virtualenvs/dummy-5fIXO6XN-py3.11/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/jlung/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/jlung/Library/Caches/pypoetry/virtualenvs/dummy-5fIXO6XN-py3.11/bin/ansible
  python version = 3.11.4 (main, Jun 10 2023, 09:42:27) [Clang 14.0.3 (clang-1403.0.22.14.1)] (/Users/jlung/Library/Caches/pypoetry/virtualenvs/dummy-5fIXO6XN-py3.11/bin/python)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /Users/jlung/.ansible/collections/ansible_collections
Collection         Version
------------------ -------
azure.azcollection 2.0.0  

CONFIGURATION

CONFIG_FILE() = None

OS / ENVIRONMENT

macOS Sonoma 14.1.2 on Apple Silicon (M2)
Dev environment is a nix shell using nix-shell -p python3 using a venv created by poetry.
As per this, setting

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

STEPS TO REPRODUCE

==> inventory/dummy.yaml <==
---
plugin: "dummy"


==> inventory_plugins/dummy/dummy.py <==
#!/usr/bin/env python
import requests
from ansible.plugins.inventory import BaseInventoryPlugin, Constructable, Cacheable

class InventoryModule(BaseInventoryPlugin, Constructable, Cacheable):
    NAME = 'dummy'
    def parse(self, inventory, loader, path, cache=True):
        try:
            # If the line below is replaced with `pass`, the Azure keyvault doesn't crash.
            requests.get("http://127.0.0.1")
        except: pass

==> playbooks/test/keyvault.yaml <==
---
# Test connection to Azure keyvault.
- hosts: "localhost"
  gather_facts: no
  tasks:
    - debug:
        msg: >-
          {{
            lookup(
              "azure.azcollection.azure_keyvault_secret",
              "test",
              vault_url="https://myvault.vault.azure.net"
            )
          }}

Invoked using

ansible-playbook -i inventory/dummy.yaml playbooks/test/keyvault.yaml

EXPECTED RESULTS

Get the value associated with the key.

ACTUAL RESULTS

Things work if the requests.get call is commented out or if the requests package is used in a dynamic inventory file (as opposed to a plug-in). As far as I can tell, no other packages that use requests have had any issues. Otherwise, here are the actual results:

ansible-playbook [core 2.16.0]
  config file = /Users/jlung/Documents/checkout/proxmox/ansible.cfg
  configured module search path = ['/Users/jlung/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/jlung/Library/Caches/pypoetry/virtualenvs/dummy-5fIXO6XN-py3.11/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/jlung/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/jlung/Library/Caches/pypoetry/virtualenvs/dummy-5fIXO6XN-py3.11/bin/ansible-playbook
  python version = 3.11.4 (main, Jun 10 2023, 09:42:27) [Clang 14.0.3 (clang-1403.0.22.14.1)] (/Users/jlung/Library/Caches/pypoetry/virtualenvs/dummy-5fIXO6XN-py3.11/bin/python)
  jinja version = 3.1.2
  libyaml = True
Using /Users/jlung/Documents/checkout/proxmox/ansible.cfg as config file
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /Users/jlung/Documents/checkout/proxmox/inventory/dummy.yaml as it did not pass its verify_file() method
script declined parsing /Users/jlung/Documents/checkout/proxmox/inventory/dummy.yaml as it did not pass its verify_file() method
Using inventory plugin 'test_null' to process inventory source '/Users/jlung/Documents/checkout/proxmox/inventory/dummy.yaml'
Parsed /Users/jlung/Documents/checkout/proxmox/inventory/dummy.yaml inventory source with auto plugin
redirecting (type: cache) ansible.builtin.yaml to community.general.yaml
Loading collection community.general from /Users/jlung/.ansible/collections/ansible_collections/community/general
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
Loading callback plugin default of type stdout, v2.0 from /Users/jlung/Library/Caches/pypoetry/virtualenvs/dummy-5fIXO6XN-py3.11/lib/python3.11/site-packages/ansible/plugins/callback/default.py
Loading collection ansible.posix from /Users/jlung/Library/Caches/pypoetry/virtualenvs/dummy-5fIXO6XN-py3.11/lib/python3.11/site-packages/ansible_collections/ansible/posix
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.
Attempting to use 'ansible.posix.profile_tasks' callback.
Loading callback plugin ansible.posix.profile_tasks of type aggregate, v2.0 from /Users/jlung/Library/Caches/pypoetry/virtualenvs/dummy-5fIXO6XN-py3.11/lib/python3.11/site-packages/ansible_collections/ansible/posix/plugins/callback/profile_tasks.py
Attempting to use 'ansible.posix.profile_roles' callback.
Loading callback plugin ansible.posix.profile_roles of type aggregate, v2.0 from /Users/jlung/Library/Caches/pypoetry/virtualenvs/dummy-5fIXO6XN-py3.11/lib/python3.11/site-packages/ansible_collections/ansible/posix/plugins/callback/profile_roles.py

PLAYBOOK: keyvault.yaml ******************************************************************************************************************************************************************************************************************************************************
Positional arguments: playbooks/test/keyvault.yaml
verbosity: 5
connection: ssh
become_method: sudo
tags: ('all',)
inventory: ('/Users/jlung/Documents/checkout/proxmox/inventory/dummy.yaml',)
forks: 5
1 plays in playbooks/test/keyvault.yaml

PLAY [localhost] *************************************************************************************************************************************************************************************************************************************************************

TASK [debug] *****************************************************************************************************************************************************************************************************************************************************************
task path: /Users/jlung/Documents/checkout/proxmox/playbooks/test/keyvault.yaml:8
Thursday 07 December 2023  15:27:06 -0500 (0:00:00.008)       0:00:00.014 ***** 
Thursday 07 December 2023  15:27:06 -0500 (0:00:00.008)       0:00:00.014 ***** 
Loading collection azure.azcollection from /Users/jlung/.ansible/collections/ansible_collections/azure/azcollection
ERROR! A worker was found in a dead state

[Fred-sun]

@lungj Could you describe your problem in more detail? From the above information, it is not clear what kind of problem you have encountered, thank you!

[lungj]

Hi @Fred-sun ! The azure.azcollection.azure_keyvault_secret causes Python to hard-crash (no Python exceptions or anything of the sort). This happens if it is used after a dynamic inventory module is used if the dynamic inventory module uses requests.get(). Because Python has simply died (rather than raised an exception, let alone a proper Ansible exception), we get the ERROR! A worker was found in a dead state message.

I know my answer above doesn't provide much detail, but hopefully it makes it clearer what the problem encountered is.

I'm happy to provide any more specific details, but I'm not sure what to offer that's not already included in the original bug report.

[Fred-sun]

@lungj Are you saying that if there is a problem with the request.get() message, it will be thrown as an exception, and execution will be interrupted and the exception message will be printed? Thank you!

[lungj]

I'm not sure I understand what you mean by "request.get() message". Do you mean my code example above where I call requests.get() and try-catch it? Even if request.get() succeeds in the dynamic inventory, Python still dies while doing an azure.azcollection.azure_keyvault_secret lookup.

I think there's something going on at a lower level than at pure Python; there's a segfault or something similarly destructive that's preventing Python from raising an exception. ERROR! A worker was found in a dead state is coming from Ansible when the TaskQueueManager has a dead process (as opposed to simply a process that completed successfully or failed due to a Python exception) - at least as far as I can make out.

I've used other modules that use request.get() successfully after a dynamic inventory module uses it so this issue appears to be specific to azure.azcollection. I'm guessing azure.azcollection.azure_keyvault_secret uses the requests module (or urllib3) in a particular way that interacts with a previous call to requests.get that causes something bad to happen in a .so file. If no call to requests.get() is made in a dynamic inventory before the Azure keyvault is used, everything behaves as expected during the lookup.

As a side note, I've put using Azure Keyvault on the back burner for various reasons so resolving this isn't a priority for me. (But your responses are definitely appreciated!)