Waiter encountered a terminal failure state over NetworkInterface resource

[mtkh]

Hi,

I am getting the following error sometimes while running this code. Most of the time it works but sometimes it fails.

ec2 = boto3.resource('ec2')
ec2_client = boto3.client('ec2')

eni = ec2.create_network_interface(
    SubnetId='subnet-0c25486402ed9077b',
    Groups=['sg-01cf993d4c50f7f3b'],
    Description='Test-Interface',
    TagSpecifications=[
        {
        'ResourceType': 'network-interface',
        'Tags': [
            {
            'Key': 'Name',
            'Value': "Test-Interface"
            },
        ]},
    ])

print(eni)
print(eni.id, type(eni.id))

waiter = ec2_client.get_waiter('network_interface_available')
    waiter.wait(
        Filters= [
        {
            'Name' : 'status', 
            'Values': [
            'available'
            ]
        },
        ],
        NetworkInterfaceIds=[eni_id],
        WaiterConfig={
        'Delay': 5,
        'MaxAttempts': 5 
        })

ec2.NetworkInterface(id='eni-0f6adb00f13134b1d')
eni-0f6adb00f13134b1d <class 'str'>
Traceback (most recent call last):
  File "/home/user/./eni_availability.py", line 62, in <module>
    main()
  File "/home/user/./eni_availability.py", line 28, in main
    ret = wait_eni_available(eni.id)
  File "/home/user/./eni_availability.py", line 39, in wait_eni_available
    waiter.wait(
  File "/usr/lib/python3/dist-packages/botocore/waiter.py", line 53, in wait
    Waiter.wait(self, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/waiter.py", line 350, in wait
    raise WaiterError(
botocore.exceptions.WaiterError: Waiter NetworkInterfaceAvailable failed: Waiter encountered a terminal failure state: Matched expected service error code: InvalidNetworkInterfaceID.NotFound

[aBurmeseDev]

Hi @mtkh, thanks for reaching out.
I would recommend increasing Delay and MaxAttempts values in WaiterConfig if you haven’t already tried so. Also here’s boto3 doc for waiter.NetworkInterfaceAvailable that might be helpful. Please let us know if it resolves the issue.

[mtkh]

thanks @aBurmeseDev but I have tried that option too already. I put different Delay and MaxAttempts but it still the same issue

[romos]

Similar issue to me.

From top of my head: can there be just an issue with how InvalidNetworkInterfaceID.NotFound ENI state is treated by the waiter? Theoretically, the waiter should have retried several times according to Delay/MaxAttempts, but it just fails immediately.

I tried to look up this behavior across boto repositories - I'm not very familiar with it yet, so the only place I came across is https://github.com/boto/botocore/blob/23ee17f5446c78167ff442302471f9928c3b4b7c/botocore/data/ec2/2016-11-15/waiters-2.json#L352
In that file, ".NotFound"-states usually have "retry" action when waiting for availability of a resource, while the one which is ENI-related has "failure" instead, which may be the root cause, for example:

    "VpcExists": {
#...
        {
          "matcher": "error",
          "expected": "InvalidVpcID.NotFound",
          "state": "retry"
        }
#...
    "NetworkInterfaceAvailable": {
#...
        {
          "expected": "InvalidNetworkInterfaceID.NotFound",
          "matcher": "error",
          "state": "failure"
        }

Can it be the root cause, @aBurmeseDev ?

[romos]

I've created an issue for that: boto/botocore#2717

[aBurmeseDev]

I've contacted our SDK service team regarding this issue and opened a new issue there. aws/aws-sdk#301
Please refer to that issue for updates going forward!