Why are exceptions in iteration thrown into a generator?

[npt]

I find the behavior described by the following doctests — where an exception thrown by a loop iterating a generator-backed stream is thrown into the generator and doesn't escape the loop — very surprising:

import asyncio

from aiostream import operator


@operator
async def yielder():
    try:
        yield 1
        yield 2
        yield 3
    except Exception as e:
        print("Exception in yielder:", e)


def raw_behavior():
    """
    >>> raw_behavior()
    Exception outside iterator: boom
    """

    async def iterator():
        async for x in yielder.raw():
            if x == 2:
                raise RuntimeError("boom")

    try:
        asyncio.run(iterator())
    except Exception as e:
        print("Exception outside iterator:", e)
    else:
        print("Iterator apparently succeeded")


def stream_behavior():
    """
    >>> stream_behavior()
    Exception in yielder: boom
    Iterator apparently succeeded
    """

    async def iterator():
        async with yielder().stream() as streamer:
            async for x in streamer:
                if x == 2:
                    raise RuntimeError("boom")

    try:
        asyncio.run(iterator())
    except Exception as e:
        print("Exception outside iterator:", e)
    else:
        print("Iterator apparently succeeded")

I don't understand the purpose of this, and don't see it described in the documentation, but it does look deliberate (AsyncIteratorContext's __aexit__ athrows any exception into the generator). What's the use case for it?

[vxgmichel]

Hi @npt

First I want to point out that there are two parts in your original question

where an exception thrown by a loop iterating a generator-backed stream is thrown into the generator

This I'll address later.

and doesn't escape the loop

This one is simple: it doesn't escape the loop because it's not re-raised in yielder. Try this code instead:

@operator
async def yielder():
    try:
        yield 1
        yield 2
        yield 3
    except Exception as e:
        print("Exception in yielder:", e)
        raise

and you'll get the expected behavior.

Now about the first point:

where an exception thrown by a loop iterating a generator-backed stream is thrown into the generator

This behavior is indeed on purpose because we want to emulate the behavior of a context manager. Note the same athrow in __aexit__ logic in the asynccontextmanager implementation:
https://github.com/python/cpython/blob/d2a8d57123676fe45c8268c461bbe8922a07b015/Lib/contextlib.py#L235

In the context of aiostream, this allows us to treat streams as both a context and an iterator, i.e the stream is both responsible for managing resources and producing items. The operator decorator allows us to write both logic in the same scope.

Note than when several operators are piped, this creates a stack of nested contexts. If an exception is raised in the deepest level (in your example the async for loop), then the exception will bubble up through all the operators, from right to left (just like several context managers stack together).

However, the exception usually traverse the operators without being affected. It's only in rare situations that the operators catch the exception in order to do something specific with it (for instance if you were to write a silence operator or something like that)

I hope this answers your question :)