Generators and coroutines

The origin story

Generators came to Python in 2.2 (2001) via PEP 255. The motivation was simple: writing iterators by hand (implementing __iter__ and __next__) was tedious. Generators let you write a function with yield and Python automatically turns it into an iterator.

But generators ended up being more important than that. PEP 342 (2005) extended them with send and throw, turning one-way iterators into bidirectional coroutines. PEP 380 (2009) added yield from for delegation. By the time PEP 492 (2015) introduced async/await, the runtime mechanics had been built up over a decade.

Today, every coroutine in Python is a generator under the hood. The async syntax is a friendlier API over the same bytecodes.

Generator bytecode

A function with yield in its body becomes a generator function. The compiler sets a flag (CO_GENERATOR) on the code object. Calling the function does not execute the body; it creates a generator object with a saved frame.

The body is compiled normally, but yield becomes special bytecode (YIELD_VALUE). When the interpreter hits YIELD_VALUE, it:

1. Pops the value to yield from the stack.
2. Saves the current frame state (instruction pointer, stack, locals).
3. Returns control to the caller with the yielded value.

The frame is kept alive. Next call to next(gen) resumes execution from the saved frame, pushing whatever the caller sent (or None) onto the stack and continuing.

import dis
 
def gen():
    yield 1
    yield 2
 
dis.dis(gen)
# Look for YIELD_VALUE instructions

Sending values in (PEP 342)

Before PEP 342, generators were one-way. PEP 342 added:

• generator.send(value): resume the generator, with value becoming the result of the yield expression.
• generator.throw(exception): resume by raising an exception at the yield point.
• generator.close(): raise GeneratorExit to allow cleanup.

The change made yield a two-way valve. The yielded value goes out; the sent value comes in.

def accumulator():
    total = 0
    while True:
        x = yield total
        if x is None: break
        total += x
 
g = accumulator()
next(g)          # 0 (prime to first yield)
g.send(5)        # 5
g.send(3)        # 8
g.send(None)     # StopIteration

This was the first appearance of coroutines in Python. You could write cooperative state machines that consumed inputs and emitted outputs without threads.

yield from (PEP 380)

yield from sub_gen delegates iteration to another generator. Equivalent to:

for x in sub_gen:
    yield x

But also forwards sent values, thrown exceptions, and the final return value. It is the right way to compose generators.

def inner():
    x = yield 1
    print("inner got", x)
    return "done"
 
def outer():
    result = yield from inner()
    print("inner returned", result)
    yield 2
 
g = outer()
next(g)         # 1
g.send("hi")    # "inner got hi", "inner returned done", 2

PEP 380 also gave generators a return value syntax (previously a SyntaxError). The returned value is attached to the StopIteration exception (StopIteration.value).

yield from was the killer feature for early asyncio. You could write coroutines as generators and chain them naturally. Before yield from, you had to manually pump sub-generators in a loop.

async and await (PEP 492)

PEP 492 introduced new syntax: async def to declare a coroutine function and await to wait on an awaitable. The motivation was twofold:

1. Make coroutines syntactically distinct from generators, to avoid confusion.
2. Add new awaitable protocols (__aiter__, __anext__, __aenter__, __aexit__) for async iteration and context managers.

Internally, async def produces a "coroutine object" - similar to a generator object but with a different flag (CO_COROUTINE). It cannot be iterated with for; it must be awaited.

await expr is roughly equivalent to yield from expr.__await__(). The runtime mechanics are the same as generators. The Task system in asyncio drives coroutines by calling their send method repeatedly.

async def fetch():
    data = await get_data()
    return process(data)
 
# Behind the scenes:
def fetch():
    data = yield from get_data().__await__()
    return process(data)

The async syntax is more than sugar though. It enforces separation: you cannot await in a sync function, you cannot use for on a coroutine. The compiler enforces these rules, which catches bugs at parse time instead of runtime.

Async iterators and context managers

async for and async with extend the protocol:

class AsyncCounter:
    def __init__(self, n): self.n = n; self.i = 0
    def __aiter__(self): return self
    async def __anext__(self):
        if self.i >= self.n: raise StopAsyncIteration
        await asyncio.sleep(0.1)
        self.i += 1
        return self.i
 
async def main():
    async for x in AsyncCounter(3):
        print(x)

async for calls __aiter__ to get an async iterator, then repeatedly calls __anext__ (which returns a coroutine) and awaits it.

async with calls __aenter__ and __aexit__ (both async). Useful for resources that need async setup or cleanup, like database connections or HTTP sessions.

Async generators (PEP 525)

Python 3.6 added async generators: async def functions that contain yield. They produce values asynchronously.

async def fetch_pages(urls):
    for url in urls:
        data = await fetch(url)
        yield data
 
async def main():
    async for page in fetch_pages(urls):
        process(page)

Async generators have their own protocol: __aiter__ returns self; __anext__ returns a coroutine that yields the next value or raises StopAsyncIteration. They give you streaming with async I/O cleanly.

Generator-based coroutines (deprecated path)

Before PEP 492, asyncio used @asyncio.coroutine to mark generators as coroutines. This pattern is now deprecated and will be removed.

# Old way (do not use):
@asyncio.coroutine
def old_coro():
    data = yield from sub_coro()
    return data
 
# Modern way:
async def new_coro():
    data = await sub_coro()
    return data

If you see @asyncio.coroutine or yield from in asyncio code, it's pre-3.5 style. Modernize it.

Common patterns and pitfalls

Memory-efficient pipelines. Generators chain naturally without intermediate lists:

def read_lines(path):
    with open(path) as f:
        for line in f: yield line
 
def parse(lines):
    for line in lines: yield json.loads(line)
 
def filter_errors(records):
    for r in records:
        if r['level'] == 'ERROR': yield r
 
errors = filter_errors(parse(read_lines("big.log")))
for err in errors: print(err)

Memory: O(1). The whole pipeline runs lazily, one record at a time.

The one-shot trap. Generators exhaust:

g = (x*x for x in range(5))
sum(g)   # 30
sum(g)   # 0! Generator is exhausted.

If you need to iterate twice, store the values: vals = list(g). Or use a generator function (callable) instead of a generator (object) and call it each time.

Eager vs lazy with bool(). bool(generator) is always True, even if the generator would yield nothing. To check if a generator has values, use next(gen, sentinel) and handle the sentinel.

Closing generators. When a generator goes out of scope mid-iteration, Python calls close() on it, which throws GeneratorExit at the yield point. If the generator catches and ignores it, you get a RuntimeError. Always re-raise or let it propagate.

try/finally and yield. Code in a finally block runs when the generator is closed or exhausted. Useful for cleanup:

def with_resource():
    r = acquire()
    try:
        for x in process(r):
            yield x
    finally:
        release(r)

Mixing async and sync iteration. You cannot use async for on a sync iterable, and vice versa. Adapt with asyncio.to_thread or async wrapper libraries.

Why generators became the basis for asyncio

The connection between generators and coroutines is not coincidence. A coroutine fundamentally is "a function that can pause." Generators already had the pause/resume machinery (yield/next). PEP 342 added the two-way communication (send/throw). PEP 380 added composition (yield from). By the time asyncio was designed in PEP 3156, all the pieces were in place.

Reusing generators for coroutines meant no new interpreter machinery. The same YIELD_VALUE bytecode that suspended a generator suspended a coroutine. The same frame-saving logic worked for both. The async syntax in PEP 492 added type distinctions and new protocols, but the runtime is identical.

This is why you sometimes see "coroutines are just generators that yield to an event loop instead of to a for loop." It is the correct mental model.

Greenlets, threads, and other alternatives

Generators are not the only way to implement coroutines. Alternatives:

Threads. True OS threads. Heavy (MB of stack each), preemptive. Use when you need parallelism (with the GIL caveat).

Greenlets. Userspace coroutines with full stacks, switched explicitly. Used by gevent. They look like threads (any function call can yield) but are cooperative. Powerful but harder to reason about.

Async/await. What we covered. Cooperative, explicit yield points, no per-task stack.

Trio and asyncio both use the async/await model. gevent uses greenlets. Twisted historically used callbacks but now supports async/await too. The async/await model has won mindshare because its explicit yields make reasoning about concurrency tractable.

Mental model

A generator is a function that can pause. The pause point is yield. Each call to next runs the function until the next yield, returns the yielded value, and saves the state.

A coroutine is the same thing with async/await syntax and a different driver. The event loop drives coroutines by repeatedly calling send (or throw) on them. Each await is a save point.

The two share the same bytecode mechanism. The async syntax adds type safety (you can't mix them) and new protocols (async iteration, async context managers, async generators). But the runtime is one system, evolved over 25 years.