Weekly Python tip: learn the iterator protocol

Python tip: learn the iterator protocol

Unlike counter-based for loops in other languages, Python's for loops don't use indexes.

They can't rely on indexes because many of the things we loop over don't have them: sets, files, and generators all work with for loops, but none of them can be indexed.

So if not by index, how does a for loop get each item?

Well, for loops rely on Python's iterator protocol.

You may never need to use the iterator protocol directly, but learning how it works will help you understand what is (and isn't) possible with Python's loops.

The iterator protocol

The iterator protocol boils down to two terms, each defined by the built-in function it works with:

1. iterable: an object that can be passed to iter() to get an iterator from it
2. iterator: an object that can be passed to next() to get its next item (or raise a StopIteration exception when there are no more items)

In other words, you get an iterator from an iterable, and you get items from an iterator.

But there is one additional requirement for iterators:

Every iterator in Python should also be an iterable and should be its own iterator (meaning iter(iterator) is iterator should be True).

That's pretty much the whole iterator protocol. But those definitions alone don't explain why it exists or how it's used.

I think it's easiest to understand the iterator protocol by seeing how it powers for loops in Python.

How for loops work

This function, which uses a for loop:

def print_each(iterable):
for item in iterable:
print(item)

Is equivalent to this function, which does the same thing with a while loop and the iter and next functions:

def print_each(iterable):
iterator = iter(iterable)
while True:
try:
item = next(iterator)
except StopIteration:
break # Iterator exhausted: stop the loop
else:
print(item)

To loop over an iterable, a for loop gets an iterator from it, and then asks that iterator for its next item, over and over, until the iterator raises StopIteration to say it's out of items.

There are no indexes involved, so this works with any iterable:

>>> print_each({1, 2, 3})
1
2
3

Comprehensions, tuple unpacking, the list constructor, and nearly every other form of iteration in Python work this same way: they're all powered by the iterator protocol.

Handy iterator facts

It's interesting to know that a for loop calls iter once and then calls next over and over, but you're probably not going to re-implement a for loop yourself... there's no point!

But that doesn't mean that understanding iterators is pointless. It does explain a lot of under-the-hood Python behavior that can otherwise seem strange.

Here are some iterator facts worth internalizing, ideally through practice:

• Files are iterators: each next call on a file object gives you the file's next line.
• Iterators pick up where they left off: calling iter on an iterator gives you that same iterator back, so a loop over an iterator that's already been partially used continues from wherever the last next call stopped. That's why calling next(file) just before looping over a CSV file will skip its header line.
• Generators are iterators: calling a generator function doesn't run its code right away. It returns a generator object, and each next call on that object runs the function's code (from where it last paused) until it reaches its next yield.
• Looping helpers are usually powered by iterators: enumerate, zip, reversed, etc. are all iterator-powered tools.
• Iterators can be infinite: a for loop only stops on its own when StopIteration is raised, and some iterators never raise it. For example, itertools.count() counts upward forever, so a loop over it needs a break (or break-equivalent) to stop.
• next accepts a default value: next(iterator, default) returns default instead of raising StopIteration when the iterator is exhausted. So next(iter(items), None) gets the first item from any iterable, or None if the iterable is empty.

Iteration is iter and next

The next time you write a for loop, remember what Python is doing under the hood: getting an iterator from your iterable and asking that iterator for its next item until there aren't any more.

For more on how all of this fits together, see my article on the iterator protocol.

And to really practice this topic... see my brand new course below.

New course 📣: Lazy Looping in Python

7 daily Python tips on advanced Python patterns

You are receiving this email because you signed up for Python Morsels weekly newsletter.

Want fewer emails?

or

12651 Monterey Cypress Way, San Diego, CA 92130