Python Iterator and Iterable Basics

Introduction

A Python iterator is an object that enables the elements inside a collection object, such as a lists, tuples, dictionaries, and strings, to be looped over and accessed one at a time. Such collections are called iterables and their common characteristic is that they can provide an iterator.

The ability to iterate over elements is achieved through a common interface, i.e. protocol, that allows you to use a consistent approach regardless of the specific type of iterable object. Many Python functions are able to work through the iterable and iterator interfaces and give the same result regardless of the type of underlying object.

Iterator vs. Iterable

Every iterable object is capable of returning an iterator. The distinction between an iterator and an iterable can be confusing, but simply put, an iterable is the object that provides the collection of items, i.e. the data, while the iterator is a tool for traversing that data.

The only technical requirement for an object to be an iterable is to provide a method called __iter__() that returns its iterator.

Example:

l = [1,2,3,4,5]
l.__iter__()

Output:

<list_iterator object at 0x7ff6962eb190>

The above example shows a call to __iter__() for a list of five elements. You can see the printed version of the iterator object in the output.

The __iter__() method, however, should not be used directly because it’s considered a special type of hidden method signified by the double underscores on either side. (Such methods are called “dunder” methods for their double underscores.) Instead, you can use Python’s built-in iter() function that calls the __iter__() method of its given parameter for you.

Example:

l = [1,2,3,4,5]
iter(l)

Output:

<list_iterator object at 0x7ff6962eb190>

How the Iterator Object Works

The iterator object works by providing a method named __next__(), which returns the next item in the iterable with each call. The iterator object remembers the state where the last call to __next__() left off so that each time, the following item is returned. It is also a method you won’t normally call directly.

Example:

l = [1,2,3]
i = iter(l)
print(i.__next__())
print(i.__next__())
print(i.__next__())

Output:

1
2
3

Note that if you get a new iterator from an iterable, you’ll have an iterator that’s back at the beginning.

Example:

l = [1,2,3]

iter1 = iter(l)
print(iter1.__next__())
print(iter1.__next__())

iter2 = iter(l)
print(iter2.__next__())
print(iter2.__next__())

Output:

1
2
1
2

In the example above, iter1 is the first iterator. We call __next__() twice and print out 1 and 2. After that, we get another iterator iter2, and print out two more items. Those are 1 and 2 again since iter2 is a new iterator.

An iterator that is used beyond the last item in the iterable generates an error.

Example:

l = [1,2,3]
i = iter(l)
print(i.__next__())
print(i.__next__())
print(i.__next__())
print(i.__next__())  # No more items here so we'll get an error.

Output:

1
2
3
Traceback
...
StopIteration

Aside from providing the __next__() method, an Python iterator is also required to provide an __iter__() method. This means that a Python iterator is also an iterable, i.e. an object that can be iterated over. Standard iterators return themselves, which is the sensible thing to do.

Example:

l = [1,2,3]
iter1 = iter(l)
iter2 = iter(iter1)

print(iter1) 
print(iter2)

Output:

<list_iterator object at 0x7ff6962ebee0>
<list_iterator object at 0x7ff6962ebee0>

In the example above, iter1 is the iterator of list l. iter2 is assigned the iterator of iter1. As you can see from the printed object address, 0x7ff6962ebee0, they’re the same object. The iterator of iter2 is indeed itself.

Iterating Using for Loops

Python offers a simple and powerful way to loop through the elements of an iterable using a for loop. It handles the details of the iterator protocol, so you rarely have to.

The basic syntax is as follows:

for element in iterable:
    # Do something with the element

Here, element is a variable that represents each item in the iterable object during each iteration of the loop.

Example:

dog_breeds = ["corgie", "poodle", "labrador"]
for breed in dog_breeds:
    print(breed)

Output:

corgie
poodle
labrador

Using enumerate() to Iterate With an Index

Sometimes, you need both the value and its corresponding index while iterating. The enumerate() function simplifies this by returning an iterator that iterates over index-value pairs in the form of two-element tuples.

Example:

dog_breeds = ["corgie", "poodle", "labrador"]
i = enumerate(dog_breeds)
print(list(i))

Output:

[(0, 'corgie'), (1, 'poodle'), (2, 'labrador')]

In the example above, enumerate() receives a list argument, dog_breeds. The resulting iterator is then converted to a list and printed.

enumerate() in a for Loop

You can use enumerate() to get the indices and values separately with a for loop using the following syntax:

for index, value in enumerate(iterable):
    # Do something with the index and value

With this syntax, the tuple index-value pairs, which the iterator provides, are unpacked into the separate loop variables, index and value. You can give those any legal Python identifier name.

Example:

dog_breeds = ["corgie", "poodle", "labrador"]
for index, breed in enumerate(dog_breeds):
    print(f"The {breed} breed has index {index}.")

Output:

The corgie breed has index 0.
The poodle breed has index 1.
The labrador breed has index 2.

Iterators as Different Views Into an Object

An iterator can make an object viewed in a different way than it normally is.

A prominent example of this is a Python string. A string is often treated as a single chunk of text, for example when it’s printed with print().

But as an iterable, it’s see as a sequence of individual characters.

Example:

s = "Hello"
for ch in s:
    print(ch)

Output:

H
e
l
l
o

Creating Custom Iterables

You can create custom iterable objects by implementing an __iter__() method that returns an iterator. However, this topic is more advanced and will be covered in a different lesson.

Summary & Reference for Python Iterable

A Python iterable is a collection object that can be looped over, providing access to its elements one at a time.

Every iterable object is capable of returning an object called an iterator, which provides the ability to traverse the iterable’s data.

An iterable provides its iterator through a method called __iter__(). Implementing this method is the only requirement for being an iterable object.

l = [1,2,3,4,5]
l.__iter__()  # <list_iterator object at 0x7ff6962eb190>

Python’s built-in iter() function returns an iterable’s iterator by calling its __iter__() method. The __iter__() method is not normally used directly as it’s considered a special private method.

l = [1,2,3,4,5]
iter(l)  # <list_iterator object at 0x7ff6962eb190>

The iterator object works by providing a method named __next__(), which returns the next item in the iterable with each call. It’s also a method you won’t normally call directly.

l = [1,2,3]
i = iter(l)
print(i.__next__())  # 1
print(i.__next__())  # 2
print(i.__next__())  # 3

An iterator that is used beyond the last item in the iterable generates a StopIteration error.

A Python iterator is also required to implement an __iter__() method, which means that an iterator is an iterable. It normally implements it by returning itself.

Python offers way to loop through the elements of an iterable using a for loop. It handles the details of the iterator protocol, so you don’t have to.

dog_breeds = ["corgie", "poodle", "labrador"]
for breed in dog_breeds:
    print(breed)  

# corgie
# poodle
# labrador

The enumerate() function returns an iterator that iterates over index-value pairs in the form of two-element tuples

dog_breeds = ["corgie", "poodle", "labrador"]
i = enumerate(dog_breeds)
print(list(i))  # [(0, 'corgie'), (1, 'poodle'), (2, 'labrador')]

You can use enumerate() to get the indices and values separately with a for loop.

dog_breeds = ["corgie", "poodle", "labrador"]
for index, breed in enumerate(dog_breeds):
    print(f"The {breed} breed has index {index}.")

# The corgie breed has index 0.
# The poodle breed has index 1.
# The labrador breed has index 2.

An iterator can make an object viewed in a different way than it normally is. A prominent example of this is a Python string, which as an iterator, is seen as a sequence of individual characters.

s = "Hello"
for ch in s:
    print(ch)

# H
# e
# l
# l
# o