Loop like a Pro with Python Iterators

18 May 2023 / 33 minutes to read

Elena Daehnhardt

E-mail Twitter GitHub Pinterest LinkedIn

Table of Contents

Introduction

Iterators are one of the most powerful features of Python, allowing you to iterate over a sequence of values without having to keep track of an index. In this post, we’ll explore iterators in Python and learn how to use them effectively. We’ll dive into some basic examples of iterators and show you how to create your own. Finally, we’ll explore advanced techniques for using iterators and discuss some best practices for working with them.

Python Iterators

An iterator is an object that allows you to traverse a sequence of values. In Python, an iterator is an object that implements the iterator protocol, which consists of two methods: iter() and next(). The iter() method returns the iterator object itself, while the next() method returns the next value in the sequence. If there are no more values to return, the next() method should raise a StopIteration exception.

Here’s a simple example of using an iterator in Python:

my_list = [1, 2, 3, 4, 5]
my_iterator = iter(my_list)
next(my_iterator)

next(my_iterator)

next(my_iterator)

next(my_iterator)

next(my_iterator)

next(my_iterator)

Traceback (most recent call last):
  File "<input>", line 1, in <module>
StopIteration

In this example, we create a list my_list with five values. We then create an iterator object my_iterator by calling the iter() function and passing in my_list as an argument. We can then use the next() function to retrieve each value in turn. When no more values are retrieved, the StopIteration exception is raised.

Python provides several built-in objects that are iterable, including lists, tuples, strings, and dictionaries. You can also create your own iterable objects by implementing the iterator protocol.

Creating Iterators

Creating your own iterators in Python is relatively simple. You must define a class that implements the iterator protocol to create an iterator. Here’s an example of a simple iterator that returns the first 10 even numbers:

class EvenNumbers:
    def __init__(self):
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        self.current += 2
        if self.current <= 20:
            return self.current
        else:
            raise StopIteration

This code works for Python 3. In this example, we define a class called EvenNumbers that implements the iterator protocol. The init() method initializes the current value to 0. The iter() method returns the iterator object itself, and the next() method returns the following even number in the sequence. If the current value is greater than 20, the StopIteration exception is raised.

To use this iterator, we can create an instance of the EvenNumbers class and then iterate over it using a for loop:

even_numbers = EvenNumbers()
for number in even_numbers:
    print(number)

In this example, we create an instance of the EvenNumbers class called even_numbers. We then use a for loop to iterate over the even numbers returned by the iterator. As you can see, the output of the for loop matches the sequence of even numbers defined in the EvenNumbers class.

Advanced usage techniques

Now that we’ve covered the basics of iterators and how to create them, let’s look at some more advanced techniques for using iterators in Python 3.

itertools module

Using the itertools module: Python’s itertools module provides several functions for working with iterators, including count(), cycle(), and repeat(). These functions can generate infinite sequences, cycle through a sequence of values, or repeat a value a certain number of times. Here’s an example:

import itertools

print("Count from 1 to 5")
for number in itertools.count(start=1, step=1):
    if number > 5:
        break
    print(number)

print("Cycle through a sequence of values")
values = [1, 2, 3]
for value in itertools.cycle(values):
    print(value)
    if value == 3:
        break

print("Repeat a value 3 times")
for value in itertools.repeat('hello', times=3):
    print(value)

You may have noticed that we have checked the number and value variable and broken the code. We do it to avoid an infinitive loop.

The output:

Count from 1 to 5
1
2
3
4
5
Cycle through a sequence of values
1
2
3
Repeat a value 3 times
hello
hello
hello

In this example, we import the itertools module and use its count(), cycle(), and repeat() functions to generate sequences of numbers, cycle through a sequence of values, and repeat a value a certain number of times.

yield

In addition to iterators, Python also provides a way to create generators, which are functions that return an iterator. You can use the yield keyword to create a generator to return values one at a time. Here’s an example:

def even_numbers():
    for i in range(2, 21, 2):
        yield i

for number in even_numbers():
    print(number)

In this example, we defined a function called even_numbers() that uses the yield keyword to return each even number from 2 to 20. We can then use a for loop to iterate over the values the generator returns.

with

When working with files, it’s important to ensure that file handles are closed when they’re no longer needed. One way to do this is to use the with statement, which automatically closes the file handle when the block of code inside the with statement is complete. Here’s an example:

with open('file.txt', 'r') as f:
    for line in f:
        print(line)

In this example, we use the with statement to open the file file.txt for reading. We can then use a for loop to iterate over the lines in the file. The file handle is automatically closed when the code block inside the with statement is complete.

A file object is an iterator and can traverse the file once. You may reset the file cursor with .seek(0).

Are there any differences in iterator syntax between Python2.7 and Python3?

Yes, there are some differences in iterator syntax between Python 2.7 and Python 3.

Python 2.7 has two main types of iterators: lists and generators. Lists are created using the range() function or by explicitly listing the items in the list, while generators are made using the yield keyword inside a function. We will further return back to exploring generators in the dedicated section below.

Here’s an example of using a list iterator in Python 2.7:

# Create a list of even numbers
even_numbers = [2, 4, 6, 8, 10]

# Iterate over the list
for number in even_numbers:
    print(number)

The output:

And here’s an example of using a generator iterator in Python 2.7:

# Define a generator function that yields even numbers
def even_numbers():
    for i in range(2, 11, 2):
        yield i

# Iterate over the generator
for number in even_numbers():
    print(number)

The output is:

In Python 3, the syntax for creating iterators is largely the same as in Python 2.7, but there are some small differences. For example, the range() function in Python 3 returns an iterator instead of a list, so you can use it in a for loop without creating a list first. Here’s an example:

# Iterate over a range of numbers
for number in range(2, 11, 2):
    print(number)

In addition, the print() function in Python 3 requires parentheses around its arguments, while in Python 2.7, it does not. Here’s an example:

# Print a message in Python 2.7
print "Hello, world!"

# Print a message in Python 3
print("Hello, world!")

Overall, the syntax for creating and using iterators is similar in both Python 2.7 and Python 3. Still, there are some differences that you should be aware of if you’re transitioning from one version to the other.

Iterator alternatives

To avoid further confusion, let’s focus on Python 3. I have checked all the code below in Pycharm with Python 3.9 interpreter. In addition to iterators, Python offers several alternatives that you can use to iterate over sequences and perform operations on them. Here are some of the most useful alternatives to iterators in Python:

List comprehensions

List comprehensions provide a concise way of creating lists based on existing lists. Here’s an example:

# Create a list of numbers
numbers = [1, 2, 3, 4, 5]

# Use a list comprehension to create a new list of squares
squares = [x**2 for x in numbers]

# Print the squares
print(squares)

Output:

[1, 4, 9, 16, 25]

They can perform operations on elements of a sequence and filter them based on certain conditions. Here’s an example:

# Create a list of even numbers
even_numbers = [i for i in range(10) if i % 2 == 0]

# Print the list
print(even_numbers)

Output:

[0, 2, 4, 6, 8]

Generators

Generators are similar to iterators but are created using a function that uses the yield keyword to produce a sequence of values. The advantage of generators over iterators is that they can generate an infinite sequence of values. Here’s an example:

# Define a generator function that yields even numbers
def even_numbers():
    i = 0
    while True:
        yield i
        i += 2

# Create a generator object
even_numbers_generator = even_numbers()

# Print the first five even numbers
for i in range(5):
    print(next(even_numbers_generator))

Output:

Map and filter functions

The map() and filter() functions can be used to apply a function to each sequence element and filter the elements based on a certain condition, respectively. Here’s an example:

# Define a function that squares a number
def square(x):
    return x ** 2

# Create a list of numbers
numbers = [1, 2, 3, 4, 5]

# Use map() to square each number
squares = list(map(square, numbers))

# Print the squares
print(squares)

# Use filter() to get the even numbers
even_numbers = list(filter(lambda x: x % 2 == 0, numbers))

# Print the even numbers
print(even_numbers)

Output:

[1, 4, 9, 16, 25]
[2, 4]

Combinatorial iterators

Python also provides several combinatorial iterators that can be used to generate combinations, permutations, and other sequences of elements. Here are some examples:

# Generate all possible combinations of two letters
import itertools

letters = ['a', 'b', 'c']
combinations = list(itertools.combinations(letters, 2))
print(combinations)

[('a', 'b'), ('a', 'c'), ('b', 'c')]

# Generate all possible permutations of three numbers
numbers = [1, 2, 3]
permutations = list(itertools.permutations(numbers))
print(permutations)

[(1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1)]

# Generate all possible products of two lists
list1 = [1, 2, 3]
list2 = [4, 5, 6]
products = list(itertools.product(list1, list2))
print(products)

[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]

Zip function

The zip() function can combine multiple sequences into a single sequence of tuples. Here’s an example:

# Create two lists
names = ['Alice', 'Bob', 'Charlie']
ages = [25, 30, 35]

# Use zip() to combine the lists
name_age_pairs = list(zip(names, ages))

# Print the name-age pairs
print(name_age_pairs)

Output:

[('Alice', 25), ('Bob', 30), ('Charlie', 35)]

Sorted function

The sorted() function can be used to sort a sequence of elements based on certain criteria. Here’s an example:

# Create a list of tuples
students = [('Alice', 25), ('Bob', 30), ('Charlie', 35)]

# Sort the list based on the second element of each tuple
sorted_students = sorted(students, key=lambda x: x[1])

# Print the sorted list
print(sorted_students)

Output:

[('Alice', 25), ('Bob', 30), ('Charlie', 35)]

Reduce function

The reduce() function can be applied to pairs of elements in a sequence until a single value is obtained. Here’s an example:

# Import the reduce() function from the functools module
from functools import reduce

# Define a function that computes the product of two numbers
def multiply(x, y):
    return x * y

# Create a list of numbers
numbers = [1, 2, 3, 4, 5]

# Use reduce() to compute the product of the numbers
product = reduce(multiply, numbers)

# Print the product
print(product)

Output:

These alternatives to iterators provide more concise and readable ways of iterating over sequences and performing operations on them. They can be used to make your code more efficient and easier to understand.

Generator Expressions

Generator expressions provide a memory-efficient way of creating iterators. They are similar to list comprehensions, but instead of creating a list, they create a generator object that can be iterated over. Here’s an example:

# Create a generator expression that yields squares of numbers
squares = (x**2 for x in range(1, 6))

# Iterate over the squares
for square in squares:
    print(square)

Output:

Generator expressions are helpful when you need to iterate over an extensive sequence of elements but don’t want to store all of them in memory at once.

multiples = (x*x for x in range(1, 6))

According to PEP 289 – Generator Expressions, generator expressions help to consider computer memory when used instead of list comprehensions. Notice the usage of square brackets instead of round brackets. It’s when the little detail matters :)

# Create a list of squares of numbers
squares = [x**2 for x in range(1, 6)]

# Iterate over the squares
for square in squares:
    print(square)

Output:

These alternatives provide additional functionality and flexibility beyond what iterators offer. Understanding these tools and how to use them can make your code more concise, efficient, and readable.

A wave iterator

Suppose you want to generate a specific frequency and duration wave signal. You can create a Wave class representing the wave signal and define an iter() method that returns an iterator object.

Here’s an example implementation:

import math
import numpy as np

class Wave:
    def __init__(self, frequency, duration, amplitude=1.0, sample_rate=44100):
        self.frequency = frequency
        self.duration = duration
        self.amplitude = amplitude
        self.sample_rate = sample_rate
    
    def __iter__(self):
        return WaveIterator(self)
       
class WaveIterator:
    def __init__(self, wave):
        self.wave = wave
        self.index = 0
    
    def __next__(self):
        if self.index >= self.wave.sample_rate * self.wave.duration:
            raise StopIteration
        else:
            t = float(self.index) / self.wave.sample_rate
            value = self.wave.amplitude * math.sin(2.0 * math.pi * self.wave.frequency * t)
            self.index += 1
            return value

In this implementation, the Wave class contains information about the wave signal, including its frequency, duration, amplitude, and sample rate. The iter() method returns a WaveIterator object, which generates a sine wave signal with the specified frequency and duration.

wave = Wave(440, 1, 0.5)

To use this implementation, you can create an instance of the Wave class and iterate over it using a for loop.

for value in wave:
    print(value)

0.0
0.031324162089371846
0.06252526184726405
0.09348072041362668
0.12406892397186894
0.15416970152955017
0.18366479703068941
....

I have cut the massive floats’ output. Do you know why our values are within these ranges?

max(wave)

0.4999998731289437

min(wave)

-0.4999998731289436

That’s because we defined our wave amplitude=0.5.

To convert these values to 16-bit integers, we must multiply them by 32767 and cast them to int16. For this, we can use a list comprehension for creating an integers array with the help of a numpy array.

# Convert the float values to a 16-bit signed integer
wav_data = np.array([int(value * 32767) for value in wave], dtype=np.int16)
wav_data

array([    0,  1026,  2048, ..., -3063, -2048, -1026], dtype=int16)

Firstly, we add the float conversion into the int16 data type right into our Wave class and name it get_wav_data() as follows:

class Wave:
    def __init__(self, frequency, duration, amplitude=1.0, sample_rate=44100):
        self.frequency = frequency
        self.duration = duration
        self.amplitude = amplitude
        self.sample_rate = sample_rate
    
    def get_wav_data(self):    
        # Convert the float values to 16-bit signed integers
        return np.array([int(value * 32767) for value in self], dtype=np.int16)
        
    def __iter__(self):
        return WaveIterator(self)

Have you noticed how we used the self reference for the generated wave data behind the scenes?

That integer array we can further use for playing the WAV sound!

Let’s update our Wave class for playing the resulting wave with the sounddevice module, as usually, we beforehand install it should it not be installed yet with pip:

pip install sounddevice

Now, let’s play the sound:

# Import sounddevice and set the sample rate
import sounddevice as sd
sd.default.samplerate = 44100

# Create a 440Hz sine wave with a duration of 3 seconds and amplitude of 0.5
wave = Wave(440, 3, 0.5)

# Play the sound wave that we get with get_wav_data() method 
# using sounddevice package
sd.play(wave.get_wav_data(), blocking=True)

In this example, a Wave object is created to generate a 440Hz sine wave signal with a duration of 3 seconds and amplitude of 0.5. The wave signal is then converted into a wav int16 array and played with the sounddevice.

As your homework, should you like coding yourself, you can extend this code using a for loop, and the values are output to a file in WAV format. You can use the .to_bytes() method for this.

You can even create a melody by combining several waves of different frequencies and durations! This handy Note-To-Frequancy-Chart is your friend :)

Conclusion

In this post, we’ve covered the basics of Python iterators and their very successful alternatives, such as list comprehension, which uses more memory but is very useful in practice. We’ve also explored advanced techniques for working with iterators, including using the itertools module and creating generators with the yield keyword. Additionally, we have built our own iterator for generating waves and played them with the sounddevice. By mastering iterators in Python, we can create beautiful code, which is efficient and elegant, and become more effective Python programmers.

References

1. PEP 289 – Generator Expressions

2. Note-To-Frequancy-Chart

3. New Chat (chatGPT by OpenAI)

About Elena

Elena, a PhD in Computer Science, simplifies AI concepts and helps you use machine learning.

E-mail Twitter GitHub Pinterest LinkedIn

Citation

Elena Daehnhardt. (2023) 'Loop like a Pro with Python Iterators', daehnhardt.com, 18 May 2023. Available at: https://daehnhardt.com/blog/2023/05/18/python-iterators/

« Previous: The Token Way to GitHub Security All Posts Next: The Remarkable Evolution and Milestones of AI »

Loop like a Pro with Python Iterators

Introduction

Python Iterators

Creating Iterators

Advanced usage techniques

itertools module

yield

with

Python 2.7 and 3 differences related

Iterator alternatives

List comprehensions

Generators

Map and filter functions

Combinatorial iterators

Zip function

Sorted function

Reduce function

Generator Expressions

A wave iterator

Conclusion

References

Citation