Comprehensible Comprehensions

/ Python & Django Trainer / @treyhunner

On-site training for Django teams
Weekly Python Chat live webcast host
San Diego Python meetup co-organizer
Django Girls San Diego co-organizer
Python Software Foundation director

Assumptions

  • You know how for loops work in Python
  • You're not already a big fan of list comprehensions and generator expressions

Definitions

  • iterable: anything you can loop over
  • list comprehension: for making a new list from an iterable
  • generator: for making a "lazy" (pause-able) iterable
  • generator expression: used to make a generator

What are comprehensions?

List comprehensions are a special syntax for turning one list into another list

Turning Old Lists Into New Lists

for loops


numbers = [1, 2, 3, 4, 5]

# Print out square of odd numbers
for n in numbers:
    if n % 2 == 1:
        print(n ** 2)
          

Turning lists into lists


numbers = [1, 2, 3, 4, 5]

# Populate new list with square of odd numbers
squared_odds = []
for n in numbers:
    if n % 2 == 1:
        squared_odds.append(n ** 2)
          

Variables Are References


>>> old = [1, 2, 3]
>>> new = old
>>> new.pop()
>>> new
[1, 2]
>>> old
[1, 2]

>>> old = [1, 2, 3] >>> new = old[:-1] >>> new [1, 2] >>> old [1, 2, 3]

Turning lists into lists


squared_odds = []
for n in numbers:
    if n % 2 == 1:
        squared_odds.append(n ** 2)
          

With comprehensions


squared_odds = [n ** 2 for n in numbers if n % 2 == 1]
          

Breaking it down

squared_odds = []
for n in numbers:
    if n % 2 == 1:
        squared_odds.append(n ** 2)

squared_odds = [n ** 2 for n in numbers if n % 2 == 1]

Copy-pasting into a comprehension

You can always copy-paste

new_list = []
for x in old_list:
    if condition(x):
        new_list.append(operation(x))

new_list = [operation(x) for x in old_list if condition(x)]

Readability

But they're not readable


for n in numbers:
    if n % 2 == 1:
        squared_odds.append(n ** 2)
          

squared_odds = [n ** 2 for n in numbers if n % 2 == 1]
          

Breaking up the components

squared_odds = [n ** 2 for n in numbers if n % 2 == 1]
squared_odds = [
    n ** 2
    for n in numbers
    if n % 2 == 1
]

Breaking up the components


squared_odds = [n ** 2 for n in numbers if n % 2 == 1]
          

squared_odds = [
    n ** 2
    for n in numbers
    if n % 2 == 1
]
          

They can be more readable


squared_odds = []
for n in numbers:
    if n % 2 == 1:
        squared_odds.append(n ** 2)
          

squared_odds = [
    n ** 2
    for n in numbers
    if n % 2 == 1
]
          

They can be more readable


squared_odds = []
for n in squared_odds:
    if n % 2 == 1:
        squared_odds.append(n ** 2)
          

squared_odds = [
    n ** 2
    for n in numbers
    if n % 2 == 1
]
          

But wait... there's more!

With Filtering

squared_odds = []
for n in numbers:
    if n % 2 == 1:
        squared_odds.append(n ** 2)

squared_odds = [
    n ** 2
    for n in numbers
    if n % 2 == 1
]

Without Filtering

squared_numbers = []
for n in numbers:
    squared_numbers.append(n ** 2)

squared_numbers = [
    n ** 2
    for n in numbers
]

With Multiple Loops

flattened = []
for row in matrix:
    for item in row:
        flattened.append(item)
flattened = [
    item
    for row in matrix
    for item in row
]

Set comprehensions


numbers = {1, 2, 3, 4, 5}

squared_odds = set()
for n in numbers:
    if n % 2 == 1:
        squared_odds.add(n ** 2)
          

numbers = {1, 2, 3, 4, 5}

squared_odds = {
    n ** 2
    for n in numbers
    if n % 2 == 1
}
          

Dictionary comprehensions


from string import ascii_lowercase

letter_positions = {}
for n, letter in enumerate(ascii_lowercase, start=1):
    letter_positions[letter] = n
          

from string import ascii_lowercase

letter_positions = {
    letter: n
    for n, letter in enumerate(ascii_lowercase, start=1)
}
          

Examples

Modifying each item in a list


lowercased_words = [word.lower() for word in words]
          

Filtering down a list of strings


def get_anagrams(target_word, words):
    return [
        candidate_word
        for candidate_word in words
        if is_anagram(candidate_word, target_word)
    ]


def is_anagram(word1, word2):
    """Return True if words contain same letters."""
    return sorted(word1) == sorted(word2)
          

Modifying a list of lists


negative_matrix = []
for row in matrix:
    new_row = []
    for n in row:
        new_row.append(-n)
    negative_matrix.append(new_row)
          

negative_matrix = []
for row in matrix:
    negative_matrix.append([-n for n in row])
          

negative_matrix = [
    [-n for n in row]
    for row in matrix
]
          

Generator Comprehensions

Generator Expressions

Generator Expressions

  • Generator expressions are to generators as list comprehensions are to lists
  • Generators are lazy single-use iterables
  • If you're making a list to loop over it exactly once, use a generator expression instead of a list comprehension

Generator Expression Syntax


>>> numbers = range(1_000_000)
>>> squared_numbers = (n ** 2 for n in numbers)
>>> squared_numbers
<generator object <genexpr> at 0x7f129b187780>
          

Generators are weird


>>> numbers = [1, 2, 3, 4, 5]
>>> squared_numbers = (n ** 2 for n in numbers)
>>> len(squared_numbers)
TypeError: object of type 'generator' has no len()
>>> squared_numbers[0]
TypeError: 'generator' object is not subscriptable
>>> list(squared_numbers)
[1, 4, 9, 16, 25]
>>> list(squared_numbers)
[]

          

Generators are lazy


>>> from itertools import islice
>>> numbers = range(1_000_000_000)
>>> squared_numbers = (n ** 2 for n in numbers)
>>> next(squared_numbers)
0
>>> next(squared_numbers)
1
>>> next(squared_numbers)
4
>>> next(squared_numbers)
9
>>> next(squared_numbers)
16
          

Generators are single use iterables


>>> numbers = [1, 2, 3, 4, 5]
>>> squared_numbers = (n ** 2 for n in numbers)
>>> sum(squared_numbers) 333332833333500000
>>> sum(squared_numbers) 0
>>> cubed_numbers = (n ** 3 for n in numbers) >>> list(cubed_numbers) [1, 8, 27, 64, 125]
>>> list(cubed_numbers) []

Loop over them right away


>>> numbers = range(1_000_000)
>>> sum((n ** 2 for n in numbers)) 333332833333500000
>>> sum(n ** 2 for n in numbers) 333332833333500000

Generator Examples

str.join


def translate(sentence):
    translation = []
    for w in sentence.split():
        translation.append(DICTIONARY[w])
    return " ".join(translation)
          

def translate(sentence):
    translation = [
        DICTIONARY[w]
        for w in sentence.split()
    ]
    return " ".join(translation)
          

def translate(sentence):
    return " ".join(
        DICTIONARY[w]
        for w in sentence.split()
    )
          

sum


sum_of_squares = 0
for n in numbers:
    sum_of_squares += n**2
          

squares = []
for n in numbers:
    squares.append(n**2)
sum_of_squares = sum(squares)
          

sum_of_squares = sum(n**2 for n in numbers)
          

When not to use comprehensions

Don't overdo it

  • Do not call functions with side effects in a comprehension
  • Don't use comprehensions except for making lists
  • List comprehensions should only be used for turning one list (or iterable) into another list

Don't abuse comprehensions


>>> [print(n**2) for n in range(10) if n % 2 == 1]
1
9
25
49
81
[None, None, None, None, None]
          


color_ratios = {}
for color, ratio in zip(colors, ratios):
    color_ratios[color] = ratio
          

color_ratios = {
    color: ratio
    for (color, ratio) in zip(colors, ratios)
}
          

color_ratios = dict(zip(colors, ratios))
          


with open(filename) as my_file:
    lines = [line for line in my_file]
          

with open(filename) as my_file:
    lines = list(my_file)
          

with open(filename) as my_file:
    lines = my_file.readlines()
          

Cheat Sheet

Building up a new list

new_list = []
for item in old_list:
    new_list.append(operation(item))

new_list = [
    operation(item)
    for item in old_list
]

Building up a list while filtering

new_list = []
for item in old_list:
    if condition(item):
        new_list.append(operation(item))

new_list = [
    operation(item)
    for item in old_list
    if condition(item)
]

Building up a new dictionary

new_dict = {}
for key, value in old_dict.items():
    if condition(key, value):
        new_dict[change1(key)] = change2(value)

new_dict = {
    change1(key): change2(value)
    for key, value in old_dict.items()
    if condition(key, value)
}

Looping Deeply

new_list = []
for inner_list in outer_list:
    for item in inner_list:
        new_list.append(item)
new_list = [
    item
    for inner_list in outer_list
    for item in inner_list
]

Not filtering or changing anything?

new_list = [
    item
    for n in old_list
]

new_list = list(old_list)

Making a list and immediately looping

new_list = []
for item in old_list:
    new_list.append(operation(item))
some_result = some_operation(new_list)

some_result = some_operation(
    operation(item)
    for item in old_list
)

Remember

  • Use comprehensions for turning one iterable into another
  • Copy-paste your way from a for loop to a comprehension
  • Use generators when you'll be looping over your new iterable only one time
  • Break comprehensions and generator expressions over multiple lines to improve readability

Things to look up later

Trey Hunner / @treyhunner

Python & Django Trainer for hire
http://truthful.technology