# Comprehensible Comprehensions

/ Python & Django Trainer / @treyhunner Weekly Python Chat live webcast host San Diego Python meetup co-organizer Django Girls San Diego co-organizer Python Software Foundation director

## Assumptions

• You know how `for` loops work in Python
• You're not already a big fan of list comprehensions and generator expressions

## Definitions

• iterable: anything you can loop over
• list comprehension: for making a new list from an iterable
• generator: for making a "lazy" (pause-able) iterable
• generator expression: used to make a generator

## What are comprehensions?

List comprehensions are a special syntax for turning one list into another list

## `for` loops

``````
numbers = [1, 2, 3, 4, 5]

# Print out square of odd numbers
for n in numbers:
if n % 2 == 1:
print(n ** 2)
``````

## Turning lists into lists

``````
numbers = [1, 2, 3, 4, 5]

# Populate new list with square of odd numbers
squared_odds = []
for n in numbers:
if n % 2 == 1:
squared_odds.append(n ** 2)
``````

## Variables Are References

``````
>>> old = [1, 2, 3]
>>> new = old
>>> new.pop()
>>> new
[1, 2]
>>> old
[1, 2]

>>> old = [1, 2, 3]
>>> new = old[:-1]
>>> new
[1, 2]
>>> old
[1, 2, 3]
``````

## Turning lists into lists

``````
squared_odds = []
for n in numbers:
if n % 2 == 1:
squared_odds.append(n ** 2)
``````

## With comprehensions

``````
squared_odds = [n ** 2 for n in numbers if n % 2 == 1]
``````

## Breaking it down

``````squared_odds = []
for n in numbers:
if n % 2 == 1:
squared_odds.append(n ** 2)``````

``squared_odds = [n ** 2 for n in numbers if n % 2 == 1]``

## Copy-pasting into a comprehension ## You can always copy-paste

``````new_list = []
for x in old_list:
if condition(x):
new_list.append(operation(x))``````

``new_list = [operation(x) for x in old_list if condition(x)]``

``````
for n in numbers:
if n % 2 == 1:
squared_odds.append(n ** 2)
``````
``````
squared_odds = [n ** 2 for n in numbers if n % 2 == 1]
``````

## Breaking up the components

``squared_odds = [n ** 2 for n in numbers if n % 2 == 1]``
``````squared_odds = [
n ** 2
for n in numbers
if n % 2 == 1
]``````

## Breaking up the components

``````
squared_odds = [n ** 2 for n in numbers if n % 2 == 1]
``````
``````
squared_odds = [
n ** 2
for n in numbers
if n % 2 == 1
]
``````

## They can be more readable

``````
squared_odds = []
for n in numbers:
if n % 2 == 1:
squared_odds.append(n ** 2)
``````
``````
squared_odds = [
n ** 2
for n in numbers
if n % 2 == 1
]
``````

## They can be more readable

``````
squared_odds = []
for n in squared_odds:
if n % 2 == 1:
squared_odds.append(n ** 2)
``````
``````
squared_odds = [
n ** 2
for n in numbers
if n % 2 == 1
]
``````

## With Filtering

``````squared_odds = []
for n in numbers:
if n % 2 == 1:
squared_odds.append(n ** 2)``````

``````squared_odds = [
n ** 2
for n in numbers
if n % 2 == 1
]``````

## Without Filtering

``````squared_numbers = []
for n in numbers:
squared_numbers.append(n ** 2)``````

``````squared_numbers = [
n ** 2
for n in numbers
]``````

## With Multiple Loops

``````flattened = []
for row in matrix:
for item in row:
flattened.append(item)``````
``````flattened = [
item
for row in matrix
for item in row
]``````

## Set comprehensions

``````
numbers = {1, 2, 3, 4, 5}

squared_odds = set()
for n in numbers:
if n % 2 == 1:
``````
``````
numbers = {1, 2, 3, 4, 5}

squared_odds = {
n ** 2
for n in numbers
if n % 2 == 1
}
``````

## Dictionary comprehensions

``````
from string import ascii_lowercase

letter_positions = {}
for n, letter in enumerate(ascii_lowercase, start=1):
letter_positions[letter] = n
``````
``````
from string import ascii_lowercase

letter_positions = {
letter: n
for n, letter in enumerate(ascii_lowercase, start=1)
}
``````

## Modifying each item in a list

``````
lowercased_words = [word.lower() for word in words]
``````

## Filtering down a list of strings

``````
def get_anagrams(target_word, words):
return [
candidate_word
for candidate_word in words
if is_anagram(candidate_word, target_word)
]

def is_anagram(word1, word2):
"""Return True if words contain same letters."""
return sorted(word1) == sorted(word2)
``````

## Modifying a list of lists

``````
negative_matrix = []
for row in matrix:
new_row = []
for n in row:
new_row.append(-n)
negative_matrix.append(new_row)
``````
``````
negative_matrix = []
for row in matrix:
negative_matrix.append([-n for n in row])
``````
``````
negative_matrix = [
[-n for n in row]
for row in matrix
]
``````

## Generator Expressions

• Generator expressions are to generators as list comprehensions are to lists
• Generators are lazy single-use iterables
• If you're making a list to loop over it exactly once, use a generator expression instead of a list comprehension

## Generator Expression Syntax

``````
>>> numbers = range(1_000_000)
>>> squared_numbers = (n ** 2 for n in numbers)
>>> squared_numbers
<generator object <genexpr> at 0x7f129b187780>
``````

## Generators are weird

``````
>>> numbers = [1, 2, 3, 4, 5]
>>> squared_numbers = (n ** 2 for n in numbers)
>>> len(squared_numbers)
TypeError: object of type 'generator' has no len()
>>> squared_numbers
TypeError: 'generator' object is not subscriptable
>>> list(squared_numbers)
[1, 4, 9, 16, 25]
>>> list(squared_numbers)
[]

``````

## Generators are lazy

``````
>>> from itertools import islice
>>> numbers = range(1_000_000_000)
>>> squared_numbers = (n ** 2 for n in numbers)
>>> next(squared_numbers)
0
>>> next(squared_numbers)
1
>>> next(squared_numbers)
4
>>> next(squared_numbers)
9
>>> next(squared_numbers)
16
``````

## Generators are single use iterables

``````
>>> numbers = [1, 2, 3, 4, 5]
>>> squared_numbers = (n ** 2 for n in numbers)
>>> sum(squared_numbers)
333332833333500000
>>> sum(squared_numbers)
0>>> cubed_numbers = (n ** 3 for n in numbers)
>>> list(cubed_numbers)
[1, 8, 27, 64, 125]>>> list(cubed_numbers)
[]
``````

## Loop over them right away

``````
>>> numbers = range(1_000_000)
>>> sum((n ** 2 for n in numbers))
333332833333500000
>>> sum(n ** 2 for n in numbers)
333332833333500000

``````

## `str.join`

``````
def translate(sentence):
translation = []
for w in sentence.split():
translation.append(DICTIONARY[w])
return " ".join(translation)
``````
``````
def translate(sentence):
translation = [
DICTIONARY[w]
for w in sentence.split()
]
return " ".join(translation)
``````
``````
def translate(sentence):
return " ".join(
DICTIONARY[w]
for w in sentence.split()
)
``````

## `sum`

``````
sum_of_squares = 0
for n in numbers:
sum_of_squares += n**2
``````
``````
squares = []
for n in numbers:
squares.append(n**2)
sum_of_squares = sum(squares)
``````
``````
sum_of_squares = sum(n**2 for n in numbers)
``````

## Don't overdo it

• Do not call functions with side effects in a comprehension
• Don't use comprehensions except for making lists
• List comprehensions should only be used for turning one list (or iterable) into another list

## Don't abuse comprehensions

``````
>>> [print(n**2) for n in range(10) if n % 2 == 1]
1
9
25
49
81
[None, None, None, None, None]
``````

``````
color_ratios = {}
for color, ratio in zip(colors, ratios):
color_ratios[color] = ratio
``````
``````
color_ratios = {
color: ratio
for (color, ratio) in zip(colors, ratios)
}
``````
``````
color_ratios = dict(zip(colors, ratios))
``````

``````
with open(filename) as my_file:
lines = [line for line in my_file]
``````
``````
with open(filename) as my_file:
lines = list(my_file)
``````
``````
with open(filename) as my_file:
``````

## Building up a new list

``````new_list = []
for item in old_list:
new_list.append(operation(item))``````

``````new_list = [
operation(item)
for item in old_list
]``````

## Building up a list while filtering

``````new_list = []
for item in old_list:
if condition(item):
new_list.append(operation(item))``````

``````new_list = [
operation(item)
for item in old_list
if condition(item)
]``````

## Building up a new dictionary

``````new_dict = {}
for key, value in old_dict.items():
if condition(key, value):
new_dict[change1(key)] = change2(value)``````

``````new_dict = {
change1(key): change2(value)
for key, value in old_dict.items()
if condition(key, value)
}``````

## Looping Deeply

``````new_list = []
for inner_list in outer_list:
for item in inner_list:
new_list.append(item)``````
``````new_list = [
item
for inner_list in outer_list
for item in inner_list
]``````

## Not filtering or changing anything?

``````new_list = [
item
for n in old_list
]``````

``new_list = list(old_list)``

## Making a list and immediately looping

``````new_list = []
for item in old_list:
new_list.append(operation(item))
some_result = some_operation(new_list)``````

``````some_result = some_operation(
operation(item)
for item in old_list
)``````

## Remember

• Use comprehensions for turning one iterable into another
• Copy-paste your way from a for loop to a comprehension
• Use generators when you'll be looping over your new iterable only one time
• Break comprehensions and generator expressions over multiple lines to improve readability

### Things to look up later

Python & Django Trainer for hire
http://truthful.technology 