Python Oddities Explained

/ @treyhunner

Python Morsels
Truthful Technology

I teach Python to teams

My students show me very interesting code

That's odd. What's going on here?

Thank you to the people I've taught!

This talk supports Python 3 only

Ask me what oddities Python 2 has

Scope Scares


>>> x = 0
>>> numbers = [1, 1, 2, 3, 5, 8]
>>> for x in numbers:
...     y = x**2
...
>>> y
64
>>> x
8
          

>>> x = 0
>>> numbers = [1, 1, 2, 3, 5, 8]
>>> squares = [x**2 for x in numbers]
>>> x
0
          

>>> NUMBERS = [1, 2, 3]
>>> def add_numbers(*nums):
...     NUMBERS += nums
...
>>> add_numbers([4, 5, 6])
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in add_numbers
UnboundLocalError: local variable 'NUMBERS' referenced before assignment
          

>>> NUMBERS = [1, 2, 3]
>>> def set_numbers(*nums):
...     print(NUMBERS)
...     NUMBERS = nums
...
>>> set_numbers([4, 5, 6])
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in set_numbers
UnboundLocalError: local variable 'NUMBERS' referenced before assignment
          

>>> NUMBERS = [1, 2, 3]
>>> def set_numbers(*nums):
...     NUMBERS = nums
...     print(NUMBERS)
...
>>> set_numbers([4, 5, 6])
[4, 5, 6]
>>> NUMBERS
[1, 2, 3]
          

>>> NUMBERS = [1, 2, 3]
>>> def add_numbers(*nums):
...     NUMBERS.extend(nums)
...     print(NUMBERS)
...
>>> add_numbers([4, 5, 6])
[1, 2, 3, 4, 5, 6]
>>> NUMBERS
[1, 2, 3, 4, 5, 6]
          

Takeaways

  • Reading global variables is perfectly fine
  • But don't try to assign to them from a local scope
  • List comprehensions have their own scope
  • Scope matters with assignment, not with mutation

Teenage Mutable Ninja Turtles


>>> numbers = [1, 2, 3]
>>> numbers2 = numbers
>>> numbers2.append(4)
>>> numbers2
[1, 2, 3, 4]
>>> numbers
[1, 2, 3, 4]
>>> id(numbers)
139670455619848
>>> id(numbers2)
139670455619848
>>> numbers is numbers2
True
          

>>> x = ([1], [4])
>>> x[0].append(2)
>>> x
([1, 2], [4])
>>> y = x[0]
>>> y.append(3)
>>> x
([1, 2, 3], [4])
          

>>> x = []
>>> x.append(x)
>>> x
[[...]]
>>> [[...]]
[[Ellipsis]]
          

Takeaways

Intriguing In-Place Additions


>>> a = b = (1, 2)
>>> a += (3, 4)
>>> a
(1, 2, 3, 4)
>>> b
(1, 2)
>>> a = a + (3, 4)

          

>>> a = b = [1, 2]
>>> a += [3, 4]
>>> a
[1, 2, 3, 4]
>>> b
[1, 2, 3, 4]
          

>>> a = b = [0]
>>> a += b
>>> a, b
([0, 0], [0, 0])
>>> a = b = [0]
>>> a = a + b
>>> a, b
([0, 0, 0, 0], [0, 0])
          

>>> x = ([1, 2],)
>>> x[0] += [3, 4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> x
([1, 2, 3, 4],)
          

>>> x = ([1, 2],)
>>> x[0] = x[0].__iadd__([3, 4])  # x[0] += [3, 4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> x[0]
[1, 2, 3, 4]
>>> x[0].__iadd__([5])
[1, 2, 3, 4, 5]
>>> x[0] = [1, 2, 3, 4, 5]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
          

Takeaways

  • In-place addition (+=) and other "augmented assignment" operations perform assignments
  • In-place addition calls the __iadd__ method which allows the object to mutate itself if it chooses to do so
  • In-place addition do the addition "in-place" whenever possible

Closing thoughts

  • Understanding how variables, mutability, and data structures work is important
  • If it looks like a bug, it might just be a misunderstanding
  • Found something odd? Try to learn from it!

#pythonoddity

Need help leveling up
your team's Python skills?

Trey Hunner
Python & Django Team Trainer

Contact me: trey@truthful.technology


>>> '8' < 8
False
>>> '8' < 9
False
>>> '8' < 99999999999999999999999999999999
False
>>> [8] > 8
True
>>> [8] < '8'
True
>>> sorted([str(type(8)), str(type('8')), str(type([8]))])
["<type 'int'>", "<type 'list'>", "<type 'str'>"]
>>> 8 < [8] < '8'
True
          

>>> class A:
...     @property
...     def x(self):
...         return 'x value'
...
>>> a = A()
>>> a.x
'x value'
>>> a.x = 4
>>> a.x
4


          

>>> class Cipher:
...     alphabet = 'abcdefghijklmnopqrstuvwxyz'
...     letter_a = alphabet[0]
...     letters = {
...         letter: ord(letter) - ord(letter_a)
...         for letter in alphabet
...     }
...
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 6, in Cipher
  File "", line 6, in 
NameError: name 'letter_a' is not defined
          

>>> class Cipher:
...     alphabet = 'abcdefghijklmnopqrstuvwxyz'
...     letter_a = alphabet[0]
...     letters = dict([
...         (letter, ord(letter) - ord(letter_a))
...         for letter in alphabet
...     ])
...
>>> Cipher.letters['a']
0



          

Scope Silliness


>>> numbers = [1]
>>> class A:
...     numbers = [2]
...     m = 3
...     squares = [n**2 for n in numbers]
...
>>> numbers[0], A.numbers[0], A.m, A.squares[0]
(1, 2, 3, 4)
          

>>> class A:
...     numbers = [2]
...     for n in numbers:
...         print(n**2)
...
4
>>> A.numbers
[2]
>>> A.n
2
          

>>> class A:
...     numbers = [1, 2]
...     m = 3
...     squares = [n**m for n in numbers]
...     print(squares[0])
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in A
  File "<stdin>", line 3, in <listcomp>
NameError: name 'm' is not defined
          

>>> class A:
...     numbers = [1, 2]
...     def hi(): print(numbers)
...     hi()
...
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 4, in A
  File "", line 3, in hi
NameError: name 'numbers' is not defined
          

>>> VOWELS = "aeiou"
>>> class B:
...     VOWELS += "y"
...
>>> B.VOWELS
'aeiouy'
>>> VOWELS
'aeiou'
          

>>> NUMBERS = [1, 2, 3]
>>> class MyFavoriteClass:
...     NUMBERS += [4, 5, 6]
...
>>> MyFavoriteClass.NUMBERS
[1, 2, 3, 4, 5, 6]
>>> NUMBERS
[1, 2, 3, 4, 5, 6]
          

Mysterious Multiplication


>>> [0] * 3
[0, 0, 0]
>>> [[0] * 3] * 3
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]
>>> matrix = [[0] * 3] * 3
>>> matrix[1][1] = 1
>>> matrix
[[0, 1, 0], [0, 1, 0], [0, 1, 0]]
          

>>> row = [0, 0, 0]
>>> matrix = [row, row, row]
>>> row[1] = 2
>>> matrix
[[0, 2, 0], [0, 2, 0], [0, 2, 0]]
>>> matrix[1][1] = 1
>>> matrix
[[0, 1, 0], [0, 1, 0], [0, 1, 0]]
>>> matrix[0] is matrix[1] is row
True
>>> matrix = [row] * 3

          

Tricky Types


>>> x = [1, 2]
>>> x += (3, 4)
>>> x
[1, 2, 3, 4]
>>> y = (1, 2)
>>> y += [3, 4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate tuple (not "list") to tuple
          

>>> x = []
>>> x += "hey"
>>> x
['h', 'e', 'y']
>>> x.extend('iterable')
>>> x
['h', 'e', 'y', 'i', 't', 'e', 'r', 'a', 'b', 'l', 'e']
          

Miscellaneous Mishaps


>>> 'Python' in 'Python' in 'Python'
True
>>> 'Python' in 'Python'
True
>>> True in 'Python'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'in <string>' requires string as left operand, not bool
>>> 'Python' in 'Python' in 'Python'
True
>>> ('Python' in 'Python') in 'Python'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'in <string>' requires string as left operand, not bool
>>> 'Python' in ['Python'] in [['Python']]
True
>>> x = 25
>>> 10 < x < 20
False
          

>>> """multiline
... strings"""
'multiline\nstrings'
>>> """""strings with
... five quotes around them!"""""
'""strings with\nfive quotes around them!'
>>> """""strings with five
... quotes around them!"""""
'""strings with five\nquotes around them!'
          

Errors of Equality

Equality and Identity


>>> a = [1, 2]
>>> b = [1, 2]
>>> a == b
True
>>> a is b
False
          

>>> a = [1, 2]
>>> b = [1, 2]
>>> a is b
False
>>> id(a), id(b)
(140678342682600, 140678342683752)
>>> c = a
>>> id(a), id(c)
(140678342682600, 140678342682600)
>>> a == c
True
>>> a is c
True
          

>>> x = 999
>>> y = 999
>>> x == y
True
>>> x is y
False
>>> x = 4
>>> y = 4
>>> x is y
True
          

lots_of_numbers = range(-1000, 1000)
the_same_numbers = range(-1000, 1000)
same_numbers = (
    i
    for i, j in zip(lots_of_numbers, the_same_numbers)
    if i is j
)
print(*same_numbers, sep=", ")
          

-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64,
65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81,
82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98,
99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112
113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125,
126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138,
139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151,
152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164,
165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190,
191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203,
204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216,
217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229,
230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242,
243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256
          

>>> x = 'a' * 10
>>> y = 'a' * 10
>>> x is y
True
>>> x = 'a' * 1000
>>> y = 'a' * 1000
>>> x is y
False
          

>>> x = 'a' * 1000
>>> y = 'a' * 1000
>>> m = x[:10]
>>> n = y[:10]
>>> m is n
False
>>> m, n
('aaaaaaaaaa', 'aaaaaaaaaa')
>>> m == n
True
          

So... should you ever use the is operator?

Yes, when you care about identity, not just equality


def from_node(node):
    node_list = []
    while node is not None:
        node_list.append(node)
        node = node.next
    return node_list

def is_iterator(iterable):
    """Return True if the given iterable is also an iterator."""
    return iter(iterable) is iterable
          

Takeaways

  • Equality and identity are different
  • Checking for equality is very common
  • Checking for identity is not nearly as common
  • Python includes optimizations that make for strange identity patterns with strings and numbers
  • Checking for identity when you need equality causes bugs