Python Oddities Explained

/ @treyhunner

Python Morsels
Truthful Technology

I teach Python to teams

My students show me very interesting code

That's odd. What's going on here?

This talk supports Python 3 only

Ask me what oddities Python 2 has

Scope Scares


>>> x = 0
>>> numbers = [1, 1, 2, 3, 5, 8]
>>> for x in numbers:
...     y = x**2
...
>>> y
64
>>> x
8
          

>>> x = 0
>>> numbers = [1, 1, 2, 3, 5, 8]
>>> squares = [x**2 for x in numbers]
>>> x
0  # In Python 2 this was 8
          

>>> NUMBERS = [1, 2, 3]
>>> def add_numbers(nums):
...     NUMBERS += nums
...
>>> add_numbers([4, 5, 6])
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in add_numbers
UnboundLocalError: local variable 'NUMBERS' referenced before assignment
          

>>> NUMBERS = [1, 2, 3]
>>> def add_numbers(nums):
...     NUMBERS = NUMBERS + nums
...
>>> add_numbers([4, 5, 6])
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in add_numbers
UnboundLocalError: local variable 'NUMBERS' referenced before assignment
          

>>> NUMBERS = [1, 2, 3]
>>> def set_numbers(nums):
...     print(NUMBERS)
...     NUMBERS = nums
...
>>> set_numbers([4, 5, 6])
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in set_numbers
UnboundLocalError: local variable 'NUMBERS' referenced before assignment
          

>>> NUMBERS = [1, 2, 3]
>>> def set_numbers(nums):
...     NUMBERS = nums
...     print(NUMBERS)
...
>>> set_numbers([4, 5, 6])
[4, 5, 6]
>>> NUMBERS
[1, 2, 3]
          

>>> NUMBERS = [1, 2, 3]
>>> def add_numbers(nums):
...     NUMBERS.extend(nums)
...     print(NUMBERS)
...
>>> add_numbers([4, 5, 6])
[1, 2, 3, 4, 5, 6]
>>> NUMBERS
[1, 2, 3, 4, 5, 6]
          

Takeaways

  • Reading global variables is perfectly fine
  • But don't try to assign to them from a local scope
  • List comprehensions have their own scope, loops don't
  • Python has no block-level scoping
  • Scope matters with assignment, not with mutation

Mesmerizing Mutations


>>> numbers = [1, 2, 3]
>>> numbers2 = numbers
>>> numbers2.append(4)
>>> numbers2
[1, 2, 3, 4]
>>> numbers
[1, 2, 3, 4]
>>> id(numbers)
139670455619848
>>> id(numbers2)
139670455619848
>>> numbers is numbers2
True
          

>>> x = ([1], [4])
>>> x[0].append(2)
>>> x
([1, 2], [4])
>>> y = x[0]
>>> y.append(3)
>>> x
([1, 2, 3], [4])
          

>>> x = []
>>> x.append(x)
>>> x
[[...]]
          

Takeaways

Devious Ducks


>>> duck_list = ['mallard']
>>> duck_list += ('eider', 'Pekin')
>>> duck_list
['mallard', 'eider', 'Pekin']
>>> duck_tuple = ('Muscovy', 'mandarin')
>>> duck_tuple += ['ruddy', 'Indian Runner']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate tuple (not "list") to tuple
          

>>> ducks = ('Muscovy', 'Indian Runner')
>>> ducks += ('mallard', 'ruddy')
>>> ducks += ['eider', 'Pekin']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate tuple (not "list") to tuple
          

>>> things = []
>>> things += "duck"
>>> things
['d', 'u', 'c', 'k']
>>> things.extend(' quack')
>>> things
['d', 'u', 'c', 'k', ' ', 'q', 'u', 'a', 'c', 'k']
          

>>> " ".join(["hello", "there!"])
'hello there!'
>>> " ".join(("hello", "there!"))
'hello there!'
>>> " ".join("hello")
'h e l l o'
>>> dict(('ab', 'cd'))
{'a': 'b', 'c': 'd'}
          

Duck Typing

Duck Typing

  • If it looks like a duck and quacks like a duck, it's a duck
  • Don't type check: rely on specific behaviors instead
  • "Iterable" and "callable" describe behaviors (not types)

Takeaways

  • The list extend method accepts any iterable
  • The list += operator also accepts any iterable of strings
  • The tuple += operator only accepts another tuple
  • Python thinks in terms of behaviors, not types
  • Embrace duck typing by thinking in terms of behaviors: iterable, callable, sequence, mapping, hashable
  • More on iterables: Loop Better: a deeper look at iteration

Intriguing In-Place Additions


>>> a = b = (1, 2)
>>> a += (3, 4)
>>> a
(1, 2, 3, 4)
>>> b
(1, 2)
>>> a = a + (3, 4)

          

>>> a = b = [1, 2]
>>> a += [3, 4]
>>> a
[1, 2, 3, 4]
>>> b
[1, 2, 3, 4]
          

>>> a = b = [0]
>>> a += b
>>> a, b
([0, 0], [0, 0])
>>> a = b = [0]
>>> a = a + b
>>> a, b
([0, 0, 0, 0], [0, 0])
          

>>> a = b = [1, 2]
>>> a += [3, 4]
>>> a, b
([1, 2, 3, 4], [1, 2, 3, 4])
>>> c = d = "Python"
>>> c += "!!!"  # Same as: c = c + "!!!"
>>> c, d
('Python!!!', 'Python')
>>> c is d
False
>>> a is b
True
          

>>> x = ([1, 2],)
>>> x[0] += [3, 4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> x
([1, 2, 3, 4],)
          

>>> x = ([1, 2],)
>>> x[0] = x[0].__iadd__([3, 4])  # x[0] += [3, 4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> x[0]
[1, 2, 3, 4]
>>> x[0].__iadd__([5])
[1, 2, 3, 4, 5]
>>> x[0] = [1, 2, 3, 4, 5]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
          

Takeaways

  • In-place addition (+=) and other "augmented assignment" operations perform assignments
  • In-place addition calls the __iadd__ method which allows the object to mutate itself if it chooses to do so
  • In-place addition do the addition "in-place" whenever possible

Closing thoughts

  • Understanding how variables, mutability, data structures, operators work is important
  • It's important to understand how your language "thinks"
  • If it looks like a bug, it might just be a misunderstanding
  • Found something odd? Try to learn from it!

#pythonoddity

Recommended resources at

trey.io/oddities

Trey Hunner
Python Team Trainer


>>> '8' < 8
False
>>> '8' < 9
False
>>> '8' < 99999999999999999999999999999999
False
>>> [8] > 8
True
>>> [8] < '8'
True
>>> sorted([str(type(8)), str(type('8')), str(type([8]))])
["<type 'int'>", "<type 'list'>", "<type 'str'>"]
>>> 8 < [8] < '8'
True
          

>>> class A:
...     @property
...     def x(self):
...         return 'x value'
...
>>> a = A()
>>> a.x
'x value'
>>> a.x = 4
>>> a.x
4


          

>>> class Cipher:
...     alphabet = 'abcdefghijklmnopqrstuvwxyz'
...     letter_a = alphabet[0]
...     letters = {
...         letter: ord(letter) - ord(letter_a)
...         for letter in alphabet
...     }
...
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 6, in Cipher
  File "", line 6, in 
NameError: name 'letter_a' is not defined
          

>>> class Cipher:
...     alphabet = 'abcdefghijklmnopqrstuvwxyz'
...     letter_a = alphabet[0]
...     letters = dict([
...         (letter, ord(letter) - ord(letter_a))
...         for letter in alphabet
...     ])
...
>>> Cipher.letters['a']
0



          

Scope Silliness


>>> numbers = [1]
>>> class A:
...     numbers = [2]
...     m = 3
...     squares = [n**2 for n in numbers]
...
>>> numbers[0], A.numbers[0], A.m, A.squares[0]
(1, 2, 3, 4)
          

>>> class A:
...     numbers = [2]
...     for n in numbers:
...         print(n**2)
...
4
>>> A.numbers
[2]
>>> A.n
2
          

>>> class A:
...     numbers = [1, 2]
...     m = 3
...     squares = [n**m for n in numbers]
...     print(squares[0])
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in A
  File "<stdin>", line 3, in <listcomp>
NameError: name 'm' is not defined
          

>>> class A:
...     numbers = [1, 2]
...     def hi(): print(numbers)
...     hi()
...
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 4, in A
  File "", line 3, in hi
NameError: name 'numbers' is not defined
          

>>> VOWELS = "aeiou"
>>> class B:
...     VOWELS += "y"
...
>>> B.VOWELS
'aeiouy'
>>> VOWELS
'aeiou'
          

>>> NUMBERS = [1, 2, 3]
>>> class MyFavoriteClass:
...     NUMBERS += [4, 5, 6]
...
>>> MyFavoriteClass.NUMBERS
[1, 2, 3, 4, 5, 6]
>>> NUMBERS
[1, 2, 3, 4, 5, 6]
          

Illogical Identities


>>> x = 999
>>> y = 999
>>> x == y
True
>>> x is y
False
>>> x = 4
>>> y = 4
>>> x is y
True
          

lots_of_numbers = range(-1000, 1000)
the_same_numbers = range(-1000, 1000)
same_numbers = (
    i
    for i, j in zip(lots_of_numbers, the_same_numbers)
    if i is j
)
print(*same_numbers, sep=", ")
          

-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64,
65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81,
82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98,
99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112
113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125,
126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138,
139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151,
152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164,
165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190,
191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203,
204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216,
217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229,
230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242,
243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256
          

>>> x = 'a' * 10
>>> y = 'a' * 10
>>> x is y
True
>>> x = 'a' * 1000
>>> y = 'a' * 1000
>>> x is y
False
          

>>> x = 'a' * 1000
>>> y = 'a' * 1000
>>> m = x[:10]
>>> n = y[:10]
>>> m is n
False
>>> m, n
('aaaaaaaaaa', 'aaaaaaaaaa')
>>> m == n
True
          

So... should you ever use the is operator?

Yes, when you care about identity, not just equality


def from_node(node):
    node_list = []
    while node is not None:
        node_list.append(node)
        node = node.next
    return node_list

def is_iterator(iterable):
    """Return True if the given iterable is also an iterator."""
    return iter(iterable) is iterable
          

Takeaways

  • Equality and identity are different
  • Checking for equality is very common
  • Checking for identity is not nearly as common
  • Python includes optimizations that make for strange identity patterns with strings and numbers
  • Checking for identity when you need equality causes bugs

Mysterious Multiplication


>>> [0] * 3
[0, 0, 0]
>>> [[0] * 3] * 3
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]
>>> matrix = [[0] * 3] * 3
>>> matrix[1][1] = 1
>>> matrix
[[0, 1, 0], [0, 1, 0], [0, 1, 0]]
          

>>> row = [0, 0, 0]
>>> matrix = [row, row, row]
>>> row[1] = 2
>>> matrix
[[0, 2, 0], [0, 2, 0], [0, 2, 0]]
>>> matrix[1][1] = 1
>>> matrix
[[0, 1, 0], [0, 1, 0], [0, 1, 0]]
>>> matrix[0] is matrix[1] is row
True
>>> matrix = [row] * 3

          

Miscellaneous Mishaps


>>> 'Python' in 'Python' in 'Python'
True
>>> 'Python' in 'Python'
True
>>> True in 'Python'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'in <string>' requires string as left operand, not bool
>>> 'Python' in 'Python' in 'Python'
True
>>> ('Python' in 'Python') in 'Python'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'in <string>' requires string as left operand, not bool
>>> 'Python' in ['Python'] in [['Python']]
True
>>> x = 25
>>> 10 < x < 20
False
          

>>> """multiline
... strings"""
'multiline\nstrings'
>>> """""strings with
... five quotes around them!"""""
'""strings with\nfive quotes around them!'
>>> """""strings with five
... quotes around them!"""""
'""strings with five\nquotes around them!'