Variables and Objects

It's pointers all the way down

Trey Hunner

trey.io/news — a weekly Python tip

Trey Hunner

Variables & Objects

Mental Models

Gotchas

Changing `first` doesn't always change `second`


>>> first = [2, 3]
>>> second = first
>>> first.append(5)
>>> second
[2, 3, 5]


>>> first = [2, 3]
>>> second = first
>>> first = [2, 3, 5]
>>> second
[2, 3]


>>> def square_all(numbers):
...     for i, n in enumerate(numbers):
...         numbers[i] = n ** 2
...     return numbers
...
>>> numbers = [2, 1, 3, 4, 7]
>>> square_all(numbers)
[4, 1, 9, 16, 49]
>>> numbers
[4, 1, 9, 16, 49]


>>> matrix = [[0] * 3] * 3
>>> matrix
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]
>>> matrix[1][1] = 9
>>> matrix
[[0, 9, 0], [0, 9, 0], [0, 9, 0]]


>>> colors = ["purple", "blue", "green"]
>>> groups = dict.fromkeys(colors, [])
>>> groups
{'purple': [], 'blue': [], 'green': []}
>>> groups["purple"].append("duck")
>>> groups
{'purple': ['duck'], 'blue': ['duck'], 'green': ['duck']}


>>> class TodoList:
...     def __init__(self, tasks=[]):
...         self.tasks = tasks
...
>>> mon = TodoList()
>>> tue = TodoList()
>>> mon.tasks.append("Work on talk")
>>> tue.tasks
['Work on talk']

Gotchas

1. second = first
2. for i, n in enumerate(numbers): numbers[i] = n ** 2
3. matrix = [[0] * 3] * 3
4. groups = dict.fromkeys(colors, [])
5. def __init__(self, tasks=[]): self.tasks = tasks

Buckets


>>> first = []
>>> second = first
>>> second.append(9)
>>> second
[9]
>>> first
[9]

This mental model breaks down

- If you think of variables as like **(CLICK)** buckets that contain objects - Then when we assign a variable to an empty list **(CLICK)**, that variable would **contain** that list **(CLICK)** - If we then assigned another variable to the first one, **(CLICK)** the list would be **copied** into a new bucket **(CLICK)** - But if we then append `9` to our second list **(CLICK)**, which changes **(CLICK)** that list - The first list shouldn't change... **(CLICK)** - ... But it *does* **(CLICK)** change! - In Python, this mental model **(CLICK)** doesn't work... - Variables in Python are *not* like **buckets that contain objects** - And assignment statements don't copy anything

Pointers 🤔


>>> first = []
>>> second = first
>>> second.append(9)
>>> second
[9]
>>> first
[9]
>>> rows = [second]
>>>

- In Python, variables **(CLICK)** and objects **(CLICK)** live in two different places - When we assign a variable **(CLICK)** we draw an arrow **(CLICK)** from the variable to the object - Then when we assign **(CLICK)** another variable to the first one, we draw another arrow **(CLICK)** from the second variable to the *same exact* object - So when we **(CLICK)** modify our second list **(CLICK)**... - The second list changes **(CLICK)** and so does the first **(CLICK)** because *they're the same list* - This is the mental model I usually recommend for thinking about how Python's variables and objects work - I call this the "pointers" mental model **(CLICK)** because it looks like variables *point* to objects - ... - But "pointer" is also **(CLICK)** a technical term that has a specific meaning - Python's pointers...

➡️

“But they're not like pointers in C”

binding	🪢
reference	📖
pointer	👉

“Change”

mutation


>>> first = [2, 3]
>>> second = first
>>> first.append(5)
>>> second
[2, 3, 5]

assignment


>>> first = [2, 3]
>>> second = first
>>> first = [2, 3, 5]
>>> second
[2, 3]

“change”


>>> first = [2, 1, 3, 4]
>>> second = first
>>> first.append(7)
>>> second
[2, 1, 3, 4, 7]
>>> first = [100, 200, 300]
>>> second
[2, 1, 3, 4, 7]
>>>

Remember: variables point to objects

The 2 Types of Change

Mutations change an object
Assignments change a variable

“We changed `x`”

Changed the variable?


>>> x = [2, 3]
>>> x = [2, 3, 5]

assignment

Changed the object?


>>> x = [2, 3]
>>> x.append(5)

mutation

Gotchas

1. second = first
2. for i, n in enumerate(numbers): numbers[i] = n ** 2
3. matrix = [[0] * 3] * 3
4. groups = dict.fromkeys(colors, [])
5. def __init__(self, tasks=[]): self.tasks = tasks

2 types of

“are they the same?”


>>> first == second  # Equality
True
>>> first is second  # Identity
True

Identity: the exact same object

Equality: an equivalent object

Equality is about objects

Identity is about pointers


>>> first == second  # Are these objects equal to the other?
True
>>> first is second  # Do these variables point to the same object?
True

Equality:
the usual way to ask
“are they the same?”


value is None


value == another_value

Assignments happen


>>> e = 2.7
>>> from math import e
>>> for e in range(3):
...     print(e)
...
0
1
2
>>> def e(): return 2.7
...
>>> class e: pass
...
>>> e
<class '__main__.e'>

Function arguments are also assignments


def square_all(numbers):
    for i, n in enumerate(numbers):
        numbers[i] = n ** 2
    return numbers


>>> numbers = [2, 1, 3, 4, 7]
>>> square_all(numbers)
[4, 1, 9, 16, 49]
>>> numbers
[4, 1, 9, 16, 49]


def square_all(numbers):
    for i, n in enumerate(numbers):
        numbers[i] = n ** 2  # DON'T DO THIS
    return numbers


def square_all(numbers):
    squares = []
    for n in numbers:
        squares.append(n ** 2)
    return squares


def square_all(numbers):
    return [
        n ** 2
        for n in numbers
    ]

Assignments that mutate


def square_all(numbers):
    for i, n in enumerate(numbers):
        numbers[i] = n ** 2
    return numbers


>>> some_list[0] = 8
>>> some_object.attribute = 4

Gotchas

1. second = first
2. for i, n in enumerate(numbers): numbers[i] = n ** 2
3. matrix = [[0] * 3] * 3
4. groups = dict.fromkeys(colors, [])
5. def __init__(self, tasks=[]): self.tasks = tasks

Objects

Variables

Data structures don't contain data*

*for some definitions of “data”

Data structures don't contain objects

Pointers


>>> first = []
>>> second = first
>>> second.append(9)
>>> second
[9]
>>> first
[9]
>>> rows = [second]
>>>
>>>

Pointers


>>> first = []
>>> second = first
>>> second.append(9)
>>> second
[9]
>>> first
[9]
>>> rows = [second]
>>>

“a list of lists”

“a list of references to lists”

Containment

... is a lie


x in y

“y contains x”


>>> matrix = [[0] * 3] * 3
>>> matrix
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]
>>> matrix[1][1] = 9
>>> matrix
[[0, 9, 0], [0, 9, 0], [0, 9, 0]]


>>> matrix = [[0] * 3] * 3
>>> matrix
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]
>>> matrix[1][1] = 9
>>> matrix
[[0, 9, 0], [0, 9, 0], [0, 9, 0]]


>>> matrix = [[0] * 3 for _ in range(3)]
>>> matrix[1][1] = 9
>>> matrix
[[0, 0, 0], [0, 9, 0], [0, 0, 0]]

Gotchas

1. second = first
2. for i, n in enumerate(numbers): numbers[i] = n ** 2
3. matrix = [[0] * 3] * 3
4. groups = dict.fromkeys(colors, [])
5. def __init__(self, tasks=[]): self.tasks = tasks


>>> colors = ["purple", "blue", "green"]
>>> groups = dict.fromkeys(colors, [])
>>> groups["purple"].append("duck")
>>> groups
{'purple': ['duck'], 'blue': ['duck'], 'green': ['duck']}


>>> colors = ["purple", "blue", "green"]
>>> groups = {color: [] for color in colors}
>>> groups["purple"].append("duck")
>>> groups
{'purple': ['duck'], 'blue': [], 'green': []}

Gotchas

1. second = first
2. for i, n in enumerate(numbers): numbers[i] = n ** 2
3. matrix = [[0] * 3] * 3
4. groups = dict.fromkeys(colors, [])
5. def __init__(self, tasks=[]): self.tasks = tasks


>>> class TodoList:
...     def __init__(self, tasks=[]):
...         self.tasks = tasks
...
>>> mon = TodoList()
>>> tue = TodoList()
>>> mon.tasks.append("Work on talk")
>>> tue.tasks
['Work on talk']
>>> TodoList.__init__.__defaults__[0]
['Work on talk']

- Remember our `TodoList` problem with the mutable default argument? - When we create **(CLICK)** two instances without passing a `tasks` list... - What happens is that **(CLICK)** both instances' `.tasks` attributes point to the **same** default list object - So when we append to one **(CLICK)**... - We're mutating **(CLICK)** the one list that both instances share - And that means **(CLICK)** both instances show the change - In fact, **(CLICK)** we can look at the actual default value stored on the method itself and see that it changed as well - That **(CLICK)** empty list expression in our function definition was evaluated just *once*, when the method was defined, and the resulting object was attached to the method - So mutating it through one instance changes the default value itself - To fix this, we could use the `list` constructor to copy the default list...


>>> class TodoList:
...     def __init__(self, tasks=[]):
...         self.tasks = list(tasks)
...
>>> mon = TodoList()
>>> tue = TodoList()
>>> mon.tasks.append("Work on talk")
>>> tue.tasks
[]


>>> class TodoList:
...     def __init__(self, tasks=[]):
...         self.tasks = list(tasks)
...
>>> default_todos = ["Reflect on last week"]
>>> mon = TodoList(default_todos)
>>> tue = TodoList(default_todos)
>>> mon.tasks.append("Work on talk")
>>> mon.tasks
['Reflect on last week', 'Work on talk']
>>> tue.tasks
['Reflect on last week']
>>> wed = TodoList({"Here is", "a set", "of tasks"})

Gotchas

1. second = first
2. for i, n in enumerate(numbers): numbers[i] = n ** 2
3. matrix = [[0] * 3] * 3
4. groups = dict.fromkeys(colors, [])
5. def __init__(self, tasks=[]): self.tasks = tasks

Recap

Variables

Variables don't contain objects
Variables contain pointers to objects
Assignments point a variable to an object
Assignments do not make a copy

So...

second = first is a problem for mutable objects
But that's fine for immutable objects like strings

Functions

Function arguments are just like every other variable
Function calls assign an object to each function parameter
Function calls do not copy any objects

So...

Functions should not mutate passed-in objects, unless the caller expects it
Functions often implicitly copy passed-in objects using comprehensions, slicing, list(...), etc.

Objects

Objects can't contain other objects
Objects can only contain pointers to other objects

So...

Avoid storing multiple references to the same mutable object
Using dict.fromkeys with a mutable default value will cause problems
Be mindful of whether your initializer methods store references to a passed-in object or copy those objects

It's pointers all the way down 🐢

Variables store object references
Objects store object references
Variables/objects do not "contain" objects
Function arguments are variables
Attributes act just like variables

We didn't discuss

Variable scope in Python
Hashability and its relation to immutability
CPython optimizations that affect identity
Direct value storage of array, data frames, etc.
Tuples containing mutable objects

https://trey.io/pycascades26

Names, Objects, and Plummeting From The Cliff - Brandon Rhodes

Facts and Myths about Names and Values - Ned Batchelder

But... why?

Imagine an alternative...

Would every assignment statement make a copy?

Would storing a reference to an object ever be possible?

"reference assignments" and "copying assignments" 😬

- *Why* does Python work this way? - Wouldn't it be easier if variables and data structures contained objects and assignments copied objects? - Well... **(CLICK)** - Imagine if every assignment made a copy of an object **(CLICK)** - Copying objects all the time takes time and it takes memory - But also, how would we ever model parent-child relationships **(CLICK)** or any kind of relationship that involved storing a *reference* to an object instead of a copy of an object? - Would we need to make a distinction **(CLICK)** between assignments that copied and assignments that made a reference... - Other programming languages have done that before... - And... **(CLICK)** that requires adding another layer of complexity to the language - ... - Python's way isn't necessarily the *right* way... but *most days* I really appreciate the trade-offs that Python made around this topic

This thing is bad

Compared to what alternative?

Everything is a trade-off

Resources https://trey.io/pycascades26

Thank you

Trey Hunner
Python Team Trainer
[email protected]

https://www.psy.ritsumei.ac.jp/akitaoka/saishin72e.html

#7c7c7c

https://www.psy.ritsumei.ac.jp/akitaoka/saishin72e.html

Mental Models

"All models are wrong,
but some models are useful." — George E. P. Box

Assignments can mutate

Assignments change the object a reference refers to
Objects can contain references to other object
Assigning to an object attribute/index mutates the object

Augmented Assignments

+=, -=, *=, etc.

Are these assignments?

Are these mutations?

YES

In-place addition


>>> a = [2, 1, 3]
>>> b = a
>>> b += [4, 7, 11]
>>> b
[2, 1, 3, 4, 7, 11]
>>> a
[2, 1, 3, 4, 7, 11]


>>> name = "North Bay Python"
>>> name += " 2025"  # name = name + "2025"
>>> name
'North Bay Python 2025'

- But wait... doesn't `+=` work on strings? **(click)** - Strings is Python are immutable... - Which means you can't *change* a string object - But strings work with `+=`... **(click)** - If `+=` is an *in-place* addition, how does it work on strings? **(click)** - Well, augmented assignments *can* perform a mutation, but they don't *have* to... it's up to the object that they're operating on - If an object doesn't specially handle `+=` then **(click)** Python falls back to using the `+` operator and a plain old assignment - The distinction between assignments and mutations isn't always very clear... augmented assignments do perform an assignment, but they can also mutate. - But this assignment or mutation fuzziness comes up in other ways too **(CLICK)**

Changing tuples

Tuples cannot be mutated*

* For some definitions of "mutate"

Tuples can contain lists


>>> result = (True, [2, 1, 3])
>>> result[1].append(4)
>>> result
(True, [2, 1, 3, 4])

Immutability is shallow


>>> result = (True, [2, 1, 3])
>>> result2 = (True, [2, 1, 3])
>>> result == result2
True
>>> result[1].append(4)
>>> result
(True, [2, 1, 3, 4])
>>> result == result2
False

Augmented assignments in tuples


>>> result = (True, [2, 1, 3, 4])
>>> result[1] += [7, 11, 18]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    result[1] += [7, 11, 18]
    ~~~~~~^^^
TypeError: 'tuple' object does not support item assignment
>>> result
(True, [2, 1, 3, 4, 7, 11, 18])

Sticky Notes


>>> first = []
>>> second = first
>>> second.append(9)
>>> second
[9]
>>> first
[9]
>>>

- Let's mentally model variables as **(CLICK)** sticky notes on objects - When we assign a variable to an empty list **(CLICK)**, we attach that variable name to the object **(CLICK)** - Then when we assign **(CLICK)** another variable to the first one, we simply **(CLICK)** attach **another name** to the *same* object - So when we change **(CLICK)** that second list, the second one changes **(CLICK)** but the first one changes also **(CLICK)**... because there **is no** first and second list - it's all just *one* list! - This is the mental model that you'll find used in many books on Python and in many introductory Python courses - And this mental model works **pretty well**! - But, as we'll see later, it *does* have limits

Sticky Notes


>>> first = []
>>> second = first
>>> second.append(9)
>>> second
[9]
>>> first
[9]
>>> rows = [second]
>>> lists = rows


>>> first = []
>>> second = first
>>> second.append(9)
>>> second
[9]
>>> first
[9]
>>> rows = [second]
>>> lists = rows

- First we'll take another look at our sticky note mental model - What should happen **(CLICK)** if we put our `second` list inside another list?... - Should we **add another sticky note** that says **(CLICK)** which list it's in and which index it's at? - Also... what if we point **(CLICK)** another variable to that new outer list? - Would we need to add yet *another* note with another name and another index? **(CLICK)** - This *does* work, but it seems a bit complicated - This sticky note mental model works *well* for explaining how **variables** in Python work... - But it doesn't work well for explaining how **objects** work - Let's take a look at how our pointers mental model handles this...

One more puzzle


>>> x = []
>>> x.append(x)
>>> x
???


>>> x = []
>>> x.append(x)
>>> x
[[...]]


>>> x = []
>>> x.append(x)
>>> x
[[...]]
>>> x[0] is x
True
>>> x in x
True

Variables and Objects

It's pointers all the way down

Trey Hunner

Variables & Objects

Mental Models

Gotchas

Changing first doesn't always change second

Gotchas

Buckets

This mental model breaks down

Pointers 🤔

➡️

“But they're not like pointers in C”

“Change”

mutation

assignment

“change”

The 2 Types of Change

“We changed x”

assignment

mutation

Gotchas

2 types of

“are they the same?”

Equality is about objects

Identity is about pointers

Equality:the usual way to ask“are they the same?”

Assignments happen

Function arguments are also assignments

Assignments that mutate

Gotchas

Objects

Variables

Data structures don't contain data*

Data structures don't contain objects

Pointers

Pointers

“a list of lists”

“a list of references to lists”

Containment

... is a lie

“y contains x”

Gotchas

Gotchas

Gotchas

Recap

Variables

So...

Functions

So...

Objects

So...

It's pointers all the way down 🐢

We didn't discuss

https://trey.io/pycascades26

But... why?

Imagine an alternative...

This thing is bad

Compared to what alternative?

Everything is a trade-off

Resources https://trey.io/pycascades26

Thank you

Mental Models

Assignments can mutate

Augmented Assignments

YES

In-place addition

Changing tuples

Tuples cannot be mutated*

Tuples can contain lists

Immutability is shallow

Augmented assignments in tuples

Sticky Notes

Sticky Notes

One more puzzle

Changing `first` doesn't always change `second`

“We changed `x`”

Equality:
the usual way to ask
“are they the same?”