A Guide to Python’s Secret Superpower: Magic Methods
Python has a secret superpower with a similarly stupendous name: Magic Methods. These methods can fundamentally change the way you code with Python classes and introduce code that seems ✨ magical ✨ to handle complex logic. They’re more powerful than list comprehensions and more exciting than any new PEP8 linter.
Today, we’ll be talking about a few things:
- What magic methods are
- Some simple introductory magic method usage
- How to programmatically manage class properties
- How to overwrite operator symbol functionality
- How to make your classes iterable
We also have a cheat sheet for utilizing these magic methods quicker within your projects:
Without further ado, let’s dive in!
What are magic methods?
Magic methods are methods that Python calls on your behalf in specific circumstances. These methods are named in a particular way to quickly distinguish them from other Python methods: they’re preceded and followed by two underscores.
class Speaker: # This is a magic method def __init__(self): print("Hello, world!") # This will call __init__ and print "Hello, world!" instance = Speaker()Code language: Python (python)
This is why magic methods also called “dunder methods,” which is a shorthand for “Double underscore methods.”
In the above code you can see what I’m talking about: Python calls the
__init__ dunder method on your behalf when a new class instance is created.
This barely scratches the surface when it comes to the power that magic methods provide. Let’s dive into their usage.
Simple magic method usage
If you’ve ever created a class, you’re likely familiar with the following method:
__init__(self, …args) –
It’s probably the best-known magic method, Python’s init acts as a class constructor. You can use this to pass initial arguments to a Python class.
For example, take the following:
class Speaker: message = "" def __init__(self, val): self.message = val def sayIt(self): print(self.message) instance = Speaker("Hello, world!") instance.sayIt()Code language: Python (python)
Here, whenever the
Speaker class is initialized, it will assign
self.message to the passed value. We’re then able to use a custom “sayIt” method that utilizes
Clean up class instantiation with
In addition to a class initializer, there’s also a class deletion handler:
This method will run any time you call
del on a class instance. This is particularly useful whenever you have an I/O operation in the constructor in order to cleanup said I/O operations.
import os class Test: def __init__(self): f = open("temp.csv", "w") f.write("data,more data,testing") f.close() def __del__(self): os.remove('temp.csv') print("Cleanup done!") firstItem = Test() del firstItemCode language: Python (python)
This type is cleanup is integral to ensure your applications are deterministic on each run, which in turn increases general application stability. After all, if you leave remnants of your cache, they’re likely to be picked up by subsequent runs and cause havoc with your application logic.
How to programmatically manage class properties
Stuff like class constructors and cleanup are par for the course when it comes to class management. Ready for the weird stuff?
What about declaring attributes that don’t exist?
__getattr__ has you covered.
__getattr__(self, key) –
property doesn’t exist)
Simply check what the lookup key is (in this case with the
__name property) and return a value if you want to create a new property programmatically:
class Test: number = 1 def __getattr__(self, __name: str): if __name == "string": return "Test" pass test = Test() print(test.number) # Will print `1` print(test.string) # Will print `"Test"`Code language: Python (python)
There also exists a slightly different getattribute built-in:
__getattribute__(self, key) –
instance.property (regardless of if
class Test: number = 1 def __getattribute__(self, __name: str): if __name == "string": return "Test" pass test = Test() print(test.number) # `None` print(test.string) # `"Test"`Code language: Python (python)
Notice how instead of
test.number returning the expected
1 value, it returns a
This is because while
__getattr__ will resolve the existing variables and fallback to the special method when nothing is found,
__getattribute__ runs first and doesn’t fall back to existing values in the class instance.
In order to have
__getattribute__ to have the same behavior as
__getattr__, we need to explicitly tell Python not to get stuck in the
__getattribute__ trap we’ve set up.
To do this, we can call
class Test: number = 1 def __getattribute__(self, __name: str): """ We need a "try/except" here, otherwise it will fail during lookup of an invalid key """ try: existingVal = super().__getattribute__(__name) if existingVal: return existingVal except: if __name == "string": return "Test" pass test = Test() print(test.number) # Will print `1` print(test.string) # Will print `"Test"`Code language: Python (python)
Customize class property dictionary lookup
__getattribute__ both work wonders for adding in keys programmatically, there’s a problem with that method. When usingthe
dir built-in method, it won’t show the new keys.
Let’s show you what I’m talking about with a code sample. Take the following:
class Test: number = 1 def __getattr__(self, __name: str): if __name == "string": return "Test" pass test = Test() print(dir(test))Code language: Python (python)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'number']Code language: Python (python)
This list of keys includes other magic methods, which muddies the output a bit for our needs. Let’s filter those out with the following logic:
def simpledir(obj): return [x for x in dir(obj) if not x.startswith('__')]Code language: Python (python)
Now, when we run
simpledir(test), we only see:
['number']Code language: Python (python)
But where is our
’string’ field? It doesn’t show up.
This is because while we’ve told Python how to look up the overwritten values, we’ve not told Python which keys we’ve added.
To do this, we can use the
__dir__ magic method.
class Test: number = 1 def __dir__(self): originalList = super().__dir__() originalList.append("string") return originalList def __getattr__(self, __name: str): if __name == "string": return "Test" passCode language: Python (python)
dir behavior like this will now enable us to treat our dynamic properties as if they existed normally. Now all we’re missing is a way to set values to those properties…
Set programmatically created keys
While we’re now telling Python which keys we’re programmatically creating and how to lookup the value of those keys, we’re not telling Python how to store those values.
Take the following code:
class Test: number = 1 def __getattr__(self, __name: str): print("Test"); if __name == "string": return "Test" pass test = Test() test.string = "Value" print(test.string)Code language: Python (python)
Here, we might expect the
print(test.string) to output "Test" as well as "Value", since
getattr should be called. But, if we look at the log, we only see the following:
"Value"Code language: Python (python)
This is because, once we assign
test.string, it no longer calls
getattr the way we expect it to.
To solve this problem, we need to use the
__setattr__ magic method to “listen” for property assignment.
__setattr__(self, key, val) –
instance.property = newVal
class Test: updateCount = 0 valid = 1 def __setattr__(self, key, val): super().__setattr__("updateCount", self.updateCount + 1) pass test = Test() test.valid = 12 print(test.updateCount)Code language: Python (python)
> Notice our usage of
super().__setattr__. We need to do this similarly to how we utilized the
super() method in
self.updateCount += 1 would trigger an infinite loop of calls to
Clean up programmatic property instanciation
Just as we can hook into the setting and getting behavior of an attribute, we can also hook into the
del behavior of an attribute using
For example, what if we wanted to create a class that acted like a dictionary. For each key created in this dictionary we’d want to automatically create a temporary file. Then, on cleanup (using
del), let’s remove that file with
__delattr__(self, key) –
import os class FileDictionary: def __setattr__(self, filename, val): f = open(filename, "w") f.write(val) f.close() def __delattr__(self, filename): os.remove(filename) fileDictionary = FileDictionary() fileDictionary.README = "Hello" del fileDictionary.READMECode language: Python (python)
Remember, if you’re not cleaning up your side effects, it may cause havoc with future usage of your app. This is why it’s so important to add in
__delattr__ when relevant.
Convert programatic lookups to index properties
In our most recent
FileDictionary example, we created a class called “FileDictionary”, but then accessed the child values with the dot accessor:
fileDictionary.README = "Hello"Code language: Python (python)
However, this dot syntax causes some minor headache: it’s not consistent with how you access properties from a dictionary. The reason we’re not using the standard dictionary syntax is because if you do the following:
fileDictionary['README'] = "Hello"Code language: Python (python)
We would quickly get an error from Python:
> TypeError: ‘FileDictionary’ object is not subscriptable
To solve this problem, we need to migrate away from
__setattr__, which only supports dot notation, to
__setitem__, which only supports the dictionary-style notation.
__getitem__(self, key) –
__setitem__(self, key, val) –
instance[property] = newVal
__delitem__(self, key) –
import os class FileDictionary: def __setitem__(self, filename, val): f = open(filename, "w") f.write(val) f.close() def __delitem__(self, filename): os.remove(filename) fileDictionary = FileDictionary() fileDictionary['README'] = "Hello" del fileDictionary['README']Code language: Python (python)
As a wonderful side effect, you’re now able to add in a file extension to the
fileDictionry. This is because bracket notation supports non-ASCII symbols while the dot notation does not.
fileDictionary['README.md'] = "Hello" del fileDictionary['README.md']Code language: Python (python)
How to replace operator symbol functionality with custom logic
There’s nothing more Pythonic than the simplicity of using simple mathematical symbols to represent mathematic actions.
After all, what could more clearly represent the sum of two numbers than:
sum = 2 + 2Code language: Python (python)
Meanwhile, if we have a wrapper around a number:
sum = numInstance.getNumber() + numInstance.getNumber()Code language: Python (python)
It gets a bit harder to read through.
What if we could utilize those symbols to handle this custom class logic for us?
sum = numInstance + numInstance;Code language: Python (python)
Luckily we can!
For example, here’s how we can make the
+ symbol run custom logic:
__add__(self, other) –
instance + other
class Test: __internal = 0 def __init__(self, val): self.__internal = val def __add__(self, other): return self.__internal + other.__internal firstItem = Test(12) secondItem = Test(31) # This will call "__add__" instead of the traditional arithmetic operation print(firstItem + secondItem)Code language: Python (python)
There’s also other math symbols you can overwrite:
__sub__(self, other) –
instance - other
__mul__(self, other) –
instance * other
Manage comparison symbol behavior
Addition, subtraction, and multiplication aren’t the only usages for operator overloading, however. We can also modify the comparison operators in Python to run custom logic.
Let’s say we want to check if two strings match, regardless of casing:
__eq__(self, other) –
instance == other
class Test(): str = "" def __init__(self, val): self.str = val def __eq__(self, other): return self.str.lower() == other.str.lower() firstItem = Test("AB") secondItem = Test("ab") print(firstItem == secondItem)Code language: Python (python)
You can also have different logic for
__ne__(self, other) –
instance != other
However, if you don’t provide a
__ne__, butdoprovide a
__eq__, Python will simply negate the
__eq__ logic on your behalf when
instance != other is called.
There’s also a slew of magic methods for customizing other comparison operators:
__lt__(self, other) –
instance < other
__gt__(self, other) –
instance > other
__le__(self, other) –
instance <= other
__ge__(self, other) –
instance >= other
Overwrite a class’s type casting logic
Python, like any other programming language, has the concept of data types. Similarly, you’re able to convert easily from any of those types to another type using built-in methods of type-casting data.
For example, if you call
bool() on a string, it will cast the truthy value to a Boolean.
What if you could customize the behavior of the
bool() method? You see where we’re going with this…
from os.path import exists class File: file_path = "" def __init__(self, file_path): self.file_path = file_path # This method should return `True` or `False` def __bool__(self): return exists(self.file_path) file = File("temp.txt") # Will return True or False depending on if file exists print(bool(file))Code language: Python (python)
There’s also other type casts logic you can customize:
How to make your classes iterable
Let’s say that we’ve used a custom class to build a replacement for a List:
class ListLike: length = 0 def __getitem__(self, key): return self.__getattribute__(str(key)) def __setitem__(self, key, val): self.__setattr__(str(key), val) def __delitem__(self, key): self.__delattr__(key) def append(self, val): self[str(self.length)] = val self.length += 1 listLike = ListLike() print(listLike.length) # 0 listLike.append("Hello") listLike.append("World") print(listLike.length) # 2 print(listLike) # "Hello"Code language: Python (python)
This appears to work amazingly at first glance, until you try to do the following:
[x for x in listLike]Code language: Python (python)
Or any other kind of iteration on the ListLike. You’ll get the following confusingly named error:
'ListLike' object has no attribute '2'Code language: plaintext (plaintext)
This is because Python doesn’t know howto iterate through your class, and therefore attempts to access a property in the class. This is where
__iter__ comes into play: It allows you to return an iterable to utilize anytime Python might request iterating through the class, like ina list comprehension.
[x for x in instance]
class ListLike: length = 0 def __getitem__(self, key): return self.__getattribute__(str(key)) def __setitem__(self, key, val): self.__setattr__(str(key), val) def __delitem__(self, key): self.__delattr__(key) def __iter__(self): isMethod = lambda x: type(x).__name__ == 'method' # Only return non-method keys that are not "length" return iter([x for x in dir(self) if not x.startswith('__') and x != 'length' and not isMethod(self[x])]) def append(self, val): self[str(self.length)] = val self.length += 1 listLike = ListLike() listLike.append("Hello") listLike.append("World") [print(x) for x in listLike]Code language: Python (python)
Notice that we’re having to return a real list wrapped in the
itermethod for the
__iter__return value: This is required by Python.
If you don’t do this, you’ll get the error:
iter() returned non-iterator of type 'list'
Check if an item exists using the “in” keyword
__iter__ magic method isn’t the only way to customize traditionally list-like behavior for a class. You can also use the
__contains__ method to add support for simple “is this in the class” checks.
__contains__(self, item) –
key in instance
Something to keep in mind is that if
__contains__ isn’t defined, Python will use the information provided by
__iter__ to check if the key is present. However,
__contains__ is a more optimized method, since the default
__iter__ checking behavior will iterate through every key until it finds a match.
Python magic method cheat sheet
Python magic methods can level up your application logic by reducing the amount of boilerplate required to do specific actions, but that’s not its only usecase. Othertimes, you might want to use magic methods to provide an API with a nicer development experience for consuming developers.
That said, we know that with so many magic methods it can be difficult to remember them all. This is why we made a cheat sheet that you can download or print out to reference when writing code.