Saturday, May 15, 2021

The 2021 Python Language Summit: PEP 654 — Exception Groups and except*

PEP 654 was authored by Irit Katriel, Yury Selivanov, and Guido van Rossum. This PEP is currently at the draft stage. At the 2021 Python Language Summit, the authors shared what it is, why we need it, and which ideas they rejected.

Irit Katriel, Yury Selivanov, and Guido van Rossum

 What Is PEP 654?

The purpose of this PEP is to help Python users handle unrelated exceptions. Right now, if you're dealing with several unrelated exceptions, you can:

  • Raise one exception and throw away the others, in which case you're losing exceptions
  • Return a list of exceptions instead of raising them, in which case they become error codes rather than exceptions, so you can't handle them with exception-handling mechanisms
  • Wrap the list of exceptions in a wrapper exception and use it as a list of error codes, which still can't be handled with exception-handling mechanisms

PEP 654 proposes:

Each except* clause will be executed once, at most. Each leaf exception will be handled by one except* clause, at most.

In the discussions about the PEP that have happened so far, there were no major objections to these ideas, but there are still disagreements about how to represent an exception group. Exception groups can be nested, and each exception has its own metadata:

A nested ExceptionGroup, with metadata

Originally, the authors thought that they could make exception groups iterable, but that wasn't the best option because metadata has to be preserved. Their solution was to use a .split() operation to take a condition on a leaf and copy the metadata:

Splitting exception groups with .split()

 

Why Do We Need Exception Groups and except*?

There are some differences between operational errors and control flow errors that you need to take into account when you're dealing with exceptions:

Operational errors vs control flow errors in Python

In Example 1, there is a clearly defined operation with a straightforward error. But in Example 2, there are concurrent tasks that could contain any number of lines of code, so you don't know what caused the KeyError. In this case, handling one KeyError could potentially be useful for logging, but it isn't helpful otherwise. But there are other exceptions that it could make more sense to handle:

asyncio.CancelledError

It's important to understand the differences between operational errors and control flow errors, as they relate to try-except statements:

  • Operational errors are typically handled right where they happen and work well with try-except statements. 
  • Control flow errors are essentially signals, and the current semantics of the try-except statement doesn't handle them adequately.

Before asyncio, it wasn't as big of a problem that there weren't advanced mechanisms to react to these kinds of control flow errors, but now it's more important that we have a better way to deal with these sorts of issues. asyncio.gather() is an unusual API because it has two entirely different operation modes controlled by one keyword argument, return_exceptions

asyncio.gather()

The problem with this API is that, if an error happens, you still wait for all of the tasks to complete. In addition, you can't use try-except to handle the exceptions but instead have to unpack
the results of those tasks and manually check them, which can be cumbersome.

The solution to this problem was to implement another way of controlling concurrent tasks:

asyncio.TaskGroup

If one tasks fails, then all other tasks will be cancelled. Users of asyncio have been requesting this kind of solution, but it needed a new way of dealing with exceptions and was part of the inspiration behind PEP 654.

Which Ideas Were Rejected?

Whether or not exception groups should be iterable is still an open question. For that to work, tracebacks would need to be concatenated, with shared parts copied, which isn't very efficient. But iteration isn't usually the right approach for working with exception groups anyway. A potential compromise could be to have an iteration utility in traceback.py.

The authors considered teaching except to handle exception groups instead of adding except*, but there would be too many backwards compatibility problems. They also thought about using an except* clause on one exception at a time. Backwards compatibility issues wouldn't apply there, but this would essentially be iteration, which wouldn't help.