Thinking about running for the Python Software Foundation Board of Directors? Let’s talk!

This year’s Board Election Nomination period is opening tomorrow. Current board members want to share what being on the board is like and are making themselves available to answer all your questions about responsibilities, activities and time commitments via online chat. Please come join us on Slack anytime in June to talk with us about being on the PSF board.

Board Election Timeline:

• Nominations are open, Thursday, June 1st, 2:00 pm UTC
• Board Director Nomination cut-off: Thursday, June 15, 11:59 pm UTC
• Voter application cut-off date: Thursday, June 15, 11:59 pm UTC
• Announce candidates: Friday, June 16th
• Voting start date: Tuesday, June 20, 12:01 am UTC
• Voting end date: Friday, June 30, 11:59 pm UTC

Not sure what UTC is for you locally? Check here!

Nominations will be accepted here. (Note: you will need to sign into or create your python.org user account first). Learn more about membership here or if you have questions about membership or nominations please email psf-elections@python.orgIn addition to Slack you are welcome to join the discussion about the PSF Board election on our forum.

Also, you can see your membership record and status on psfmember.org. If you are a voting-eligible member and do not already have a login there, please sign up and then email psf-donations@python.org so we can link your membership to your account.

The Python Language Summit 2023: Towards Native Profiling for Python

Joannah Nanjekye came to the Python Language Summit 2023 to discuss innovations by Scalene, a sampling-based Python profiler that can distinguish between native code and Python code in its reports. After its initial release in late 2019, Scalene has become one of the most popular Python profiling tools. It has now been downloaded 500,000 times from PyPI.

The Scalene project logo

A profiler is a tool that can monitor a program as it is running. Once the program has run, the profiler can provide a report analysing which lines of code were visited most often, which were the most expensive in terms of time spent, and which were the most expensive in terms of memory usage. Profilers can therefore be hugely useful tools for addressing performance issues in code. If you’re unsure where your program is spending most of its time, it can be hard to optimise it.

Profilers can be split into two broad categories: trace-based profilers and sampling-based profilers. Trace-based profilers work by intercepting each function call as your program is running and logging information about the time spent, memory usage, etc. Sampling-based profilers, meanwhile, take snapshots of your program at periodic intervals to monitor these things. A trace-based profiler has the advantage that it can provide a granular and precise level of detail about which lines of code were executed and when each function call finishes; this makes it ideal for use as a tool to monitor test coverage, for example. However, injecting tracing hooks into each function call can sometimes slow down a program and distort the analysis of where most time was spent. As a result, sampling-based profilers are sometimes preferred for profiling performance.

Scalene is a sampling-based profiler, and aims to address the shortcomings of previous sampling-based profilers for Python. One of the key challenges sampling-based profilers have faced in the past has been accurately measuring the time Python programs spend in “native code”.

Slide from Nanjekye’s talk, illustrating sampling-based profiling

Handling the problem of native code

“Native code”, also sometimes referred to as “machine code”, refers to code consisting of low-level instructions that can be interpreted directly by the hardware processor. Using extensions to Python written in C, C++ or Rust that will compile to native code – such as NumPy, scikit-learn, and TensorFlow – can lead to dramatic speedups for a program written in Python.

It also, however, makes life difficult for sampling-based profilers. Samplers often use Python’s signal module as a way of knowing when to take a periodic snapshot of a program as it is running. However, due to the way the signal module works, no signalling events will be delivered while a Python program is spending time in a function that has been compiled to native code via an extension module. The upshot of this is that sample-based profilers are often “flying blind” for Python code that makes extensive use of C extensions, and will sometimes erroneously report that no time at all was spent executing native code, even if the program in fact spent the majority of its time there.

Scalene’s solution to this problem is to monitor delays in signal delivery. It uses this information to deduce the amount of time that the program spent outside CPython’s main interpreter loop (due to the use of native, compiled code from an extension module). Further details on Scalene’s methods, and comparisons with other leading Python profilers, can be found in a recent paper by Emery D. Berger, Sam Stern and Juan Altmayer Pizzorno, “Triangulating Python Performance Issues with Scalene”.

Nanjekye also detailed Scalene’s sophisticated approach to measuring performance in child threads. Signal-based profilers often struggle with multi-threaded code, as signals can only be delivered and received from the main thread in Python. Scalene’s solution is to monkey-patch functions that might block the main thread, and add timeouts to these functions. This allows signals to be delivered even in multithreaded code.

Discussion

Nanjekye asked attendees at the Language Summit if they would be interested in integrating Scalene's ideas into the standard library's cProfile module, which was met with a somewhat muted response.

Pablo Galindo Salgado, a leading contributor to the Memray profiler, criticised Scalene’s signal-based approach, arguing it relied on inherently brittle monkey-patching of the standard library. It also reported unreliable timings, Salgado said: for example, if code in a C extension checks for signals to support CTRL-C, the resulting delays measured by Scalene will be distorted.

Salgado argued that integration with the perf profiler, which Python is introducing support for in Python 3.12, would be a better option for users. Mark Shannon, however, argued that perf distorted the execution time of Python programs; Salgado responded that Scalene did as well, as the use of signals came with its own overhead.

Nanjekye argued that the huge popularity of Scalene in the Python ecosystem was evidence that it had proved its worth. Carol Willing concurred, noting that Scalene was an especially useful tool with code that made heavy use of libraries such as NumPy, Scikit-Learn and PyTorch.

The Python Language Summit 2023: Burnout is Real

Guido van Rossum, creator of the Python programming language, spoke at the 2023 Python Language Summit on the subject of open-source burnout, strategies for tackling it, and how to avoid it.

The first known case of burnout in the field of open-source software, van Rossum speculated, may have been Charles Babbage, who gave up the post of Lucasian Professor of Mathematics (the “Chair of Newton”) at Cambridge University in 1839.

“In 1839 the demands of the Analytical Engine upon my attention had become so incessant and so exhausting, that even the few duties of the Lucasian Chair had a sensible effect in impairing my bodily strength. I therefore sent in my resignation.” 

-- Charles Babbage, “Passages from the Life of a Philosopher” (Chapter 4)

Van Rossum described how the Python community had been hit multiple times by core developers, suffering from burnout, suddenly disappearing for extended periods of time. Van Rossum told the story of one core developer, previously one of the most prolific contributors to CPython, who had abruptly ceased contributing around a decade ago. He had hardly been heard from since.

Van Rossum himself, he recounted, had felt so burned out by the acrimonious debate around PEP 572 (proposing the “walrus operator”, :=), that it led to him stepping down from his post as Benevolent Dictator For Life (“BDFL”) of Python in 2018. Decisions about the language were ceded to a democratically elected Steering Council, which has governed Python ever since.

Burnout, van Rossum noted, was often connected to conflict – and it often didn’t matter whether or not you ended the conflict victorious. Merely having the conflict at all could be exhausting.

Van Rossum’s talk itself was fairly short, but was followed by a lengthy discussion among the assembled core developers on the experiences they’d had with burnout, and strategies that could be employed to tackle it.

Several attendees in the room commented that learning to recognise burnout in yourself was an important skill. Some participants in the discussion described times when they had suddenly realised that things that had previously been enjoyable had morphed into a source of stress. One core developer told the story of a conference they had organised, at which they had felt such extreme stress that they were unable to think of any of the things that had gone well. Instead, they found themselves fixated on all of the minor things that had gone wrong.

Learning to recognise burnout in others was perhaps an even harder problem to solve. Van Rossum noted that the developers most susceptible to burnout were generally those who were most active and engaged with the community. But how can you distinguish between somebody devoting their time to CPython because of the intense enjoyment they found in contributing to the project, somebody who might have formed an unhealthy addiction to open source, and somebody who was only continuing to contribute out of a misplaced sense of obligation?

Some developers spoke of strategies they used to decompress. Brett Cannon described how he periodically takes “open source breaks”, in which he forces himself to spend a period of time without looking at his GitHub notifications or thinking about his open-source commitments. Mariatta Wijaya spoke about how she found mentoring other Python programmers to be deeply healing. All agreed that it was crucial to talk to friends and relatives when you were feeling close to burnout.

It was agreed that the Python community needed to do better in many ways. We needed to become better, as a community, at understanding when other people said that they were unable to complete something they had previously committed to. And perhaps we needed to normalise questions such as, “Hey, you’ve been super productive and responsive for too long. When do you think you’ll burn out?”

Russell Keith-Magee remarked that systems with single points of failure were bound to lead to situations of intense stress. These systems would inevitably, at some point, fail. The transition from a single BDFL (with an indefinite term) to a five-member Steering Council with one-year terms had been a very positive change in that regard, Keith-Magee said. But there were still places in CPython development where there were single points of failure. Examples included various esoteric platforms where support from CPython depended on a single core developer being able to give up their time to review PRs relating to the platform.

Carol Willing agreed with Keith-Magee, pointing out that no matter who you were, you were rarely the only person who could do a certain task. You might be the person who could do it fastest or best – but sometimes, it was important to “see the people, and share the joy”.

Łukasz Langa spoke about his role as part of the current Code of Conduct Working Group, to which any violations of the Python Code of Conduct can be reported. Langa remarked that being part of the Working Group had brought to the fore parts of the community which he had previously been mostly unaware of. He encouraged everybody in the room to report toxic members of the community who were discouraging or aggressive online.

Speaking personally, for a moment: I tried to take an open-source break earlier this year, when I felt myself close to burning out on some of my open-source commitments. I can’t say it was fully successful – my addiction to GitHub was too great for me to resist glancing at my notifications occasionally. But it was helpful, and re-energising, to spend some time doing a creative activity that bore with it a 0% risk of people shouting at me on the internet about it:

The Python Language Summit 2023: Making the Global Interpreter Lock Optional

The Global Interpreter Lock (“GIL”), is one of the most fundamental parts of how Python works today. It’s also one of the most controversial parts, as it prevents true concurrency between threads – another way of saying that it’s difficult to run two functions simultaneously while writing pure-Python code.

If there’s one blog that really “took off” after I wrote last year’s coverage on the Python Language Summit, it was my blog on Sam Gross’s proposal to make Python’s Global Interpreter Lock (the “GIL”) optional. One week following the publication of my articles, the blog had been viewed nearly 38,000 times; the blog in “second place” had only been viewed 5,300 times.

Interest in removing the GIL is clear, therefore – and this year, Gross returned to the Python Language Summit to discuss the development of his plans.

Dare to dream of a GIL-free world

Gross started off by giving an update on how work on nogil – Gross’s fork of CPython with the GIL removed – had progressed over the past year. Gross had been spending the last few months rebasing his fork onto CPython 3.12. As a result, he was now able to give an accurate estimate of how bad the performance costs to single-threaded code would be, if the GIL were removed.

Over the past year, Gross had also written a PEP – PEP 703 – which, following the Language Summit, was submitted to the Steering Council for their consideration on May 12. If the PEP is accepted by the Steering Council, a version of Python with the GIL disabled could be available as soon as Python 3.13. (To discuss the ideas in the PEP, head to the thread on discuss.python.org.)

Gross reported that the latest version of nogil was around 6% slower on single-threaded code than the CPython main branch, and that he was confident that the performance overhead could be reduced even further, possibly to nearly 0%. Most of the overhead was due to reference counting and changes that had been required to the operation of CPython’s new-in-3.11 specialising adaptive interpreter. With multiple threads, the performance overhead was around 8%.

A slide from Gross's talk
(numbers indicate the per-thread overhead)

Having rebased onto Python 3.12, Gross was also able to give an estimate on the extent of the code changes that would be required to CPython, were nogil to be accepted. Excluding generated files, Gross reported that around 15,000 lines of code would need to be changed. There would also need to be some alterations to mimalloc, which nogil depends on – but Gross was hopeful that these would be accepted to mimalloc itself, reducing the need for CPython to maintain mimalloc patches in its codebase.

Some audience members expressed concern about the prospect of “#ifdef hell” across the code base, to account for the two possible CPython build options. However, Gross countered that most of the changes required could be made unconditionally without resorting to #ifdefs, since the changes were simply no-ops with the GIL enabled.

The plan for nogil remains that it would be enabled via a compile-time flag, named --disable-gil. Third-party C extensions would need to provide separate wheels for GIL-disabled Python.

GIL gotta go

Last year, it felt as though Gross’s proposal was met with a certain degree of excitement, but also a certain degree of scepticism. It was thrilling to see how far nogil had come, but there was a certain amount of frustration in the room at the lack of a concrete plan for where to go next.

This year, it felt like the proposal was greeted much more warmly. Gross had come to the summit with a far more clearly defined roadmap for how we might get to nogil, and had taken the time to put together detailed estimates of the performance impact and the extent of the code changes.

“Thank you for doing this… now we have something we can actually consider!” 

-- Brandt Bucher, CPython Core Developer

Attendees at the Language Summit were also impressed at how low the performance overhead of nogil was, although Mark Shannon commented that he thought the numbers might be “a slight underestimate”. Gross explained that he had managed to achieve a 2-3% speedup by optimising for the case where only a single thread was active. Even in multithreaded programs, he explained, this provided a performance boost, since even in multithreaded code, it was often the case that only a single thread would be attempting to access an object at any given point in time.

Larry Hastings expressed concern that nogil might make debugging code harder; Gross responded that there was some impact on debuggability, but that it wasn’t too bad, in his opinion. Pablo Galindo Salgado, release manager for Python 3.10 and 3.11, expressed concern that nogil could also make implementing debuggers trickier – but commented that “it might be worth the price” anyway.

Another point of discussion was the changes Gross had made to the specialising adaptive interpreter in the nogil fork. In order for the specialisations to work with the GIL disabled, Gross had had to guard the adaptive specialisations to the bytecode behind a lock. As well as this, each thread had been limited to a single specialisation of any given bytecode; with the GIL enabled, the adaptive interpreter can respecialise bytecode multiple times. Gross commented that he thought it would probably be possible to allow multiple specialisations of bytecode in multithreaded code, but that this would require further investigation. His current solution was the simplest one he had found, for now.

The Python Language Summit 2023: Three Talks on the C API

The Python Language Summit 2023 began with an extended discussion of Python’s C API – the interface through which it is possible to communicate between code written in Python and code written in low-level languages, like C. The fullness of this interface is a significant factor behind the vibrancy of Python’s ecosystem, enabling libraries such as NumPy and pandas that are foundational to Python’s widespread use in data science.

Three speakers had booked slots to discuss the C API this year: Mark Shannon, Guido van Rossum, and Antonio Cuni. The conversation evolved naturally from one talk to the next, so in this blog post, I’ll be discussing the three talks together.

All at sea on the C API

The C API at sea (illustration by DALL-E)

“I still don’t know what the C-API is, and I’ve been trying for years!” 

-- Mark Shannon, CPython Core Developer

Mark Shannon spoke on the problems (as he saw them) with Python’s C API. Shannon lamented that with every minor Python version, a slew of third-party C extensions seemed to break. He argued that the root cause was that the needs of C extensions were not adequately met by the formal C API, which had evolved in a haphazard and often-unplanned way over the past three decades of Python’s existence.

As a result of the C API’s flaws, Shannon said, extension authors were forced to reach beyond the formal API and into implementation details that had emerged as a kind of “implicit API”. The implementation details that constituted the new “implicit API” had become so widely depended upon that it was now impossible for CPython to change some parts of its code without breaking large parts of the Python ecosystem.

Shannon believes that the new “implicit API” should be formalised in Python 3.13. This, he argues, would put an end to the cycle of CPython releases inevitably leading to widespread breakages in C extensions.

Sam Gross, who (among other things) has contributed to pytorch, agreed with Shannon that the C API was lacking in many areas. Gross argued that there was a great deal of important functionality that wasn’t exposed to extension authors. “Projects just end up copying-and-pasting CPython C code,” Gross said, meaning the extensions broke with each new release of CPython.

Pablo Galindo Salgado, release manager for Python 3.10 and 3.11, said that the release process for those versions had felt like a “game of whackamole” when it came to third-party C extensions breaking. Salgado argued that CPython needed to reach out to the authors of extensions such as pytorch to gather detailed feedback on what core functionality was missing from the API. Several attendees expressed frustration with a perceived tendency among C extension authors to immediately reach into CPython implementation details when something they needed was missing from the API. The result of this was that CPython core developers were often in the dark about which things the C API should be providing, but currently wasn’t. “We might not be able to give you a solution,” Salgado said, “But please come to us and tell us what your problem is, if you have a problem!”

Gross proposed that CPython should run third-party test suites with new versions of CPython as they were being developed, so that the Core Dev team would be able to spot third-party breakages early and gauge the impact of their changes. Pytorch operated a similar programme, Gross said, and it had been successful in helping to limit breakages of third-party pytorch models as the core pytorch team evolved their API.

Brandt Bucher noted, however, that the problem often wasn’t so much that CPython was unaware when they were breaking third-party code – the benchmarks run in the pyperformance suite often served as an early warning signal for breakages in C extensions. The problem was often that CPython would offer to help affected projects, only to have their help rejected. Several core developers had previously sent pull requests to help third-party projects become compatible with an upcoming version of CPython, only for their pull requests to remain unmerged for several months due to burned-out maintainers of these projects.

Let’s get specific

Guido van Rossum speaks to the Language Summit on the C API
(photo by Hugo van Kemenade)

Shannon was clear about what he thought the problem with the C API was. The problem was that the C API was insufficient for the authors of C extensions, leading these authors to reach into CPython implementation details, leading to an unending cycle of projects breaking with each new release of CPython. Others, however, argued that this wasn’t so much a specific problem but a genre of problems. Each specific project might have a different notion about which things were imperfect with the C API, and which things were missing from the C API. Each imperfection or absence could be considered a concrete problem in its own way. “Things break for everybody, but things break in different ways for different people,” Carol Willing argued. “We need more granularity in our understanding of that.”

As Mark Shannon’s slot drew to an end, Guido van Rossum opted to continue the discussion that Shannon had started, but sought to draw attention to a more precise enumeration of the difficulties C API users were facing.

“There’s lots of ideas here, but I don’t know what the problem is!” 

-- Carol Willing, CPython Core Developer

Simon Cross, a contributor to the HPy project, reported that the HPy project had, in the early stages of the project, put together a long list of the problems, as they saw them, with the CPython C API. Cross offered to share the list with the Core Dev team. Thomas Wouters, a Google employee, also offered to provide a list of difficulties Google had experienced when upgrading to recent Python releases, something the company keeps detailed records of. There was agreement that putting together a comprehensive list of problems with the existing API was an important first step, before CPython could consider drawing up plans to fix the problem.

The C API discussions ended with an agreement that further discussion was required. Interested parties can follow the ongoing conversation at https://github.com/capi-workgroup/problems/issues. The plan is to work towards an informational PEP, with input from an array of stakeholders, outlining a consensus around the problems and pitfalls in the current C API. Once the problems with the status quo have been enumerated in detail, the community might be in a position to consider possible solutions.

HPy: A possible solution?

A slide from Antonio Cuni's talk on HPy

While the C API discussions ended with a detailed discussion of the problems in the current C API, the first talk of the day was in fact by Antonio Cuni, a core developer with the HPy project. HPy is an alternative C API for Python – an API that seeks to avoid many of the pitfalls of the current API. The contention of the HPy developers is that the current C API is bad for CPython, bad for alternative implementations of Python such as PyPy or GraalPython, and, ultimately, bad for end users.

HPy is a specification of a new API and ABI for extending Python that is Python implementation agnostic and designed to hide and abstract internal details 

-- The HPy GitHub README

Cuni began by describing the key goals of the HPy project:

• An API that doesn’t leak CPython-specific implementation details
• A 0% (or close to 0%) performance overhead when compared with CPython’s current C API
• A “Universal ABI” that allows compiled extension modules to use the same interface to communicate with PyPy (for example) as they would do to communicate with CPython
• An API that is garbage-collection friendly.

Cuni argued that if the Python ecosystem as a whole moved to using HPy, instead of the “official” C API, there would be dramatically fewer breakages of C extensions with each new Python release. Cuni’s proposal was that CPython should “officially bless” HPy as the recommended C API for Python.

HPy Hpy Hooray?

Simon Cross, HPy Core Developer
(photo by Hugo van Kemenade)

Cuni’s talk was exuberant, but it was at times somewhat unclear what exactly he was asking for from the room. “The investment from CPython,” Cuni argued “would be a political investment rather than a technical one”. Thomas Wouters, however, argued that the CPython team didn’t have the “moral authority” to coerce extension authors into using HPy. Hynek Schlawack agreed, arguing that it was perhaps unrealistic to migrate the entire Python ecosystem towards HPy.

Many were uncertain about what it would even mean to “officially bless” HPy – would CPython host HPy’s documentation on docs.python.org? Or would CPython simply add a note to the documentation of the C API that the “official” C API was no longer the recommended way to write a C extension? Guido van Rossum emphasised that a top-down approach from the Core Dev team to extension authors wouldn’t work: nobody wanted a repeat of the decade-long transition from Python 2 to Python 3. Carol Willing agreed that pushing C extension authors to use HPy could be counterproductive, arguing that it was important to remember the impact of our decisions on end users of Python.

Other core developers were sceptical about the fundamentals of HPy itself. HPy’s origin story lies in difficulties PyPy encountered when it trying to use CPython’s existing C API. These difficulties led to attempts to create an API that could be easily used on either Python implementation. Larry Hastings argued that blessing an API that was friendlier to PyPy had clear benefits to PyPy (and other implementations), but that it was less clear where CPython stood to gain from this change.

Cuni’s response was that if the ecosystem migrated to an API that exposed fewer implementation details directly, CPython would be able to refactor its internal implementation far more seamlessly, as CPython would no longer have to worry about breaking a multitude of third-party C extensions with every internal reorganisation of code. Cuni mentioned efforts to run CPython using WebAssembly as a specific area where CPython could stand to gain, as the HPy API could interact far more easily with the Javascript garbage collector. And Cuni also noted that HPy made it easy for extension authors to test whether they were sticking to the API or not, something which is famously hard to know with the current C API. “We don’t know all the experimentation that this might enable,” Cuni exclaimed, however, “Because we haven’t implemented this change yet!”

Mark Shannon was another core developer expressing scepticism about HPy. Shannon argued that while HPy had strengths over the “official” C API, it was far from perfect. “We should try to fix CPython’s API” before CPython recommended users switch to HPy, Shannon argued. Simon Cross, also of the HPy project, said that the team welcomed feedback about where they could improve. It was still easy for HPy to make changes, Cross argued, given they had not yet achieved widespread adoption.

Further Reading on HPy

1. HPy’s overview of changes needed to the C API.
2. HPy’s explanation of why the changes are needed.

The Python Language Summit 2023: Python on Mobile

At the Python Language Summit 2023, Russell Keith-Magee presented on the ongoing efforts of BeeWare, a project that aims to make it easier to run Python on mobile platforms such as Android and iOS.

The BeeWare logo

Russell Keith-Magee is one busy bee

Improving Python’s story for running on mobile has been a labour of love for Keith-Magee for eight years at this point. Keith-Magee last presented at the Python Language Summit in 2020 (a year when the summit was conducted entirely virtually due to the Covid-19 pandemic). Since then, however, great progress has been made.

The biggest change since his last update, Keith-Magee reported, wasn’t technical – it was financial. For the last year, BeeWare has no longer been a hobby project for Keith-Magee. He is now paid by Anaconda to work on the project full time, along with a colleague, Malcolm Smith. “I now have the resources to do the work” required to make this happen, he announced.

Keith-Magee came to the Language Summit this year with a proposal: to add Android and iOS as platforms with “tier-3” support from CPython in Python 3.13.

What does “tier-3 support” mean? Tier-3 support, as defined by PEP 11, describes a level of support that the CPython core developers commit to giving a specific platform. The CPython test suite is run constantly on popular platforms such as Ubuntu, Windows and MacOS, and test failures on these platforms can block releases until they are fixed. More esoteric platforms, meanwhile, are tested in CI less frequently. Test failures on those platforms will not necessarily block a release of CPython.

Tier-3 support is the current level of support Python provides to the emscripten, WASI and FreeBSD platforms, among others. If a platform has tier-3 support, the test suite will be run on the platform on a regular basis, but not on every pull request. Tier-3 support indicates that at least one core developer has committed to supporting CPython on that platform as best they can. However, test failures on that platform will not block a release of CPython.

The path to tier-3 support

Historically, a significant barrier standing in the way of mobile-platform support from CPython has been the difficulties associated with running tests on mobile platforms in CI. Keith-Magee announced, however, that it was now possible to run the CPython test suite on mobile platforms via Briefcase, BeeWare’s packaging and development tool. (Getting the test suite to pass is another issue – But Keith-Magee felt confident that it would be easy to make progress on that front.) As such, Keith-Magee reported, it should be feasible for CPython to integrate running tests on these platforms into the project’s testing infrastructure on GitHub.

One remaining issue is a fairly specific question, but an important one nonetheless: on a mobile platform, what should sys.platform be? The two major mobile platforms are currently inconsistent about this: on iOS, sys.platform == "ios", whereas on Android, sys.platform == "linux".

The advantage of the first approach is that it is easy for user code to detect whether the code is being run on iOS or not. The advantage of the second approach, meanwhile, is that most existing Python code won’t necessarily account for the possibility that it might be run on Android or iOS, so will run into difficulties with idioms such as the following:

if sys.platform == "linux":
    do_fancy_linux_only_feature()

The Android platform, Keith-Magee noted, is very similar to Linux, so by setting sys.platform to “linux”, a lot of code “just works” on Android even though the code hasn’t explicitly accounted for the possibility that it might be run on that platform.

Abuzz with excitement

Keith-Magee in flight (photo by Hugo van Kemenade)

Keith-Magee’s talk was greeted enthusiastically by the core developers in the room; there was strong consensus that Python needed a better story on mobile platforms. Carol Willing expressed excitement about the ways in which support for mobile platforms could help Python spread globally, to countries where large numbers of people had no access to desktop computers (but had easy access to phones). Łukasz Langa agreed, noting that he had received many enquiries about Python on mobile after giving a talk on the subject about a year ago. “It’s interesting to a lot of people,” Langa commented. “We need it.”

“Wooooo!” 

-- Carol Willing, CPython Core Developer

On the sys.platform question, Core Developer Filipe Laíns said that he was working on a new API for the sysconfig standard-library module, which will provide a more granular way of distinguishing between platforms from user code. In the meantime, Brett Cannon wondered if BeeWare could use the same approach as CPython builds for WebAssembly: on WebAssembly builds, unusually, sys.platform has a different value to os.name (sys.platform is either "wasi" or "emscripten", but os.name is "linux").

Another outstanding question, however, is what the release process would look like for these new platforms. There was appreciation of the work Keith-Magee had already put into BeeWare, and nobody doubted that he would continue to be committed to the project. However, Keith-Magee is not currently a core developer, leading to a concern that CPython might be supporting a platform that nobody on the core team had expertise in.

Ned Deily, release manager for Python 3.6 and 3.7, worried that distributing CPython binaries for these platforms might not be feasible, as it would make the release process “even more arduous”. Keith-Magee responded that it could be possible to automate the build process for these platforms. If it wasn’t, he said, it also wouldn’t necessarily be essential for CPython to distribute official binaries for these platforms, at least at first.

Where next for BeeWare?

Keith-Magee’s next steps are to work towards upstreaming the patches to CPython that the BeeWare project has made, so that CPython on mobile platforms can “just work” without any changes being made. The alterations that have already been made to support CPython on WebAssembly have made this task much easier, Keith-Magee noted.

The Python Language Summit 2023

Every year, just before the start of PyCon US, core developers, triagers, and special guests gather for the Python Language Summit: an all-day event of talks where the future direction of Python is discussed. The Language Summit 2023 included three back-to-back talks on the C API, an update on work towards making the Global Interpreter Lock optional, and a discussion on how to tackle burnout in the community.

This year's summit received around 45 attendees, and the summit was covered by Alex Waygood.

Attendees of the Python Language Summit

---

• Three Talks on the C API: Talks by Antonio Cuni, Mark Shannon and Guido van Rossum
• Making the Global Interpreter Lock Optional: A talk by Sam Gross
• Towards Native Profiling for Python: A talk by Joannah Nanjekye
• What is the stdlib for? A talk by Brett Cannon
• Pattern Matching, __match__, and View Patterns: A talk by Michael Sullivan
• Python on Mobile: State of the Union. A talk by Russell Keith-Magee
• Burnout is Real: A talk by Guido van Rossum
• Lightning Talks, featuring short presentations by Dong-hee Na, Carl Meyer and Amethyst Reese

The Python Language Summit 2023: Lightning Talks

The Python Language Summit 2023 closed off with a trio of lightning talks from Dong-hee Na, Carl Meyer and Amethyst Reese.

---

Dong-hee Na: Let’s support LLVM-BOLT as an official feature

CPython Core Developer Dong-hee Na gave a short presentation on the LLVM-BOLT optimiser, arguing that we should support it as a standard feature of CPython.

LLVM-BOLT is a “post link time binary optimiser” that was adopted by the Pyston project, a performance-oriented fork of CPython 3.8. The Pyston team had reported that use of the optimiser resulted in performance gains of 2-4% on their benchmarks, although the Faster CPython team and Dong-hee Na had reported smaller returns when they had applied the optimiser to Python 3.11.

Dong-hee Na showed benchmark results that showed significant speedups in some areas with LLVM-BOLT applied to Python 3.12, but noted that LLVM-BOLT also caused regressions in some other areas due to overly aggressive optimisations. He announced that he had added support for LLVM-BOLT to CPython as an optional compile-time flag, --enable-bolt, to allow experimentation with the feature.

A slide from Dong-hee Na's talk on LLVM-Bolt

---

Carl Meyer: Lazy Imports – the sequel

Carl Meyer instigated a short discussion on proposals to introduce a mechanism enabling lazy imports in Python. Following Meyer’s lightning talk on the same subject last year, Meyer – along with his colleague, Germán Méndez Bravo (who, according to Meyer, deserves “most of the credit”) – had written PEP 690, proposing a compile-time flag that would make imports in Python lazy by default. The PEP, however, was rejected by the Steering Council in December 2022, due to concern that the new flag would have created a split in the community between programmers who used Python with lazy imports enabled, and those who used Python with eager imports.

Meyer’s question to the audience was: where next for lazy imports? Was it worth modifying the proposal and trying again, or was the whole idea doomed? Meyer noted that the team at Instagram, where he worked, had seen start-up time improvements of 50-80%, and 40-90% reductions in memory usage, by adopting lazy imports in the fork of CPython they used for the Instagram web server.

Meyer floated a series of possible changes (some mutually exclusive) that could be made to the PEP. For each possible change, he asked if the change would make attendees more or less likely to support adding support for lazy imports to Python:

1. Explicit opt-in syntax marking a specific import as lazy (e.g. lazy import inspect).
2. A clear roadmap detailed in the PEP, outlining the timeframe in which it was expected that lazy-import behaviour would become the default in Python.
3. A promise that the implementation of lazy imports would not lead to any changes being made to the dict data structure.
4. Generalised support of “lazy names”, rather than just support for lazy imports specifically.

The room unanimously agreed that change (3) would make them more likely to support the PEP, and largely agreed that change (4) would make them less likely to support it. The room was (frustratingly, for Meyer) split on whether proposals (1) and (2) would make them more or less likely to give the PEP their support.

On the bright side, only one attendee said they thought they could never support a proposal for lazy imports in Python. Unfortunately for Meyer, the attendee in question was Thomas Wouters, currently serving on the Steering Council.

---

Amethyst Reese: Can we __call__ modules?

Amethyst Reese presented on an idea that has since become PEP 713: a proposal to add a mechanism allowing developers to easily create callable modules.

Amethyst Reese at the Python Language Summit
(Photo by Hugo van Kemenade)

Strictly speaking, it’s possible to create a callable module today, but it’s not exactly easy. The example given in the PEP looks something like the following:

import sys
import types

def fancy(...):
    ...

class FancyModule(types.ModuleType):
    def __call__(self, ...):
        return fancy(...)

sys.modules[__name__].__class__ = FancyModule

Reese proposes that we provide a simpler mechanism to create callable modules: simply provide special recognition for module-level __call__ functions, similar to the way that PEP 562 added special recognition of module-level __getattr__ and __dir__ functions. With the semantics specified in PEP 713, fancy.py could be rewritten as follows:

def fancy(...):
    ...

__call__ = fancy

With a module, fancy.py, defined like the above, users would simply be able to do the following:

import fancy
fancy()

This would allow users of Python to avoid constructs which often feel unnecessarily verbose and involve frustrating amounts of boilerplate, such as:

import datetime
import pprint
import dis

d = datetime.datetime()
pprint.pprint(...)
dis.dis(...)

It would also allow users to create callable modules in a way that would be easier for type checkers to support, as dynamically inserting custom objects into sys.modules can cause issues for these tools.

The proposal was met with curiosity by attendees of the Language Summit. Thomas Wouters said that he had originally opposed the addition of module-level __getattr__ and __dir__, introduced by PEP 562. However, now they had been introduced to Python, he was of the opinion that it might make sense to add support for module-level dunder methods including __call__, but also others such as __setattr__.

The Python Language Summit 2023: What is the Standard Library for?

 Brett Cannon came to the Python Language Summit this year with a fundamental question for the assembled core developers: What is the standard library for?

According to a quick python -c "import sys; print(len(sys.stdlib_module_names))" call on my laptop, the standard library in Python 3.11 consists of 305 importable modules. Many of these are implementation details that, if you’re a good citizen, you really shouldn’t be importing – but the point stands that the Python standard library is perhaps now larger than it should be.

But the goal of his question, Cannon explained, wasn’t to decide which modules to get rid of. Instead, it was to create guidelines on when and why new modules should be accepted into the standard library.

"We need to audit the standard library, and not deprecate it, but decide which bits should probably not have been added if we had perfect hindsight. 

-- Guido van Rossum, CPython Core Developer and former BDFL

Carol Willing agreed that the core dev team shouldn’t be looking to remove modules en masse, but should decide what kinds of modules they wanted to admit in the future. Łukasz Langa agreed, and pointed out that it was often hard removing modules even when we wanted to, due to the fact that “the standard library is a huge import cycle”.

Where do we go now?

Cannon himself put forward two possible answers to his question, before tossing it out to the audience:

1. The standard library should contain everything required to bootstrap an installer.
2. The standard library should make it easy for beginners to be able to write scripts without installing anything.

The conversation was free-flowing, but a common point of consensus among the attendees was that the standard library should focus on tools and utilities that allow users to write better Python code. Hynek Schlawack cited dataclasses as an example of a module that made writing classes much less painful, and generally led to them writing better code as a result. (Schlawack is the author of the attrs library, the third-party inspiration for dataclasses, which itself is still going strong.) Filipe Laíns agreed, arguing that the core dev team should focus on building business implementations for third-party libraries to build on top of.

“The default answer for ‘Should this be in the standard library?’ should be ‘No’, but we should bless smaller utilities that help people write better Python code” 

-- Antonio Cuni, HPy Core Developer

There was a certain amount of regret in the air about modules that perhaps should never have been added to the standard library, and had proved themselves to be significant maintenance burdens in the years since, but could now never be removed. tkinter, it was universally agreed, was the primary example here; possibly multiprocessing also.

Guido van Rossum pondered whether asyncio should ever have been added to the standard library, remarking that it had been difficult to evolve asyncio while it was in the standard library, and had possibly been added before it was “fully baked”. The ssl integration had probably been a mistake, he said, and should have been left to third parties.

Łukasz Langa noted that modules such as asyncio and typing, which had continued to evolve rapidly after being added to the standard library, had helped spur new syntax changes to Python that had been to the language’s betterment. Without asyncio in the standard library, Langa argued, we would probably never have adopted the async/await syntax that is now the foundation of asynchronous Python programming.

Zac Hatfield-Dods, maintainer of several prominent third-party packages, said that different standard-library packages had different impacts on the Python ecosystem. Pytest, one of the libraries he maintains, had managed to flourish and find success despite the existence of unittest in the standard library. But another library he helps out with, the asynchronous Trio framework, had struggled to attract users while asyncio had been part of the standard library. “Nobody supports alternative async implementations,” he complained, despite Trio’s development often being years ahead of where asyncio is. (In the coffee break afterwards, Hatfield-Dods was keen to emphasise that he is, in fact, a fan of asyncio and the work of the asyncio maintainers.)

Zac Hatfield-Dods (left), speaking at the Language Summit
(Photo by Hugo van Kemenade)

Cannon brought up the question of whether a module like pathlib belonged. “It’s just sugar,” he remarked – i.e., hardly a “core utility” or a protocol that allowed people to write better code. But it has nonetheless been one of the more popular additions to the standard library in recent years. Langa again pushed back, arguing that without the addition of pathlib to the standard library, we would never have added os.PathLike, a protocol that had allowed a common interface for describing file-system paths in Python. “A third-party PyPI package wouldn’t have convinced us to make that change,” Langa argued.

Several attendees noted that adding a module to the standard library often made it hard for users to use features added to the module in newer versions of Python, due to CPython’s slow development cycle. One solution could be to provide third-party versions of standard-library modules on PyPI, backporting the latest features of a module to older versions of Python. Thomas Wouters argued that previous attempts at providing these backport modules had often been disastrous. However, Jelle Zijlstra noted that typing_extensions, which backports features from the latest version of the typing module, had been incredibly successful (though it was sometimes hard to maintain).

Overall, there was agreement that the original motivations for a large, “batteries-included” standard library no longer held up to scrutiny. “In the good old days,” Ned Deily reminisced, “We said ‘batteries-included’ because we didn’t have a good story for third-party installation.” But in 2023, installing third-party packages from PyPI is much easier.

Often, Thomas Wouters noted, people preferred using standard-library modules in a corporate setting due to the fact that the installation of any third-party package would require approval from their company’s IT department. But, he noted, this was hardly Python’s problem.

The Python Language Summit 2023: Pattern Matching, __match__, and View Patterns

One of the most exciting new features in Python 3.10 was the introduction of pattern matching (introduced in PEPs 634, 635 and 636). Pattern matching has a wide variety of uses, but really shines in situations where you need to undergo complex destructurings of tree-like datastructures.

That’s a lot of words which may or may not mean very much to you – but consider, for example, using the ast module to parse Python source code. If you’re unfamiliar with the ast module: the module provides tools that enable you to compile Python source code into an “abstract syntax tree” (AST) representing the code’s structure. The Python interpreter itself converts Python source code into an AST in order to understand how to run that code – but parsing Python source code using ASTs is also a common task for linters, such as plugins for flake8 or pylint. In the following example, ast.parse() is used to parse the source code x = 42 into an ast.Module node, and ast.dump() is then used to reveal the tree-like structure of that node in a human-readable form:

>>> import ast
>>> source = "x = 42"
>>> node = ast.parse(source)
>>> node
<ast.Module object at 0x000002A70F928D80>
>>> print(ast.dump(node, indent=2))
Module(
  body=[
    Assign(
      targets=[
        Name(id='x', ctx=Store())],
      value=Constant(value=42))],
  type_ignores=[])

How does working with ASTs relate to pattern-matching? Well, a function to determine whether (to a reasonable approximation) an arbitrary AST node represents the symbol collections.deque might have looked something like this, before pattern matching…

import ast

# This obviously won't work if the symbol is imported with an alias
# in the source code we're inspecting
# (e.g. "from collections import deque as d").
# But let's not worry about that here :-)

def node_represents_collections_dot_deque(node: ast.AST) -> bool:
    """Determine if *node* represents 'deque' or 'collections.deque'"""
    return (
        isinstance(node, ast.Name) and node.id == "deque"
    ) or (
        isinstance(node, ast.Attribute)
        and isinstance(node.value, ast.Name)
        and node.value.id == "collections"
        and node.value.attr == "deque"
    )

But in Python 3.10, pattern matching allows an elegant destructuring syntax:

import ast

def node_represents_collections_dot_deque(node: ast.AST) -> bool:
    """Determine if *node* represents 'deque' or 'collections.deque'"""
    match node:
        case ast.Name("deque"):
            return True
        case ast.Attribute(ast.Name("collections"), "deque"):
            return True
        case _:
            return False

I know which one I prefer.

For some, though, this still isn’t enough – and Michael “Sully” Sullivan is one of them. At the Python Language Summit 2023, Sullivan shared ideas for where pattern matching could go next.

---

Playing with matches (without getting burned)

Sullivan’s contention is that, while pattern matching provides elegant syntactic sugar in simple cases such as the one above, our ability to chain destructurings using pattern matching is currently fairly limited. For example, say we want to write a function inspecting Python AST that takes an ast.FunctionDef node and identifies whether the node represents a synchronous function with exactly two parameters, both of them annotated as accepting integers. The function would behave so that the following holds true:

>>> import ast
>>> source = "def add_2(number1: int, number2: int): pass"
>>> node = ast.parse(source).body[0]
>>> type(node)
<class 'ast.FunctionDef'>
>>> is_function_taking_two_ints(node)
True

With pre-pattern-matching syntax, we might have written such a function like this:

def is_int(node: ast.AST | None) -> bool:
    """Determine if *node* represents 'int' or 'builtins.int'"""
    return (
        isinstance(node, ast.Name) and node.id == "int"
    ) or (
        isinstance(node, ast.Attribute)
        and isinstance(node.value, ast.Name)
        and node.value.id == "builtins"
        and node.attr == "int"
    )

def is_function_taking_two_ints(node: ast.FunctionDef) -> bool:
    """Determine if *node* represents a function that accepts two ints"""
    args = node.args.posonlyargs + node.args.args
    return len(args) == 2 and all(is_int(node.annotation) for node in args)

If we wanted to rewrite this using pattern matching, we could possibly do something like this:

def is_int(node: ast.AST | None) -> bool:
    """Determine if *node* represents 'int' or 'builtins.int'"""
    match node:
        case ast.Name("int"):
            return True
        case ast.Attribute(ast.Name("builtins"), "int"):
            return True
        case _:
            return False

def is_function_taking_two_ints(node: ast.FunctionDef) -> bool:
    """Determine if *node* represents a function that accepts two ints"""
    match node.args.posonlyargs + node.args.args:
        case [ast.arg(), ast.arg()] as arglist:
            return all(is_int(arg.annotation) for arg in arglist)
        case _:
            return False

That leaves a lot to be desired, however! The is_int() helper function can be rewritten in a much cleaner way. But integrating it into the is_function_taking_two_ints() is… somewhat icky! The code feels harder to understand than before, whereas the goal of pattern matching is to improve readability.

Something like this, (ab)using metaclasses, gets us a lot closer to what it feels pattern matching should be like. By using one of Python’s hooks for customising isinstance() logic, it’s possible to rewrite our is_int() helper function as a class, meaning we can seamlessly integrate it into our is_function_taking_two_ints() function in a very expressive way:

import abc
import ast

class PatternMeta(abc.ABCMeta):
    def __instancecheck__(cls, inst: object) -> bool:
        return cls.match(inst)

class Pattern(metaclass=PatternMeta):
    """Abstract base class for types representing 'abstract patterns'"""
    @staticmethod
    @abc.abstractmethod
    def match(node) -> bool:
        """Subclasses must override this method"""
        raise NotImplementedError

class int_node(Pattern):
    """Class representing AST patterns signifying `int` or `builtins.int`"""
    @staticmethod
    def match(node) -> bool:
        match node:
            case ast.Name("int"):
                return True
            case ast.Attribute(ast.Name("builtins"), "int"):
                return True
            case _:
                return False

def is_function_taking_two_ints(node: ast.FunctionDef) -> bool:
    """Determine if *node* represents a function that accepts two ints"""
    match node.args.posonlyargs + node.args.args:
        case [
            ast.arg(annotation=int_node()), 
            ast.arg(annotation=int_node()),
        ]:
            return True
        case _:
            return False

This is still hardly ideal, however – that’s a lot of boilerplate we’ve had to introduce to our helper function for identifying int annotations! And who wants to muck about with metaclasses?

A slide from Sullivan's talk

---

A __match__ made in heaven?

Sullivan proposes that we make it easier to write helper functions for pattern matching, such as the example above, without having to resort to custom metaclasses. Two competing approaches were brought for discussion.

The first idea – a __match__ special method – is perhaps the easier of the two to immediately grasp, and appeared in early drafts of the pattern matching PEPs. (It was eventually removed from the PEPs in order to reduce the scope of the proposed changes to Python.) The proposal is that any class could define a __match__ method that could be used to customise how match statements apply to the class. Our is_function_taking_two_ints() case could be rewritten like so:

class int_node:
    """Class representing AST patterns signifying `int` or `builtins.int`"""
    # The __match__ method is understood by Python to be a static method,
    # even without the @staticmethod decorator,
    # similar to __new__ and __init_subclass__
    def __match__(node) -> ast.Name | ast.Attribute:
        match node:
            case ast.Name("int"):
                # Successful matches can return custom objects,
                # that can be bound to new variables by the caller
                return node
            case ast.Attribute(ast.Name("builtins"), "int"):
                return node
            case _:
                # Return `None` to indicate that there was no match
                return None

def is_function_taking_two_ints(node: ast.FunctionDef) -> bool:
    """Determine if *node* represents a function that accepts two ints"""
    match node.args.posonlyargs + node.args.args:
        case [
            ast.arg(annotation=int_node()), 
            ast.arg(annotation=int_node()),
        ]:
            return True
        case _:
            return False

The second idea is more radical: the introduction of some kind of new syntax (perhaps reusing Python’s -> operator) that would allow Python coders to “apply” functions during pattern matching. With this proposal, we could rewrite is_function_taking_two_ints() like so:

def is_int(node: ast.AST | None) -> bool:
    """Determine if *node* represents 'int' or 'builtins.int'"""
    match node:
        case ast.Name("int"):
            return True
        case ast.Attribute(ast.Name("builtins"), "int"):
            return True
        case _:
            return False

def is_function_taking_two_ints(node: ast.FunctionDef) -> bool:
    """Determine if *node* represents a function that accepts two ints"""
    match node.args.posonlyargs + node.args.args:
        case [
            ast.arg(annotation=is_int -> True),
            ast.arg(annotation=is_int -> True),
        ]
        case _:
            return False

---

Match-maker, match-maker, make me a __match__…

A slide from Sullivan's talk

The reception in the room to Sullivan’s ideas was positive; the consensus seemed to be that there was clearly room for improvement in this area. Brandt Bucher, author of the original pattern matching implementation in Python 3.10, concurred that this kind of enhancement was needed. Łukasz Langa, meanwhile, said he’d received many queries from users of other programming languages such as C#, asking how to tackle this kind of problem.

The proposal for a __match__ special method follows a pattern common in Python’s data model, where double-underscore “dunder” methods are overridden to provide a class with special behaviour. As such, it will likely be less jarring, at first glance, to those new to the idea. Attendees of Sullivan’s talk seemed, broadly, to slightly prefer the __match__ proposal, and Sullivan himself said he thought it “looked prettier”.

Jelle Zijlstra argued that the __match__ dunder would provide an elegant symmetry between the construction and destruction of objects. Brandt Bucher, meanwhile, said he thought the usability improvements weren’t significant enough to merit new syntax.

Nonetheless, the alternative proposal for new syntax also has much to recommend it. Sullivan argued that having dedicated syntax to express the idea of “applying” a function during pattern matching was more explicit. Mark Shannon agreed, noting the similarity between this idea and features in the Haskell programming language. “This is functional programming,” Shannon argued. “It feels weird to apply OOP models to this.”

---

Addendum: pattern-matching resources and recipes

In the meantime, while we wait for a PEP, there are plenty of innovative uses of pattern matching springing up in the ecosystem. For further reading/watching/listening, I recommend:

• “A perfect match: The history, design and future of Python’s structural pattern matching” – A talk by Brandt Bucher at PyCon 2022
• “Structural Pattern Matching in the Real World” – A talk by Raymond Hettinger at Pycon Italia 2022
• RegexMatcher: a class integrating pattern matching with Python’s re module. A 2022 Advent of Code solution by Ned Batchelder.
• approximately: A way to compare float and complex numbers using pattern matching, while avoiding the perils of floating-point arithmetic. A StackOverflow Q&A by Raymond Hettinger.
• “A few related schemes for implementing view patterns in Python”: A gist by Michael Sullivan (from February 2023)