Tuesday, May 21, 2019

Petr Viktorin: Extension Modules And Subinterpreters

When a Python subinterpreter loads an extension module written in C, it tends to unwittingly share state with other subinterpreters that have loaded the same module, unless that module is written very carefully. Petr Viktorin addressed the Python Language Summit to describe the problem in detail and propose a cleaner isolation of subinterpreters.

Read more 2019 Python Language Summit coverage.

Python-Based Libraries Use Subinterpreters For Isolation


Python can run several interpreter instances in a single process, keeping each subinterpreter relatively isolated from the others. There are two ways this feature could be used in the future, but both require improvements to Python. First, Python could achieve parallelism by giving each subinterpreter its own Global Interpreter Lock (GIL) and passing messages between them; Eric Snow has proposed this use of subinterpreters in PEP 554.

Another scenario is when libraries happen to use Python as part of their implementation. Viktorin described, for example, a simulation library that uses Python and NumPy internally, or a chat library that uses Python and asyncio. It should be possible for one application to load multiple libraries such as this, each of which uses a Python interpreter, without cross-contamination. This use case was the subject of Viktorin’s presentation. The problem, he said, is that “CPython is not ready for this,” because it does not properly manage global state.

There Are Many Kinds Of Global State


Viktorin described a hierarchy, or perhaps a tree, of kinds of global state in an interpreter.

Process state: For example, open file descriptors.

Runtime state: The Python memory allocator’s data structures, and the GIL (until PEP 554).

Interpreter state: The contents of the "builtins" module and the dict of all imported modules.

Thread state: Thread locals like asyncio’s current event loop; fortunately this is per-interpreter.

Context state: Implicit state such as decimal.context.

Module state: Python variables declared at file scope or with the “global” keyword, which in fact creates module-local state.


Module State Behaves Surprisingly


With a series of examples, Viktorin demonstrated the subtle behavior of module-level state.

To begin with a non-surprising example, a pure-Python module’s state is recreated by re-importing it:

import enum
old_enum = enum
del sys.modules['enum']
import enum
old_enum == enum  # False

But surprisingly, a C extension module only appears to be recreated when it is re-imported:

import _sqlite3
old_sqlite3 = _sqlite3
del sys.modules['_sqlite3']
import _sqlite3
old_sqlite3 == _sqlite3 # False

The last line seems to show that the two modules are distinct, but as Viktorin said, “This is a lie.” The module’s initialization is not re-run, and the contents of the two modules are shared:

old_sqlite3.Error is _sqlite3.Error # True

It is far too easy to contaminate other subinterpreters with these shared contents—in effect, a C extension’s module state is therefore a process global state.

Modules Must Be Rewritten Thoughtfully


C extensions written in the new style avoid this problem with subinterpreters. Not all C extensions in the standard library are updated yet; Christian Heimes commented that the ssl module must be ported to the new style of initialization. Although it is simple to find modules that must be ported, the actual porting requires thought. Coders must meticulously distinguish among different kinds of global state. C static variables are process globals, PyState_FindModule returns an interpreter-global reference to a module, and PyModule_GetState returns module-local state. Each nugget of module data must be deliberately placed at one of the levels in the hierarchy.

As an example of how tricky this is, Viktorin pointed out a bug in the csv module. If it is imported twice, exception-handling breaks:

import _csv
old_csv = _csv
del sys.modules['_csv']
import _csv
try:
    # Pass an invalid array to reader(): should be a string, not 1.
    list(old_csv.reader([1]))
except old_csv.Error:
    # The exception clause should catch the error but doesn't.
    pass

The old_csv.reader function ought to raise an instance of old_csv.Error, which would match the except clause. In fact, the csv module has a bug. When it is re-imported it overwrites interpreter-level state, including the _csv.Error type, instead of keeping its state at the module-local level.

Audience members agreed this was a bug, but Viktorin insists that this particular bug is merely a symptom of a larger problem: it is too hard to write properly isolated extension modules. Viktorin and three coauthors have proposed PEP 573 to ease this problem, with special attention to exception types.

Viktorin advised all module authors to keep state at the module level. He recognized that this is not always possible: for example, the Python standard library’s readline module wraps the C readline library, which has global hooks. These are necessarily process-global state. He asked the audience, how should this scenario be handled? Should readline error if it is imported in more than one subinterpreter? He said, “There’s some thinking to do.” In any case, CPython needs a good default.

The correct way to code a C extension is to use module-local state, and that should be the most obvious place to store state from C. It seems to Viktorin that the newest style APIs do emphasize module-local state as he desires, but they are not yet well-known.

Further reading:

PEP 384 (3.2): Defining a Stable ABI

PEP 489 (3.5): Multi-phase extension module initialization

PEP 554 (3.9): Multiple Interpreters in the Stdlib

PEP 573 (3.9): Module State Access from C Extension Methods

Not a PEP yet: CPython C API Design Guidelines (layers & rings)

Saturday, May 18, 2019

Scott Shawcroft: History of CircuitPython



Scott Shawcroft is a freelance software engineer working full time for Adafruit, an open source hardware company that manufactures electronics that are easy to assemble and program. Shawcroft leads development of CircuitPython, a Python interpreter for small devices.

The presentation began with a demo of Adafruit’s Circuit Playground Express, a two-inch-wide circular board with a microcontroller, ten RGB lights, a USB port, and other components. Shawcroft connected the board to his laptop with a USB cable and it appeared as a regular USB drive with a source file called code.py. He edited the source file on his laptop to dim the brightness of the board’s lights. When he saved the file, the board automatically reloaded the code and the lights dimmed. “So that's super quick,” said Shawcroft. “I just did the demo in three minutes.”

Read more 2019 Python Language Summit coverage.

CircuitPython Is Optimized For Learning Electronics

The history of CircuitPython begins with MicroPython, a Python interpreter written from scratch for embedded systems by Damien George starting in 2013. Three years later, Adafruit hired Shawcroft to port MicroPython to the SAMD21 chip they use on many of their boards. Shawcroft’s top priority was serial and USB support for Adafruit’s boards, and then to implement communication with a variety of sensors. “The more hardware you can support externally,” he said, “the more projects people can build.”

As Shawcroft worked with MicroPython’s hardware APIs, he found them ill-fitting for Adafruit’s goals. MicroPython customizes its hardware APIs for each chip family to provide speed and flexibility for hardware experts. Adafruit’s audience, however, is first-time coders. Shawcroft said, “Our goal is to focus on the first five minutes someone has ever coded.”

To build a Python for Adafruit’s needs, Shawcroft forked MicroPython and created a new project, CircuitPython. In his Language Summit talk, he emphasized it is a “friendly fork”: both projects are MIT-licensed and share improvements in both directions. In contrast to MicroPython’s hardware APIs that vary by chip, CircuitPython has one hardware API, allowing Adafruit to write one set of libraries for them all.

MicroPython has a distinct standard library that differs from CPython’s: for example, its time functions are in a module named utime with a different feature set from the standard time module. It also ships modules with features not found in CPython’s standard library, such as advanced filesystem management features. In CircuitPython, Shawcroft removed the nonstandard features and modules. This change helps new coders ramp smoothly from CircuitPython on a microcontroller to CPython on a full-size computer, and it makes Adafruit’s libraries reusable on CPython itself.

Another motive for forking was to create a separate community for CircuitPython. In the original MicroPython project’s community, Shawcroft said, “There are great folks, and there's some not-so-great folks.” The CircuitPython community welcomes beginners, publishes documentation suitable for them, and maintains standards of conduct that are safe for minors.

Audience members were curious about CircuitPython’s support for Python 3.8 and beyond. When Damien George began MicroPython he targeted Python 3.4 compliance, which CircuitPython inherits. Shawcroft said that MicroPython has added some newer Python features, and decisions about more language features rest with Damien George.

Minimal Barrier To Entry



Photo courtesy of Adafruit.

Shawcroft aims to remove all roadblocks for beginners to be productive with CircuitPython. As he demonstrated, CircuitPython auto-reloads and runs code when the user saves it; there are two more user experience improvements in the latest release. First, serial output is shown on a connected display, so a program like print("hello world") will have visible output even before the coder learns how to control LEDs or other observable effects.

Second, error messages are now translated into nine languages, and Shawcroft encourages anyone with language skills to contribute more. Guido van Rossum and A. Jesse Jiryu Davis were excited to see these translations and suggested contributing them to CPython. Shawcroft noted that the existing translations are MIT-licensed and can be ported; however, the translations do not cover all the messages yet, and CircuitPython cannot show messages in non-Latin characters such as Chinese. Chinese fonts are several megabytes of characters, so the size alone presents an unsolved problem.

Later this year, Shawcroft will add Bluetooth support for coders to connect their phone or tablet to an Adafruit board and enjoy the same quick edit-refresh cycle there. Touchscreens will require a different sort of code editor, perhaps more like EduBlocks. Despite the challenges, Shawcroft echoed Russell Keith-Magee’s insistence on the value of mobile platforms: “My nieces, they have tablets and phones. They do not have laptops.”

Shawcroft’s sole request for the core developers was to keep new language features simple, with few special cases. First, because each new CPython feature must be reimplemented in MicroPython and CircuitPython, and special cases make this work thorny. Second, because complex logic translates into large code size, and the space for code on microcontrollers is minuscule.


Amber Brown: Batteries Included, But They're Leaking



Amber Brown of the Twisted project shared her criticisms of the Python standard library. This proved to be the day’s most controversial talk; Guido van Rossum stormed from the room during Q & A.

Read more 2019 Python Language Summit coverage.

Applications Need More Than The Standard Library

Python claims to ship with batteries included, but according to Brown, without external packages it is only “marginally useful.” For example, asyncio requires external libraries to connect to a database or to speak HTTP. Brown asserted that there were many such dependencies from the standard library to PyPI: typing works best with mypy, the ssl module requires a monkeypatch to connect to non-ASCII domain names, datetime needs pytz, and six is non-optional for writing code for Python 2 and 3.

Other standard library modules are simply inferior to alternatives on PyPI. The http.client documentation advises readers to use Requests, and the datetime module is confusing compared to its competitors such as arrow, dateutil, and moment.

Poor Quality, Lagging Features, And Obsolete Code


“Python's batteries are leaking,” said Brown. She thinks that some bugs in the standard library will never be fixed. And even when bugs are fixed, PyPI libraries like Twisted cannot assume they run on the latest Python, so they must preserve their bug workarounds forever.

There are many modules that few applications use, but there is no method to install a subset of the standard library. Brown called out the XML parser and tkinter in particular for making the standard library larger and harder to build, burdening all programmers for the sake of a few. As Russell Keith-Magee had described earlier in the day, the size of the standard library makes it difficult for PyBee to run Python on constrained devices. Brown also noted that some standard library modules were optimized in C for Python 3, but had to be reimplemented in pure Python for PyPy to support them.

Brown identified new standard library features that were “too little, too late,” leaving users to depend on backports to use those features in Python 2. For example, socket.sendmsg was added only recently, meaning Twisted must ship its own C extension to use sendmsg in Python 2. Although Python 2 is nearly at its end of life, this only holds for the core developers, according to Brown, and for users, Red Hat and other distributors will keep Python 2 alive “until the goddam end of time.” Brown also mentioned that some itertools code is shown as examples in the documentation instead of shipped as functions in the itertools module.

Guido van Rossum, sitting at the back of the room, interrupted at this moment, “Can you keep to one topic? I'm sorry but this is just one long winding rant. What is your point?” Brown responded that her point was that there are a multitude of problems in the standard library.

Standard Library Modules Crowd Out Innovation


Brown’s most controversial opinion, in her own estimation, is that adding modules to the standard library stifles innovation, by discouraging programmers from using or contributing to competing PyPI packages. Ever since asyncio was announced she has had to explain why Twisted is still worthwhile, and now that data classes are in the standard library Hynek Schlawack must defend his attrs package. Even as standard library modules crowd out other projects, they lag behind them. According to Brown, “the standard library is where code sometimes goes to die,” because it is difficult and slow to contribute code there. She acknowledged recent improvements, from Mariatta Wijaya’s efforts in particular, but Python is still harder to contribute to than PyPI packages.

“So I know a lot of this is essentially a rant,” she concluded, “but it's fully intended to be.”

Discussion


Nick Coghlan interpreted Brown’s proposal as generalizing the “ensurepip” model to ensure some packages are always available but can be upgraded separately from the standard library, and he thought this was reasonable.

Van Rossum was less convinced. He asked again, “Amber, what is your point?” Brown said her point was to move asyncio to PyPI, along with most new feature development. “We should embrace PyPI,” she exhorted. Some ecosystems such as Javascript rely too much on packages, she conceded, but there are others like Rust that have small standard libraries and high-quality package repositories. She thinks that Python should move farther in that direction.

Van Rossum argued instead that if the Twisted team wants the ecosystem to evolve, they should stop supporting older Python versions and force users to upgrade. Brown acknowledged this point, but said half of Twisted users are still on Python 2 and it is difficult to abandon them. The debate at this point became personal for Van Rossum, and he left angrily.

Nathaniel Smith commented, “I'm noticing some tension here.” He guessed that Brown and the core team were talking past each other because the core team had different concerns from other Python programmers. Brown went further adding that because few Python core developers are also major library maintainers, library authors’ complaints are devalued or ignored.

The remaining core developers continued the technical discussion. Barry Warsaw said that the core team had discussed deprecating modules in the standard library, or creating slim distributions with a subset of it, but that it required a careful design. Others objected that slimming down the standard library risked breaking downstream code, or making work for programmers in enterprises that trust the standard library but not PyPI.

Pablo Galindo Salgado was concerned that moving modules from the standard library to PyPI would create an explosion of configurations to test, but in Brown’s opinion, “We are already living that life.” Some Linux and Python distributions have selectively backported features and fixes, leading to a much more complex set of configurations than the core team realizes.

Wednesday, May 15, 2019

Paul Ganssle: Time Zones In The Standard Library

Python boasts that it comes with “batteries included,” but programmers have long been frustrated at one set of missing batteries: the standard library does not include any time zone definitions. The datetime module supports the idea of time zones, but a programmer who wants to know when Daylight Saving Time starts in Cleveland must install a third-party package. Paul Ganssle spoke to the Python Language Summit to offer a solution. Ganssle maintains the PyPI package dateutil, and contributes to the standard library datetime module. He described the state of Python time zone support and how time zone definitions could be added to the standard library.

Read more 2019 Python Language Summit coverage.

Python Comes With Limited Time Zone Support


A time zone is a function that maps a naïve time to an unambiguous Coordinated Universal Time (UTC). Individual time zones can be quite eccentric, so Python does not attempt to define time zone logic, it simply provides an abstract base class TZInfo that is subclassed by implementors. Although there could theoretically be unlimited kinds of time zones, most programmers encounter three concrete types:

1. UTC or a fixed offset from it.

2. Local time.

3. A time zone from the IANA database.

The first of these was added to the standard library in Python 3.2. Ganssle said, “Whenever I teach people about datetimes, it's really nice to be able to say, if you're using Python 3, you can just have a UTC object.” The purpose of Ganssle’s proposal was to add the second and third.

Ambiguous Times


Ganssle explained that when Eastern Daylight Time ends, clocks are set back from 2:00am to 1:00am, thus there are two UTC times that map to 1:30am local time on that day:

>>> NYC = tz.gettz("America/New_York")
>>> dt0 = datetime(2004, 10, 31, 5, 30, tzinfo=tz.UTC)
>>> print(dt0.astimezone(NYC))
2004-10-31 01:30:00-04:00
>>> print((dt0 + timedelta(hours=1)).astimezone(NYC))
2004-10-31 01:30:00-05:00

PEP 495 solved the problem of ambiguous times by adding the “fold” attribute to datetime objects. A datetime with fold=0 is the first occurrence of that local time, the second occurrence has fold=1. With this addition, standard Python provides all the prerequisites for proper time zones, so Ganssle argued they should now be added to the standard library.

How To Maintain The Time Zone Definitions?


IANA time zones are the de facto standard for time zone data, and they ship with many operating systems. Both Ganssle’s dateutil and the competing pytz package use the IANA database as their source of truth. Therefore it would be natural to include the IANA time zones in the Python standard library, but this presents a problem: the IANA database changes every time a government changes a time zone, which occurs as often as 20 times a year. Time zone changes are far more frequent than Python releases.

Ganssle offered two solutions for updating time zone data, and then offered a compromise between them as his actual proposal. The first solution is to rely on the operating system’s time zone database. Python could rely on the system update mechanism to refresh this data, and it would use the same time zone definitions as most other applications. System time zone data is not officially supported on Windows, however, and is not always installed on Linux.

The second solution is to publish IANA time zone definitions as a PyPI package. It could be updated frequently, but the core team would have to invent some way to notify users when it is time to update their time zone data. Plus, it would be risky for Python to use different time zones than the rest of the system.

Ganssle proposed a hybrid: the Python standard library should use the system’s time zone data if possible, otherwise fall back to a PyPI package which would be installed conveniently, analogous to installing pip with “ensurepip” today.

The Local Time Zone


Naïve times in Python are sometimes treated as times in the local time zone, sometimes not. Ganssle showed an example demonstrating that if a programmer converts a naïve time to UTC, Python assumes its original time zone is local:

>>> dt = datetime(2020, 1, 1, 12)
>>> dt.astimezone(timezone.utc)
2020-01-01 17:00:00+00:00

However, adding a naïve time to a UTC time is prohibited:

>>> datetime(2020, 1, 1, 12) - datetime(2020, 1, 1, tzinfo=timezone.utc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't subtract offset-naive and offset-aware datetimes

Ganssle’s dateutil package offers a more thorough implementation of “local time zone”, and he thinks Python programmers would appreciate local times in the standard library. To add them, however, the core team must first handle the astonishing behavior of local times when the system time zone changes. The first surprise is that changing the system time zone has no effect until the Python program calls time.tzset(). (And on Windows, time.tzset() is not available.) The second surprise is that changing system time and then calling time.tzset() changes the UTC offset of existing times created before the change.

Ganssle proposed several ways the standard library could act in this scenario. It could ignore changes to the system time zone while a Python program is running, or it could detect time zone changes but avoid mutating the offsets of existing time objects. He had no opinion about the best outcome.

Conclusion


Ned Deily wondered what Ganssle’s proposal would solve that which pytz does not. Ganssle responded that pytz’s author has stopped maintaining the package because he believes time zones should move to the standard library. Full time zone is a basic feature that should always be available. In Ganssle’s view, however, his own dateutil is a better package to emulate than pytz. “I would take dateutil, clean up some of the rough edges, and propose it as some of the batteries that would be included.”

Łukasz Langa said that he planned, as Python 3.8’s release manager, to issue monthly patch releases, and he thought that should be frequent enough to keep users’ time zone data updated. Russell Keith-Magee said no, North Korea once announced a time zone change with three days’ notice. Other audience members thought this scenario was obscure, and the PEP should not be required to handle such emergencies.

At the end of his talk Ganssle summarized his proposal. He believes that the standard library should support IANA time zones, using the operating system as the source of time zone data or falling back to a PyPI package. There are several options for handling local time zone changes at runtime. The design should be formalized in at least an informational PEP, “if not one where it's contentious and we all hate each other at the end of it.”

Russell Keith-Magee: Python On Other Platforms

Russell Keith-Magee spoke in his capacity as the founder and “Benevolent Dictator For Now” of the BeeWare Project. The project’s slogan is “Write once. Deploy everywhere.” The goal of the BeeWare Project is to run Python applications on Android, iOS, in the browser, even on smart watches, and to distribute Python applications using platform-specific channels like app stores. Keith-Magee described a number of obstacles to this goal, and expressed his hope that the core team would consider these problems when they plan Python’s future.

Read more 2019 Python Language Summit coverage.



Cross-Compilation Is Supported And Tested Poorly

On the server side, x86-64 dominates, but mobile architectures are varied and rapidly changing. iOS alone has encompassed six architectures in recent memory: i386, x86-64, arm6, arm7, arm7s and arm64.

To make matters worse, anyone deploying Python to these architectures must cross-compile, using a desktop or server machine to compile Python and its C extensions, but designating a mobile architecture as the target. “The good news,” said Keith-Magee, is that CPython uses Autoconf, and “Autoconf has really good support for cross compilation baked in. The bad news is that CPython doesn't so much.” Cross compilation is not tested continuously, and so it is broken occasionally in CPython’s build configuration. There is a vendored copy of libffi in the Python source tree, apparently only for supporting the ctypes module on PowerPC, which makes compilation for iOS even more difficult.

Distutils and pip present a tougher challenge. They do not support cross-compilation at all, and assume that the machine on which they run is the target architecture of the C extensions they compile. Neither do they support the “fat binary” format required to support multiple iOS architectures in a single extension module.

OS Differences Break The Test Suite

Python’s assumptions about system calls are violated, in a variety of ways, on mobile and web. The most dangerous pitfalls are fork and spawn — they are provided on iOS but any program that calls them will hang. Large amounts of code in the Python standard library assume the existence of fork and spawn, and even more in the test suite.

Some years ago Keith-Magee provided a patch to fix or disable tests as needed for iOS, but as he recalled, core developer Ned Deily found them “too invasive.” Instead of changing the test suite so pervasively, a new approach might be to skip large chunks of the test suite based on platform limitations.

Python Applications Must Include a Python Installation

Each Python application on a phone or table OS must ship an entire Python installation with it, since Python is not available as a system library, and applications cannot install it in a shared location. “If you've got ten Python apps on your phone,” says Keith-Magee, “you are going to have ten installs of Python.” Python for iOS is a fat binary that supports multiple architectures, weighing around 100 MB in total. Python’s size handicaps browser applications, too, which cannot begin running until the Python runtime has been loaded.

The Javascript community uses “tree shaking”: Javascript applications are distributed with the minimum subset of all their dependencies that are actually required. Keith-Magee proposes a tool that would automatically apply this technique to the Python standard library. Python builtins or portions of the interpreter itself could be jettisoned, too. For example, if an application is distributed as bytecode, then the parser and compiler could be removed. Keith-Magee said, “While I'm sure that someone has written code that uses Python's complex number data type, I am not that person, and in all my apps, the code for complex number handling is dead code.” To write a tool that would remove such components, however, would require some public API for editing the runtime and standard library.

asyncio Does Not Integrate With Mobile GUI Events

Mobile applications are GUIs, and GUI programming is event-based. Python’s asyncio is a natural framework for GUI programming, but it must be adapted to run atop the GUI environment’s own event loop. According to Keith-Magee this is relatively easy on Unix platforms like Linux, macOS, and iOS, with poll-based APIs. He estimates it requires a few hundred lines of code. On Android, Windows, and the web, however, the event model is entirely different, and asyncio integration has been an open bug for years. Keith-Magee said he does not know asyncio internals well enough to propose a solution but welcomes any suggestions.

Only CPython Can Use C Standard Library Modules

Keith-Magee has been experimenting with alternatives to CPython for mobile and web. Replacing the core language implementation is “relatively achievable,” he said, “but the standard library’s another thing.” The C modules in the standard library are only compatible with CPython. Some, like the decimal module, have recently been ported from Python to C, which accelerates them but presents more hindrances for non-CPython interpreters.

Keith-Magee asked the core developers to maintain pure-Python implementations of all standard library modules, for the sake of alternative interpreters. But recognizing how burdensome that would be, he requested as a fallback that the C interface for each extension module be clearly defined.

Setup.py Cannot Build App Store Bundles

An application packaged for a mobile app store requires various metadata, such as a bundle identifier, that is not currently expressible in setup.py. Keith-Magee described his confusion about how to proceed, given his confusion about the direction of Python packaging in general. Should the metadata be in the pyproject.toml file specified by PEP 518? Should he adapt pip or use a custom build tool? He felt that if the core team has any clear vision for the future of Python packaging, “I can tell you from down in the trenches that message isn't getting through.” He requested more information and more opinions about packaging from the core team.

Wish List

The talk concluded with a wish list for adapting Python to mobile and web:

1. Host/target separation testing in CI

2. Host/target separation in distutils/pip

3. Feature gating (especially in the test suite)

4. Unvendoring libffi for macOS

5. Tree (and/or root) shaking

6. AsyncIO support for other eventing styles

7. Reference implementation of modules (or a clear native interface)

8. Clearer communications on packaging

Keith-Magee claims that Python faces an “existential risk” if mobile and web support does not improve. His son, who uses only an iPad for school, asked him, “When can I learn to program like you?” Unless students like him can program Python on their devices, Python risks being left behind by the next generation of programmers.

The 2019 Python Language Summit

The Python Language Summit is a small gathering of Python language implementers, both the core developers of CPython and alternative Pythons, held on the first day of PyCon. The summit features short presentations from Python developers and community members, followed by longer discussions. The 2019 summit is the first held since Guido van Rossum stepped down as Benevolent Dictator for Life, replaced by a five-member Steering Council.

LWN.net covered the proceedings from 2015 to 2018; this year the PSF has chosen to feature summit coverage on its own blog, written by A. Jesse Jiryu Davis.

Individual writeups will be linked from this page as they are posted.

Sessions:

Lightning Talks, Round 1 (pre-selected lightning talks)


Writing Stdlib C Extension Modules in Python? Jukka Lehtosalo and Michael Sullivan

Async REPL and Async-Exec, Mattias Bussonnier

The Night’s Watch Is Fixing the CIs in the Darkness for You, Pablo Galindo Salgado

Asyncio and the Case for Re-Entrancy, Jason Fried

Optimising CPython, or Not, Mark Shannon

The Night’s Watch Is Fixing the CIs in the Darkness for You, Pablo Galindo Salgado

Black Under github.com/python, Łukasz Langa

Half-Hour Sessions


Python on Other Platforms, Russell Keith-Magee



History of CircuitPython, Scott Shawcroft


PEP 581 / PEP 588, Mariatta Wijaya

Feedback on Mentoring Python Contributors, Victor Stinner

Lightning Talks, Round 2 (on-site signup)


SSL Module Updates, Christian Heimes

Let’s Argue About Clinic, Larry Hastings

The C-API, Eric Snow

Python in the Windows Store, Steve Dower

Bors: How the Rust Team Avoids Pablo’s Problems, Nathaniel Smith

Mypyc for Stdlib: Extended Discussion, Michael Sullivan

Python Core Sprints at Bloomberg in September, Pablo Galindo

Status of the Stable ABI, Victor Stinner

Cognitive Encapsulation: Anchor of Working Together, Yarko Tymciurak

Thanks to Bloomberg Engineering for sponsoring the summit.




Wednesday, May 01, 2019

Building the PSF: the Q2 2019 Fundraiser


Thank you to everyone who has donated to our past fundraisers! Donations, memberships, and sponsorships support sprints, meetups, community events, Python documentation, fiscal sponsorships, software development, and community projects.

We can’t do any of this without your financial contributions. We’ve just launched a new fundraiser for 2019! Please donate today and help us meet our goal of $60,000 by May 22nd!

Your donations have IMPACT!


  • The PSF awarded $118,543 in financial aid to 143 PyCon attendees in 2018
  • $324,000 was paid in grants in 2018 to recipients in 51 different countries
  • Donations and fundraisers resulted in $489,152 of revenue. This represents 15% of our total 2018 revenue. 
  • PSF and PyCon sponsors contributed over $1,071K in revenue!

We understand the need for transparency and hope to help our community and stakeholders find necessary information about the PSF in a single place. We’re proud to launch our first ever Annual Report. 2018 was a year of growth for the PSF while still focussing on sustainability for our staff and community. We’re excited to share these data points with you!

Something new this year - the PSF and Jet Brains!



This year we're trying something new. In addition to our regular donation drive, we're partnering with JetBrains to help raise money for the PSF. JetBrains PyCharm & the PSF is happy to announce a 30% discount with all proceeds going to the Python Software Foundation general fund!

Please consider becoming a PSF Supporting Member today. Or simply make a one-time or recurring donation.

If you're attending PyCon this year, you can donate at the PSF booth and get your choice of a limited edition pin or sticker set!

Help spread the word about our fundraiser! Here are some suggested tweets:
===================


==================

Thank you for your support!

The PSF Team



Thursday, April 04, 2019

Update on the Python in Education Proposal Phase

In January when we launched the Python in Education project, we were a bit too ambitious with our timeline. Due to other commitments, we were not able to stick to the original time frame.

Here is the revised timeline:

April 4 - May 9: Request for Proposals phase
May 10 - May 31: Review process
June 1: Notify the accepted proposals
June-August of 2019: Accepted proposal work begins


We'd also like to take this opportunity to inform everyone interested in submitting a proposal that we selected three categories we'd like to see proposals on. Additionally, we'd like to share the evaluation rubric we will use when reviewing each proposal. 

Proposal categories

After reviewing all of the ideas we received in the first phase of this project, we have narrowed the scope of the proposals to:
  1. resources (curriculums, evaluations, studies, multidisciplinary projects)
  2. localization (translations, global currency, global timestamps, etc)
  3. mobile (development on mobile devices)
We ask that if you are considering submitting a proposal that it fall into one of these broad categories.

Evaluation rubric





Every proposal we receive will be evaluated against this rubric. We are looking for proposals that adhere to the PSF's code of conduct, align with the PSF's mission, have international reach, are feasible, and pertain to underrepresented topics.

If anyone has any questions, please contact us at edu-committee@python.org. 


Wednesday, March 27, 2019

PuPPy Presents its 1st Annual Benefit featuring Guido van Rossum

PuPPy Presents its 1st Annual Benefit

A historic discussion of programming language creators about
the past and future of language design.



April 2, 2019, 5:30 - 9:30 PM
The Collective Seattle
400 Dexter Ave N.
Seattle, WA 98109


Featuring Language Creators
James Gosling - Java
Anders Hejlsberg - Turbo Pascal, C#, TypeScript
Guido van Rossum - Python
Larry Wall - Perl


Summary


PuPPy, Seattle's Puget Sound Programming Python user group, presents its 1st annual charity event. The event will feature the creators of C#, Java, Perl, Python, and TypeScript in a conversation about programming language design. 

The charity event brings together this unique group of computer science pioneers, unlike any event held before. These great minds come together for what will surely be a fantastic night of discussion, as the panel delves into the past and future of programming language creation. The event will attract innovators and engineers from Seattle, the nation’s fastest growing technology hub.

The event is a benefit for CSforALL, a non-profit organization that represents more than 500 members who are educators, content providers, funders and researchers who share a vision for all students in the U.S. to learn computer science. CSforALL provides leadership and guidance that helps the K-12 education community implement computer science initiatives, draw from best practices and connect with national organizations to expand access to all students in the U.S.

The event follows in the spirit and culture of PuPPy, the producers of this benefit, as an inclusive community. The very first PuPPy meeting 54 months ago was a mini-conference that featured discussions on allyship and helping more women join the ranks of software professionals. 

Event tickets and further details are available at: http://bdfl-gift.pspython.com.


Speakers


Cyrus Habib - Washington State Lieutenant Governor - Opening remarks

Cyrus Habib was elected Washington State’s 16th Lieutenant Governor in November 2016 at the age of 35. He had previously been elected to the State House of Representatives in 2012 and the State Senate in 2014, where he was Democratic Whip and a member of the Democratic leadership team. As Lt. Governor, he is President of the State Senate, serves as Acting Governor whenever Governor Inslee leaves the state, and oversees an agency whose key issues include economic development, trade, and higher education.
A three-time cancer survivor, Lt. Governor Habib has been fully blind since age eight. His parents immigrated to the U.S. from Iran before he was born, and he is the first and only Iranian-American to hold statewide elected office in the United States.

Carol Willing - Moderator

Carol Willing serves as a Steering Council member for Project Jupyter. She received the 2017 ACM Software System Award for Jupyter's development. She is also a member of the inaugural Python Steering Council, a Python Software Foundation Fellow and former Director; a core developer on CPython, Jupyter, nteract, AnitaB.org’s open source projects, and PyLadies; a co-organizer of PyLadies San Diego and San Diego Python User Group; an independent developer of hardware and software. Weaving her love of art, music, and nature with wearable soft circuits, she is developing an open hardware project to assist an in-home caregiver with gentle, compassionate support of a loved one with Alzheimer’s.

James Gosling - Java

James A. Gosling, O.C., Ph.D. (born May 19, 1955, near Calgary, Alberta, Canada) is a famous software developer, best known as the father of the Java programming language.

In 1977, James Gosling received a B.Sc in Computer Science from the University of Calgary. In 1983, he earned a Ph.D. in Computer Science from Carnegie Mellon University, and his doctoral thesis was titled "The Algebraic Manipulation of Constraints". While working towards his doctorate, he wrote a version of emacs (gosmacs), and before joining Sun Microsystems he built a multi-processor version of Unix[1] while at Carnegie Mellon University, as well as several compilers and mail systems. Since 1984, Gosling has been with Sun Microsystems.

He is generally credited as the inventor of the Java programming language in 1991. He did the original design of Java and implemented its original compiler and virtual machine. For this achievement, he was elected to the United States National Academy of Engineering. He has also made major contributions to several other software systems, such as NeWS and Gosling Emacs. He also co-wrote the "bundle" program, a utility thoroughly detailed in Brian Kernighan and Rob Pike's book, "The Unix Programming Environment".

Anders Hejlsberg - Turbo Pascal, C#, TypeScript

Anders Hejlsberg is a Microsoft Technical Fellow and has been designing and implementing programming languages and development tools for over 35 years. Anders is the lead architect of the TypeScript open-source project and the original designer of the C# programming language. Before joining Microsoft in 1996, Anders was a Principal Engineer at Borland International. As one of the first employees of Borland, he was the original author of Turbo Pascal and later worked as the Chief Architect of the Delphi product line. Anders studied Engineering at the Technical University of Denmark.

Guido van Rossum - Python

Guido van Rossum is the creator of Python, one of the major programming languages on and off the web. Recently Guido retired as Benevolent Dictator For Life (“BDFL”) of Python, a title seemingly stolen from a Monty Python skit. Details of his decision were featured in an Economist article. Guido thankfully has joined the Python Steering Council. This five-person group will give guidance to the future roadmap of the Python programming language.

Van Rossum moved from the Netherlands to the USA, in 1995. He met his wife after his move. Until July 2003 they lived in the northern Virginia suburbs of Washington, DC with their son Orlijn, who was born in 2001. They then moved to Silicon Valley where Guido worked for a variety of companies including Google in the past and currently at Dropbox (spending 50% of his time on Python!).


Larry Wall - Perl

Larry Wall was educated at various places including the Cornish School of Music, the Seattle Youth Symphony, Seattle Pacific University, Multnomah School of the Bible, SIL International, U.C. Berkeley, and UCLA. Though trained primarily in music, chemistry, and linguistics, Larry has been working with computers for the last 40 years or so. He is most famous for writing rn, patch, and the Perl programming language, but prefers to think of himself as a cultural hacker whose vocation in life is to bring a bit of joy into the dreary existence of programmers. For various definitions of “work for”, Larry has worked for Seattle Pacific, MusiComedy Northwest, System Development Corporation, Burroughs, Unisys, the NSA, Telos, ConTel, GTE, JPL, NetLabs, Seagate, Tim O’Reilly, the Perl Foundation, Broadcom, and himself. He is currently serving as Artist in Residence for Craigslist.

About CSforAll

https://www.csforall.org/media-faq/

CSforALL, shorthand for “Computer Science for All,” is the central hub for the K-12 national computer science education movement. CSforALL is the community organizer of school districts, nonprofits, government agencies and corporations that share the goal of rigorous, inclusive and sustainable CS education in the U.S.


About PuPPy

https://www.pspython.com/app/

PuPPy is a fun and friendly user group dedicated to proliferating a diverse and talented Python community in the Puget Sound region. We are devoted to exploring Python-based programming knowledge, embracing new and experienced members from all walks of life, and helping those members to achieve their professional goals.

Event Leadership


Ruthe Farmer – Event Chair - Chief Evangelist for CSforALL
Carol Willing – Moderator -- Steering Council member and developer for Project Jupyter and Python
Lorena Mesa – Vice Chair -- GitHub Data Engineer - Software Intelligence Systems | PyLadies Chicago Organizer | Python Software Foundation Director
Eloisa Tran -- Fundraising Chair -- Founder of the Women in Data Science WiDS, Data Scientist for City of Bellevue

Wednesday, March 13, 2019

Commencing Security, Accessibility, and Internationalization Improvements to PyPI for 2019

The Python Software Foundation (PSF) and PSF Packaging Working Group are excited to announce that the first round of slated improvements to PyPI for 2019 are underway. This Open Technology Fund funded project will bring improvements to the accessibility and security of the service. You can read more about the scope of this project in our request for proposals document.

We are excited to have two returning contractors from the MOSS funded work that brought the full rewrite of PyPI into production.

Kabu Creative

Responsible for the user interface and user experience of PyPI, Kabu Creative will be fulfilling those aspects of new features for the project. Additionally, we are excited for their work on auditing and improving the accessibility of PyPI's web user interface.

Changeset Consulting, LLC

With experience in project management, communications, and contributions for projects and teams across the realm of open source, Changeset Consulting will be responsible for helping to lead the project to completion. Along the way Changeset Consulting will also be performing reporting, communications, and outreach to help keep the Python community up to speed on how the project is progressing.

We are also welcoming a new contractor to the team to complete this project.

Trail of Bits

Bringing their experience securing organizations and products, Trail of Bits will be handling the backend development related to the security milestones of the project, as well as the backend development necessary for implementing internationalization of the PyPI user interface.


The PSF looks forward to sharing more here as features are developed and deployed to PyPI. Subscribe to pypi-announce for announcements of big changes to PyPI, and follow this blog for updates as the work progresses.

Friday, February 22, 2019

The North Star of PyCascades, core Python developer Mariatta Wijaya, receives the 2018 Q3 Community Service Award

We in the Python community have a deep appreciation for the volunteers who organize, promote, and write the language. A phrase that has become a cornerstone of our community, afterall, ‘Come for the language, stay for the community’ (derived from Python core developer Brett Cannon’s opening remarks at PyCon 2014), reflects the passion of our community and more so of the countless volunteers building our community.

One volunteer who has been steadfast in actively building the Python community - from her contributions to CPython to her work as an organizer and co-chair of PyCascades 2018 and more - is Mariatta Wijaya. We at the Python Software Foundation are pleased to name Mariatta Wijaya as a 2018 Q3 Community Service Award recipient:

RESOLVED, that the Python Software Foundation award the Q3 2018 Community Service Award to Mariatta Wijaya for her contributions to CPython, diversity efforts for the Python Core Contributor team, and her work on PyCascades.

Come talk! The path to becoming a Core Python Developer


At Montreal PyCon 2015, Guido Van Rossum delivered the closing keynote during which Guido issued a public ask, “I want at least two female Python core developers in the next year ... and I will try to train them myself if that's what it takes. So come talk to me." Consequently, Mariatta did just that, she reached out to Guido after PyCon 2016 to learn more about starting in Python core development. Mariatta recalls, “I hadn’t contributed to open source [yet] and I wanted to know how to start”. Guido recommended some ways for Mariatta to start including reviewing the dev guide, looking at open issues and joining and introducing herself on the Python dev mailing list .

Following Guido’s advice, Mariatta “read the issues [to] see if there is anything I can help with, anything that interests me ... [when I learned that] Brett was starting migration [of Python] to GitHub”. As an engineer at Zapier, Mariatta has a background in web development so the migration provided an initial issue she could begin to explore. Mariatta has since contributed to several bots that improve the workflows for Python contributors and core developers, reviewed and merged 700+ PRs to Python, and is Co-Chair of the Language Summit in 2019 and 2020. Some examples of bots she has written include cherry-picker, a “tool used to backport CPython changes from master into one or more of the maintenance branches”. Additionally, Mariatta is the author of PEP-581: Using GitHub Issues for CPython. Her motivations behind this PEP again come back to improving the core Python development processes, “I think it will be more beneficial to use an out of the box issue tracker like GitHub as it will allow core developers to focus on developing and contributing”.

A role model for us all: Increasing the diversity of the Core Python Development team


The recent departure of Guido as BDFL and the subsequent discussion about the future governance of core Python led to several suggestions, with the Steering Council ultimately becoming the chosen model. Core Python developer Victor Stinner along with several others nominated Mariatta to the Steering Council. In Victor’s nomination he explains, “Mariatta became the first woman core developer in Python [in 2017]. She is actively sharing her experience to encourage people from underrepresented groups to contribute to Python.” The work required to become a core developer is laborious yet Mariatta has continuously gone the extra mile to lead by example and be active in public outreach. “Mariatta is my role model for mentoring and diversity which is helping a lot to get more people involved in Python,” Victor adds. Python core developer and Steering Council member Carol Willing echoed this sentiment sharing, “Mariatta works to share Python and its possible uses with others. Her blend of hard work, enthusiasm, and caring have welcomed many into the Python Community,”

Mariatta’s PyCon 2018 talk titled, “What is a Core Python Developer is an ideal example of Mariatta’s dedication to building a more diverse core Python team. Beginning with a question, “do you use f’strings” (Mariatta is a known avid fan of f’strings, she even has stickers for them) Mariatta dives into a talk about what the pathway is for core (and contributing) developers ultimately commenting on the very real, stark gender imbalance within the core team, “We have 848 contributors to Python, less than 10 are women. We have 89 core developers, only 2 are women ... This is real but this is also wrong. This is not the right representation of our community”.  While this number is starting to change as more women are promoted to core development (Cheryl Sabella’s promotion this week ups the number of women core developers up to 5 out of 97), Mariatta has continued to be a champion and advocate for diversity and inclusion in the core development team. Even the captionist in Mariatta's PyCon 2018 (seen below in tweets) talk captured their appreciation for Mariatta's dedication.







The North Star of PyCascades


Outside of her contributions to CPython, Mariatta has been an active organizer with PyCascades - a regional Python conference now about to kick off its second conference this week. The inaugural 2018 conference, held in Mariatta’s hometown Vancouver, Canada, introduced a a single track format with 30 minute talks, no question and answer, and includes lightning talks. Inspired by the single track format of Write the Docs and DjangoCon Europe, this format was not only an easier way to get a new conference off the ground but, as Mariatta observed, “is able to give [speakers] the large audience they deserve”. This format also makes it easier for attendees to navigate.

Co-Chairing the conference with Mariatta in 2018, Seb Vetter remarked, “Mariatta has been THE driving force behind PyCascades in the inaugural year”. As a co-chair Mariatta helped respond to many last minute issues such as when, the day before the conference at 10:00am local time, Guido informed the organizers he was unable to obtain a visa to travel to speak at PyCascades. Within a few hours, the team setup for Guido to speak remotely, had sent him a badge, when the team learned Guido would be able to attend after all! “When we found out he's coming, we printed one more badge for him. That's why he has multiple badges,” Mariatta explained. Juggling many changing priorities is the life of an organizer. Yet each decision made, “she ensured … considered the potential impact on the diversity of the conference,” Seb remembered adding, “[Mariatta] seems to have an endless stream of enthusiasm and energy and was our North Star for doing everything we could to make it as inclusive for attendees as possible”. The idea of Mariatta acting as a North Star was echoed by PyCascades organizer Don Sheu adding, “she gives voice to folks that aren’t sufficiently represented in tech … [as a part of] PyCascades founding team, Mariatta’s influence is creating a safe environment”.

With PyCascades 2019 happening in Seattle this upcoming weekend (February 23 - 24), Mariatta is again contributing as an organizer.

What do f-string stickers and food have in common? Mariatta’s love of them!


Outside of Python, when asked what else Mariatta likes to do she simply responded, “I love food!”. And her favorite food? Asian cuisine.

#IceCreamSelfie at North Bay Python 2018.
Source: https://mariatta.ca/img/ics-northbaypython-2018.jpg.

If you happen to see Mariatta at an event, say hi. Maybe she’ll have f-string sticker for you!



Monday, February 11, 2019

Python Community service award Q3: Mario Corchero




The PSF community service awards go to those individuals whose work and commitment complement and strengthen the PSF mission: to support and facilitate the growth of a diverse global Python community. So when thinking about individuals that go above and beyond to support the global community Mario Corchero is a name that comes easily to mind.


Not only is Mario a Senior Software Engineer for Bloomberg but he also devotes incredible amounts of his time to organise PyCon ES (Spain), PyLondinium, and more recently, the Spanish speaking track of PyCon: Las Pycon Charlas.

Mario is the true embodiment of the Python community spirit and for this reason, the Python Software Foundation has awarded Mario Corchero with the Q3 2018 Community Service Award.

RESOLVED, that the Python Software Foundation award the Q3 2018 Community Service Award to Mario Corchero for helping organize PyLondinium, the PyCon Charlas track, and PyCon Spain.


Mario's contributions to the Python Community


PyConES


With the growing popularity and global adoption of Python there also comes the need to bring together diverse community groups. Although large events such as PyCon US are incredibly important in bringing these groups together, these are not always accessible to the whole community. Smaller, localized events such as Python ES, France, Namibia, Colombia, and many many others help with the goal of bringing cohesion to the global community.

According to David Naranjo (co-organiser of PyConES), PyConES was the first event of this kind that he and Mario attended together. They loved it so much that while at PyConES16 they decided to submit an application to organise and bring this event to their region: Extremadura.

On top of the many challenges that come with organising an event of this type (i.e. drafting the programme, getting talks accepted, running the event on the day), they have faced an additional layer of complexity: neither of them lives in the region anymore.

This has made the organisation of PyConES a true community effort: from the organising committee to the sponsors and the volunteers that work together to make this a huge success. PyConEs is now a  staple Python event in Europe with more than 600 attendees every year, and it owes its success in a great deal to Mario’s efforts.



PyLondinium


A year after organising his first PyConES, Mario embarked on yet another journey: the organisation of PyLondinium. An event focused on showcasing the many use cases of Python as opposed to other events such as the PyData events.


PyLondinium is not only focused on bringing together the Python community but also to raise money for the PSF and its programmes around the world. In this particular case, Bloomberg, a long-time Python supporter, has played an important role in the success of the event. Not only do they host the event at their Europe headquarters in the heart of London but they also help to cover some of the costs as the main event sponsor, keeping the ticket prices at an affordable level.



Pylondinium 2018

Accessibility for the wider community


As a passionate community builder, from a non-English speaking country, localization and accessibility of the Python language is something that matters to Mario. Most of the coding resources out in the world are written in English, which can be a barrier to those whose primary language is not English or simply do not speak the language at all. That is why when he was presented with the opportunity to chair the Spanish track of PyCon US 2017 (Las PyCon Charlas) he did so wholeheartedly, embarking into yet another community journey alongside PSF Director Naomi Ceder.

Again, like his other endeavours, Las Charlas was an absolute success. It gathered people from all over from Latin America and Spain for a full day of talks in Spanish on such topics as machine learning, astronomy and security. In fact, it was such a success that the Charlas is back this coming year and the organisers are already receiving talks submissions (for more details visit https://us.pycon.org/2019/speaking/).



PyCon Charlas 2018


When asked why he organises all of these events, his answer is rather simple and honest. It is usually driven by a ‘how come no one is doing this yet?’" says Mario. But when digging deeper it becomes evident that Mario’s motivations lie in bringing the community together and nurturing it. Mario is extremely dedicated to the community and helping others to get involved. From creating Spanish tracks for PyCon USA or creating events serving specific areas or regions, Mario is constantly finding ways to bring Pythonistas together.