The Case for a Second Developer-in-Residence for Python

As the currently serving sole developer in residence, I’m often asked if there will be more people holding the same position in the future. I strongly believe there should be and that it’s crucial to the long term success of this role. The only open matter is finding sustainable sponsorship for the position.

The current developer in residence

My day-to-day work revolves around an array of maintenance tasks for the Python Software Foundation with focus on CPython. Since I started in July 2021, I’ve done among others:

• PR review and merging: 627 merges to CPython that lead to closing of 276 issues on the bug tracker, and many more code reviews on Github;
• release management for the 3.8 and 3.9 branches as well as release notes and announcements for other releases;
• following the Python security response team reports that lead to several security releases of Python;
• following the buildbot fleet status and reacting to failures, including maintenance of the only buildbot that runs big memory tests;
• project management of the transition from our previous custom issue tracker to Github Issues;
• migration from a previous custom CLA management bot to EdgeDB CLA bot;
• co-administering discuss.python.org including responding to moderation requests;
• co-administering core Python Discord;
• co-chairing the Python Language Summit at PyCon US;
• reviewing talk submissions on the Program Committee for PyCon US;
• facilitating cooperation with other significant Python projects: HPy, PyPy, nogil;
• public speaking (5 events in 2021, 4 events in 2022).

The missing big picture

While I find this work fulfilling and there’s always more things for me to do that other contributors suggest, one facet of the work can overshadow another. I cannot be in all places at once. Most importantly, while removing obstacles for other core developers (often volunteers) is indeed where we should put paid effort, I sometimes get asked: what’s your big project?

At this point I cannot say I had any large personal contribution over the past 18 months of work on CPython, which is ironic, given that I spent more time on it than in the preceding 11 years of core development combined. I had a few attempts at larger changes but inevitably the small busywork eats up my attention.

Adding another developer in residence more than doubles the position’s positive impact

It’s worth noting that the codebase of CPython is over a million lines of code now and even working on it full time does not mean a single person groks it all. That means that what you’re getting from Łukasz, the Developer in Residence, is something else than what you’d get from Magdalena, the Developer in Residence.

That alone means it would be worth having another person with a complementing skill set. But I believe there’s more.

Compared to working solo, having a team of two people paid full time to improve the developer experience for the rest of the core contributors, would allow us to take on larger sweeping projects. What we would end up doing would be definitely consulted with the Steering Council and we would take suggestions from the role’s sponsor. But there’s many possibilities!

We could add official build support for a new platform like iOS. We could improve test coverage of CPython tests, including coverage of trickier bits like the platform-specific code paths, C code, or code involved in CPython’s interpreter startup. We could revamp the buildbot master server to be more performant. We could be taking on implementation of accepted PEPs. We could upgrade speed.python.org to be more informative and easy to use. We could move the rest of the custom CPython bots to Github Actions, decreasing needed maintenance, improving performance and reliability. Those are just some ideas.

There is one more reason why I’m rooting for another person to join this position. Having another developer in residence would buffer any turbulence the other person has. Whenever I’m sick, or travel, or I’m stuck with a particularly stubborn problem, there would reliably be somebody else the other core developers could count on. This is important not only for them but also for me personally as it would decrease anxiety that builds up any time I’m unable to help somebody who needs me.

The ability to split work between two people is something I think about often. In theory there’s a whole team of core developers out there but since they’re mostly volunteers, I’m in no position to tell them what to do. Having a peer paid by the PSF would be different. It would be fair game to share the burden of a gnarly boring task, and that sounds like a wonderful improvement to me.

What if there isn’t another developer in residence?

I’m not saying the other person is required for me to stay productive. If we don’t find the budget for it, the situation is still better than having no developer in residence at all, I’d like to believe. So far I haven’t received much feedback on my work but I’m always open to hearing suggestions.

Python Packaging Strategy Discussion Summary - Part 1

As the Python Packaging Project Manager, my work has mostly concentrated on improving user experience on PyPI. As the Python Packaging ecosystem grows from strength to strength, one of the initiatives I have undertaken is to define a cohesive strategy for Python Packaging.

My Mission

Python Packaging is a diverse landscape dotted with many tools that cater to generic and niche user requirements. As a community, something that is poorly understood is where the community wants to be in 5 years. Understanding where we want to be is important to allow us to identify key goals for the future and how we will reach them, and to ensure we place users at the heart of everything we do. By defining key deliverables driven by community consensus, securing funding becomes easier as we can demonstrate that the community vision will drive innovation and enable better user experience and adoption of Packaging tools.

User Survey

In order to understand what end users are looking for when they use Packaging tools, Nicole Harris and I worked with the community to finalize the survey questions. Nicole developed the final set of survey questions. The survey results are summarized in this document.

From the survey responses, it was clear that while Packaging tools were improving with time, most users found the landscape too complex to navigate. Based on the survey responses, the strategy discussion was condensed to these topics-

1. Development of a single Packaging tool or a more unified experience

2. Better support for Packaging users

3. Phasing out legacy systems

4. Supporting each other in the community

5. Encouraging long-term contribution

This post will summarize the discussion around a single Packaging tool. The other four topics will be the subject of future posts on Discuss.

One tool to rule them all?

An overwhelming majority of users recommended a more unified experience when using Packaging tools. The word cloud below shows the frequency of suggestions for the open-ended question- “If you could improve only one area of Python packaging, what would it be?”

To take this discussion further, I invited maintainers and contributors of any Packaging tool to answer this question:

“Can we reduce the number of tools and bring about some form of unification? Can we do anything else to reduce the complexity?”

After an interesting and lively discussion, we still do not have a clear consensus of what a solution should look like, but we do have a way forward. 

Since we are at the beginning of this conversation, we don’t know what unification means yet. As Pradyun Gedam suggested, there are many options as to what it could look like-

1. Unification of PyPI/conda models 

2. Unification of the consumer-facing tooling

3. Unification of the publisher-facing tooling

4. Unification of the workflow setups/tooling

5. Unification/Consistency in the deployment processes

6. Unification/Consistency in “Python” installation/management experience

7. Unification of the interface of tools

Based on these criteria, there is (some) consensus in driving forward unification of workflow setups/tooling and unification of interface of tools. 

The next major question was, if we do go ahead with unification of specific elements, which elements of Packaging would offer the best solution. To provide a unified UI/UX for end users, some of the solutions that were suggested were

1. Reusable libraries

2. Expanding pip to include more functionalities

3. Recommending existing tools such as hatch, poetry etc.

This discussion also raised a few interesting concerns. Understandably, there is concern over the time and effort that volunteers have devoted in developing tools and whether all this effort will come to naught if we recommend a specific tool. As the discussion continues, there is a large question over the part that PyPA plays and whether PyPA would be willing to take a stand on recommending a specific tool. One concern that keeps coming up is that historically there has been poor communication between PyPA, Packaging tools and end users and if anything will change this time.

The way forward

In order to flesh out the details of the actual solution, I recommend the following steps:

1. Identify the tasks that a unified solution has to accomplish

2. Identify the best way to deliver the solution. This could be a new tool, expanding an existing tool or developing standards/libraries.

3. Submit PEPs to ensure community governance and consensus

4. Ensure there is buy-in from end users for proposed solution(s)

5. Define development roadmaps, milestones, key deliverables and timeline

6. Generate funding to support development

7. Develop and deliver

8. Communicate, communicate, communicate

This will be a long and intensive process. But the time and effort invested in this project will be worth it as it will result in innovation and better user experience. 

We will be discussing four more big questions over the next three months that will be used to help us plan the future of Packaging at Python. As we continue the strategy discussions, I invite everyone to participate in the discussion threads on Discuss.  

We are grateful to Bloomberg for generously providing the funding for this role and making this valuable work possible.