The PSF awarded $170,000 grant from Mozilla Open Source Program to improve sustainability of PyPI

When are we finally going to see Warehouse deployed? What's the holdup? Has this project stalled?

For the last year, we at the Python Packaging Authority have heard these questions continuously from the Python community.

Today we are excited to announce that we have applied for, and been awarded, a grant to help improve the sustainability of the Python Package Index in the amount of $170,000.  This has been awarded by Mozilla, through the Foundational Technology track of their Open Source Support Program.  We would like to thank Mozilla for their support.

This post will explain both the context for this request, and the work that will be funded.
Specifically, this grant funds several contributors' efforts to finish the development and deployment of Warehouse (pypi.org), the replacement for our legacy codebase that runs pypi.python.org -- which will allow more people to contribute.

Background

The Python Packaging Index (PyPI) is the principal repository of software packages for the Python programming language. Currently, over 100 million Python packages are downloaded from PyPI every week. The Python community (and indeed the wider technical community) depends on PyPI for the ongoing functioning of the entire Python ecosystem.

There are no paid staff at the PSF who work on PyPI, and there are only a handful of people who contribute regularly.  This leads to a situation where we have to depend on volunteers to be on-call for outages and respond to critical security vulnerabilities in core Python Infrastructure. For deeper context, read PyPI maintainer Donald Stufft's May 2016 post "Powering the Python Package Index", (but note that since his job change in the last year his paid time to work on PyPI has decreased significantly).

This isn't a tenable situation for the long term, and we hope that this grant will help us start to change this.

Developer Sustainability

Unfortunately, the codebase that runs PyPI is old, almost entirely lacking in automated test coverage and other common best practices. (PyPI contributor Ernest W. Durbin III goes into this further in his PyGotham 2017 talk "Running Vintage Software: PyPI's Aging Codebase.") The new Warehouse code base is built with Pyramid, and is much easier to maintain. This technical debt has a number of undesirable outcomes, but the one I'd like to focus on is repelling new contributors.

Eric Holscher, like many would-be contributors to the legacy PyPI codebase, once tried to contribute a small feature to PyPI years ago, thinking it would be quite simple. However, as he opened the code base and started looking around, he discovered that the code wasn't written in a modern framework, didn't follow standard conventions or best practices, and he could barely even understand where to start adding something new.

Migrating PyPI to a new codebase will stop this from happening. We hope to gain a number of new contributors who are willing and able to help maintain the code base, and decrease the chance of burnout for the staff.

Operations Sustainability

Operations are the other large sustainability issue that we're hoping to tackle with this grant. There are only two people currently who wear a pager and are on-call for PyPI. They are not currently supported by modern automation tools or served by a continuing project manager. Once all package index activity is going through pypi.org (Warehouse) and we deprecate the old pypi.python.org codebase and site, with a stabilized infrastructure resource load, we can assess our new level of staffing and hosting needs. Based on that assessment, we'll be able to fundraise for staffing and ask our sponsors for financial and in-kind donations to keep PyPI robust.

Next Steps

With the help of many people (especially the Packaging Working Group, our MOSS liaison Gervase Markham, our Mozilla champion Dan Callahan, and Ewa Jodlowska and Mark Mangoba at the PSF), we've been figuring out the timeline for this work.

The first milestone for Warehouse is redirecting portions of the production pypi.python.org to Warehouse including traffic for the simple index and package downloads. At that milestone Warehouse will be the main entryway to Python packages for all but a small fraction of the interactions PyPI sees.

The bulk of the work will be bringing Warehouse to feature parity with the administrative capabilities users need from the Package Index.  We'll keep you posted as we figure out when you can expect that to be true.

Please feel free to ask questions about the Warehouse project in the #pypa-dev channel on Freenode, or in the GitHub issues for Warehouse! If you have questions for the PSF about the grant, you can ask via email.

Thanks to Donald Stufft, Ewa Jodlowska, Nathaniel J. Smith, Nick Coghlan, Nicole Harris, Sumana Harihareswara, Ernest W. Durbin III, Dustin Ingram, Mark Mangoba, Kenneth Reitz, Eric Holscher for contributing to this post.

Eric Floehr, Community Service Award 3rd Quarter 2017 Recipient

When Eric Floehr was a child growing up in Ohio, he had three interests: space, dinosaurs, and the weather. One day, his dad brought home a computer to make video games. Eric and his dad worked together copying code from magazines, thus beginning a lifelong interest in programming.

After getting his bachelor's degree in Computer Science at Ohio State, Eric worked as a software engineer, all while nurturing Python hobbies built on his childhood interests. He now works for the company he founded called Intellovations whose primary product is ForecastWatch, a tool that helps weather forecasters be more accurate. Eric has also been a consistent leader in his local Ohio Python community, all while spreading the Python love by encouraging others to get involved and create groups of their own.

The Python Software Foundation has awarded Eric with the Q3 2017 Community Service Award.

RESOLVED, that the Python Software Foundation Q3 2017 Community Service Award be given to Eric Floehr for his work chairing the PyOhio Conference. He is the founder and co-organizer of the Central Ohio Python User Group. COhPy has served as a resource for Python programmers in Columbus, OH. Eric's efforts to "spread Python love" via discussions with leaders of PyNash and IndyPy lead directly to the creation of the annual PyTennessee conference and the Pythology quarterly mini-conferences.

Contributions to the Python Community

PyOhio

While large events like PyCon US are incredibly important for the Python community and bring diverse groups together, it could be argued that smaller groups with more frequent events provide even more benefit. Here attendees are more likely to meet a potential employer/employee, find someone to talk through a problem at work or fun project idea, and socialize with like-minded locals. Eric works from home, which is one of the reasons he enjoys engaging socially with his local Python community. He started by attending PyOhio, a free annual regional Python conference, and quickly wanted to help out. “The first year I helped with pizza, cleanup, and video”, he recalls. By the second year, he was a full-on organizer, “you don’t really need to have any particular skills to organize, just jump in there and do it.”

COhPy

In 2010, when he realized there was no Python group in Columbus, he started The Central Ohio Python User’s Group (COhPy). COhPy hosts meetings once a month where people can listen to talks and chat with local Pythonistas. It also holds networking events and offers other services for the community such as a Slack channel. “Eric's efforts as an organizer of PyOhio and COhPy have given hundreds of Python developers the opportunity to teach and share with thousands of other Python developers”, says Brian Costlow, fellow PyOhio and COhPy organizer, “It gives people the opportunity to grow into speakers and teachers in a small, safe venue, and for many, myself included, to make new and lifelong friends.“ Jason Green, a PyOhio organizer, credits Eric’s gregarious and inclusive nature with his own integration into the Python community. Not only did he welcome Jason to the group, he encouraged him to get involved. “As a leadership mentor, for the last several months,” Jason says, “he has made a point of having me introduce the speakers and welcome new guests.”

Spreading the Python Love

Eric encourages others to get outside of their comfort zone, try new things, and start groups in their own areas. For example, at the PyOhio 2013 conference, Eric put out a call for more regional Python conferences. This struck Jason Myers, a PyOhio attendee, as something that would benefit his Python community in Tennessee. Jason approached Eric with the idea of starting a PyTennessee conference, and Eric immediately offered to help. “Over the course of our first conference planning, call for proposals, and the event itself, Eric was always there with guidance.” Eric’s support didn’t stop there, Jason goes on, “for all four years that I ran PyTennessee, Eric was our best supporter, cheerleader, and advisor.” Jason credits Eric for PyTennessee’s success explaining, “I know without a shadow of a doubt that there would be no PyTennessee without him, and I am eternally grateful for his wisdom, assistance and his friendship.”

Python for Fun

Eric’s love for Python does not end with work and community but is a large part of his hobbies as well. Perhaps his most interesting hobby combines his love of Python and the weather. He set up an old digital camera out his window at home with a Raspberry Pi to take photos every 10 seconds for 3 years. Not only was he able to capture beautiful and dramatic images of weather events, he collected 6 terabytes worth of pictures and metadata that he used Python to analyze in interesting ways. When his analysis was complete in 2015, he gave a fantastic talk on this project at PyOhio.

Time-lapsed images from Eric’s digital camera and raspberry pi

When asked why Python is his language of choice, Eric beams, “Python is a great enabler. It allows people to do more in less time and to build amazing things. From creative works to scientific research, from scratching personal itches, to helping solve critical problems, Python is an incredible tool for growth and exploration. But more than the tool itself,” Eric goes on to say, “it's the community around the tool that I have really fallen in love with. Its focus on inclusivity, tolerance, and respect has been a model for other communities, and it's not only a community I love but one that I'm proud to be a part of.”

What’s next for this Python hobbyist? “Have you seen Westworld?”, he asks. “Like the piano playing by itself in the opening sequence, I’d like to make a mechanical xylophone with 30 keys and 30 hammers that plays itself like a music box.”

Community Service Award Winner Q3 2017 Eric Floehr