Releasing pip 20.3, featuring new dependency resolver

On behalf of the Python Packaging Authority and the pip team, I am pleased to announce that we have just released pip 20.3, a new version of pip. You can install it by running python -m pip install --upgrade pip.

This is an important and disruptive release -- we explained why in a blog post last year. We've even made a video about it.

Highlights

• DISRUPTION: Switch to the new dependency resolver by default. Watch out for changes in handling editable installs, constraints files, and more: https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020

• DEPRECATION: Deprecate support for Python 3.5 (to be removed in pip 21.0).

• DEPRECATION: pip freeze will stop filtering the pip, setuptools, distribute and wheel packages from pip freeze output in a future version. To keep the previous behavior, users should use the new --exclude option.

• Substantial improvements in new resolver for performance, output and error messages, avoiding infinite loops, and support for constraints files.

• Support for PEP 600: Future manylinux Platform Tags for Portable Linux Built Distributions.

• Documentation improvements: Resolver migration guide, quickstart guide, and new documentation theme.

• Add support for MacOS Big Sur compatibility tags.

The new resolver is now on by default for Python 3 users. It is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please see our guide on how to test and migrate, and how to report issues. You can use the deprecated (old) resolver, using the flag --use-deprecated=legacy-resolver, until we remove it in the pip 21.0 release in January 2021.

You can find more details (including deprecations and removals) in the changelog.

Coming soon: end of Python 2.7 support

We aim to release pip 21.0 in January 2021, per our release cadence. At that time, pip will stop supporting Python 2.7 and will therefore stop supporting Python 2 entirely.

 

When users use pip 20.3 in a Python 2 environment, the old dependency resolver is still the default.

For more info or to contribute:

We run this project as transparently as possible, so you can:

• read and participate in GitHub issues

• contribute to our user experience work: sign up to become a member of the UX Studies group (after you join, we'll notify you about future UX surveys and interviews)

• contact us in the Discourse forum or Zulip chat

• read our meeting notes

Thank you

Thanks to our contractors on this project: Simply Secure (specifically Georgia Bullen, Bernard Tyers, Nicole Harris, Ngọc Triệu, and Karissa McKelvey), Changeset Consulting (Sumana Harihareswara), Atos (Paul F. Moore), Tzu-ping Chung, Pradyun Gedam, and Ilan Schnell. Thanks also to Ernest W. Durbin III at the Python Software Foundation for liaising with the project.

 

This award continues our relationship with Mozilla, which supported Python packaging tools with a Mozilla Open Source Support Award in 2017 for Warehouse. Thank you, Mozilla! (MOSS has a number of types of awards, which are open to different sorts of open source/free software projects. If your project will seek financial support in 2021, do check the MOSS website to see if you qualify.)

This is new funding from the Chan Zuckerberg Initiative. This project is being made possible in part by a grant from the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation. Thank you, CZI! (If your free software/open source project is seeking funding and is used by researchers, check the Joint Roadmap for Open Science Tools Rapid Response Fund and consider applying.)

 

The funding for pip's overhaul will end at the end of 2020; if your organization wants to help continue improvements in Python packaging, please join the sponsorship program.

As with all pip releases, a significant amount of the work was contributed by pip's user community. Huge thanks to all who have contributed, whether through code, documentation, issue reports and/or discussion. Your help keeps pip improving, and is hugely appreciated. Thank you to the pip and PyPA maintainers, to the PSF and the Packaging WG, and to all the contributors and volunteers who work on or use Python packaging tools.

 

-Sumana Harihareswara, pip project manager

Answer these surveys to improve pip's usability

The pip team has been working on improving the usability of pip since the start of this year. We've been carrying this work out remotely - by interviewing pip users, by sending short surveys, and doing usability tests of new pip functions.

We want to thank everybody who is contributing input to this work and are taking part in this research, which is still ongoing. We've learned a lot about who uses pip and how you use it. This has helped the team make decisions to improve pip, such as error messages and documentation to help you fix dependency conflicts.

Our team has put together a User Experience (UX) section in pip's documentation to tell you about this UX work. It covers what has happened so far, how you can contribute, and what is coming in the future.

Contribute to current UX work

Right now, you can take part in a number of studies about:

1. What pip features do you use most, and what pip feature you'd like to see - give your input by completing this survey
2. How "pip force install" should behave - give your input by completing this survey
3. How "pip --force-reinstall" should behave - give your input by completing this survey
4. Help create a design brief for a pip logo - give your input by completing this survey
5. What is your experience of using pip search - give your input by completing this survey

If you have time, the team asks for you to answer all of these surveys. You can do them in your own time, all at once or over a few days.

At the end of these surveys you can give your email address to be contacted for a short interview. These interviews will be via web conference/videocall.

Contribute to future pip UX work

If you want to contribute to our UX work in the future, please sign up to become a member of the UX Studies group.

After you join, we'll notify you about future UX Studies (surveys and interviews).

Contacting the pip UX Team

You can contact the pip UX Team by email.

We look forward to talking with you!

-Bernard Tyers, user experience, pip team 

Sumana Harihareswara, project manager, pip team

New pip resolver to roll out this year

The developers of pip are in the process of developing a new resolver for pip (as we announced on the PSF blog last year). We aim to roll it out later this year. (Updated 29 September to add: see our migration guide.) As part of that work, there will be some major changes to how pip determines what to install, based on package requirements. In this post we share:

What will change
What you can do to help
When this will be happening

To understand what pip does and why we’re working on it, please read our earlier overview post.

What will change

The most significant changes to the resolver will be:

• It will reduce inconsistency: it will no longer install a combination of packages that is mutually inconsistent. At the moment, it is possible for pip to install a package which does not satisfy the declared requirements of another installed package. For example, right now, pip install "six<1.12" "virtualenv==20.0.2" does the wrong thing, “successfully” installing six==1.11, even though virtualenv==20.0.2 requires six>=1.12.0,<2 (defined here). The new resolver would, instead, outright reject installing anything if it got that input.
• It will be stricter - if you ask pip to install two packages with incompatible requirements, it will refuse (rather than installing a broken combination, like it does now).

Also, this is a major change to a key part of pip - it’s quite possible there will initially be bugs. We would like to make sure that those get caught before people start using the new version in production.

(For deep technical details, see this in-progress GitHub milestone.)

What you can do to help

We recognize that everyone’s work is being disrupted by the COVID-19 pandemic, and that many data scientists and medical researchers use Python and pip in their work. We want to make the upgrade process as smooth and bug-free as possible for our users; if you can help us, you’ll be helping each other.

1. First and most fundamentally, please help us understand how you use pip by talking with our user experience researchers. You can do this right now! You can take a survey, or have a researcher interview you over a video call. Please sign up and spread the word to anyone who uses pip (even a little bit).

2. Right now, even before we release the new resolver as a beta, you can help by running pip check on your current environment. This will report if you have any inconsistencies in your set of installed packages. Having a clean installation will make it much less likely that you will hit issues when the new resolver is released (and may address hidden problems in your current environment!). If you run pip check and run into stuff you can’t figure out, please ask for help in our issuetracker or chat.

3. Please make time to test the new version of pip, probably in May (see below). While we have tried to make sure that pip’s test suite covers as many cases as we can, we are very aware that there are people using pip with many different workflows and build processes, and we will not be able to cover all of those without your help.

   • If you use pip to install your software, try out the new resolver and let us know if it works for you.
   • If you have a build pipeline that depends on pip installing your dependencies for you, check that the new resolver does what you need.
   • Run your project’s CI (test suite, build process, etc.) using the new resolver, and let us know of any issues.
   • If you have encountered resolver issues with pip in the past, check whether the new resolver fixes them. Also, let us know if the new resolver has issues with any workarounds you put in to address the current resolver’s limitations. We’ll need to ensure that people can transition off such workarounds smoothly.

   As you and your colleagues plan for the next few months, please set aside time, ideally in May, to test the new resolver and tell us whether it breaks anything for your setup by filling out this survey. If you maintain an open source project or use pip at your job, you can make a ticket now and put it in your backlog.

4. Spread the word! If you know of other people who rely on pip, who might not have seen this message, let them know. You can do this right now.

5. And if you develop or support a tool that wraps pip or uses it to deliver part of your functionality, please make time to test your integration with our beta in May, and sure that the new resolver doesn’t cause you any issues. (It shouldn’t, as the resolver is an internal component of pip and shouldn’t be visible to people embedding pip, but we’d like to be sure of that.)

When this will be happening

We intend to release the new resolver early in the second half of 2020. We will provide alpha and beta releases before that point as the work progresses (probably starting in May), and we’ll provide a GitHub issue tracker and an email address where you can report bugs. We would appreciate as much feedback as we can get on the betas when they become available.

(We were already working as a distributed team. The COVID-19 pandemic and related disruptions are affecting us – for instance, we were aiming to meet, work together, and test and discuss our work at PyCon US. But we still anticipate releasing the new resolver in the second half of 2020.)

To find out when the new beta is available for testing, and how to report problems, subscribe to the pypi-announce mailing list. It’s very low-traffic.

(Why now? The Python Software Foundation’s Packaging Working Group obtained funding for this work because it’s sorely needed; many other features and tools are blocked waiting for this fix. We’ve spent years addressing technical debt in pip so we can properly untie this knot and refurbish the resolver. We started this chunk of donor-funded work on pip a few months ago and it’s now gotten far enough that we can make this pre-announcement.)

Thank you to the pip and PyPA maintainers, to the PSF and the Packaging WG, and to all the contributors and volunteers who work on or use Python packaging tools. And thank you to Mozilla (through its Mozilla Open Source Support Awards) and to the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation, for funding enabling this work!

Leadership for PyCon US 2011

The PyCon organizing committee has started making plans for PyCon US 2011, to be held in Atlanta, GA on March 9-17 of next year. One of the first steps is ensuring that the conference committee leadership is in place.

Van Lindberg returns as Conference Chair this year. Van manages all of the activities related to running PyCon US, including organizing the volunteer and paid efforts.

This year Jesse Noller will be assisting Van as Vice Chair. Jesse's primary responsibility is serving as the Program Committee Chair, but he will work with Van in other areas as well.

Doug Napoleone will be the Technical Chair. He is overseeing the redevelopment of the PyCon web site, including the talk selection and schedule management software. Doug will also be responsible for coordinating the technical vendors for the conference, including A/V, networking, and recording.

Greg Lindstrom is returning as Tutorials Coordinator. Greg has been organizing tutorials for a number of years, and his patience and skill have been a big factor in the successful tutorial program that has grown from eight tutorials several years ago to the 32 sessions we regularly have now.

Vern Ceder is returning for a second year as Poster Sessions Coordinator. The poster sessions were one of the big hits at PyCon 2010, and we expect that they will be much bigger for 2011.

Volunteering

Although having good leadership is important, five people cannot organize a conference as big as PyCon. The organizing committee depends on the participation of dozens of volunteers from the community to do most of the work. If you would like to help, join the PyCon organizers mailing list and follow the blog to keep up with other announcements.

PSF Officers for 2010

The 2010-2011 Board of Directors for the PSF has appointed officers to oversee activities of the Foundation.

Returning Officers

Guido van Rossum returns as President of the PSF. As President, Guido serves as the principal representative of the Foundation.

Steve Holden is Board Chair. He is responsible for organizing and presiding over all Board meetings and keeping the Foundation focused on fulfilling its mission.

Van Lindberg will serve as PyCon Chair again this year. Van manages all of the activities related to running PyCon US, including organizing the volunteer and paid efforts.

Most officers are volunteers from the community, but there are two paid positions on the Board. The Secretary and Treasurer both require regular and timely activity in order to keep up with their duties, so nominal compensation is provided for these officers to make that possible.

Pat Campbell is Secretary, and keeps all of the records of the Foundation. She also prepares the minutes of the Board meetings and delivers formal notices, for example before the start of official voting periods.

Kurt Kaiser continues as Treasurer. Kurt manages the finances of the PSF, receiving income and making payments. He reports the financial status of the Foundation to the members regularly through the members' mailing list.

New Roles

In addition to the traditional organizational roles, the Board is working to identify other areas where focused attention is needed and appoint officers to oversee them. Positions for overseeing Membership, Voting, and other areas are planned but not yet filled.

As Communications Director, Doug Hellmann is responsible for publishing information from the Board, with priority given to any actions taken at Board meetings. He will also be helping to connect other members of the community with information channels to publicize the work they are doing.

More Information

• PSF Board Responsibilities
• PSF Bylaws, including the duties of each officer.
• Board history

PSF Sponsored Sprints

At their May 10 meeting, the PSF Board create a new Sprint Committee to organize a series of sprints for working on Python. Jesse Noller is the committee chair.

This new "focused sprint" initiative is part of a concerted effort to attract more contributors to Python's development team and to speed the adoption of Python 3. The PSF has agreed to provide funding to groups interested in organizing sprints related to:

• Python core tasks such as bug triage and evaluating patches
• Python core documentation work
• Porting existing third-party libraries to run under Python 3
• Enhancing http://python.org/ with new content, organization, or design work
• Development of PyPI

Over time, other projects may be added as the committee identifies them. Complete instructions for proposing a sprint for your group will be posted once they are worked out.

In the mean time, the Sprint Committee needs volunteers to help launch this project. The first steps will be to set up some communication channels, and then start developing several guides for new contributors. If you want to help out, contact Jesse Noller via sprints@python.org.

See the announcement on Jesse's blog for more details.

Help the PSF Help Python

The uninitiated tend to view the Python Software Foundation (PSF) as some sort of shadowy legal entity that mysteriously rules the Python community. In fact, the PSF has no ambition to "manage" the community (this is open source!) and we certainly don't just work on legal things. The important message that gets lost in this misconception is:

The PSF is Here to Help!

If you're a volunteer working on some aspect of the Python community -- whether developing code, organizing a conference, supporting end users, or running servers and other infrastructure -- please keep in mind that the PSF may be able to help with problems that you run into.

In particular, we're interested in areas where volunteers are overwhelmed, or where some funding could help to bootstrap or enable a subsequently self-sustaining resource for the community. We've supported efforts as diverse as the new bug tracker, Python conferences around the world, and the python.org website redesign.

We can't always be as proactive as we'd like, so may not know you need our help. Tell us!

The other side of the message, equally important, is:

Python Needs Your Help!

Alas, the PSF isn't an interplanetary enterprise with a hundred thousand employees, so it's mostly volunteers that run the Python community. Indeed it does take significant effort to keep things going. Postmaster, website maintainer, help list responder, and job board updater are all volunteer posts that come to mind immediately. There are many other areas that can use the help of active volunteers.

Such tasks are often an excellent training ground, and a good way to get your feet wet participating in (and meeting) the Python community. For experienced hands that already know everything and everybody, it's a great way to give back to Python, and make a valuable difference in the Python universe.

Please consider helping out. You'll earn the satisfaction of sitting 'round the bar at PyCon shooting the breeze over a Pan-Galactic Gargle Blaster or three. Guaranteed!

So, What Do I Do Now?

If you have a suggestion, feel free to add a comment to this entry. To help out, please contact us at psf@python.org or join our volunteer pool email list.

Oh, and please pass the word on to others!

Thanks!