Releasing pip 20.3, featuring new dependency resolver

On behalf of the Python Packaging Authority and the pip team, I am pleased to announce that we have just released pip 20.3, a new version of pip. You can install it by running python -m pip install --upgrade pip.

This is an important and disruptive release -- we explained why in a blog post last year. We've even made a video about it.

Highlights

• DISRUPTION: Switch to the new dependency resolver by default. Watch out for changes in handling editable installs, constraints files, and more: https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020

• DEPRECATION: Deprecate support for Python 3.5 (to be removed in pip 21.0).

• DEPRECATION: pip freeze will stop filtering the pip, setuptools, distribute and wheel packages from pip freeze output in a future version. To keep the previous behavior, users should use the new --exclude option.

• Substantial improvements in new resolver for performance, output and error messages, avoiding infinite loops, and support for constraints files.

• Support for PEP 600: Future manylinux Platform Tags for Portable Linux Built Distributions.

• Documentation improvements: Resolver migration guide, quickstart guide, and new documentation theme.

• Add support for MacOS Big Sur compatibility tags.

The new resolver is now on by default for Python 3 users. It is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please see our guide on how to test and migrate, and how to report issues. You can use the deprecated (old) resolver, using the flag --use-deprecated=legacy-resolver, until we remove it in the pip 21.0 release in January 2021.

You can find more details (including deprecations and removals) in the changelog.

Coming soon: end of Python 2.7 support

We aim to release pip 21.0 in January 2021, per our release cadence. At that time, pip will stop supporting Python 2.7 and will therefore stop supporting Python 2 entirely.

 

When users use pip 20.3 in a Python 2 environment, the old dependency resolver is still the default.

For more info or to contribute:

We run this project as transparently as possible, so you can:

• read and participate in GitHub issues

• contribute to our user experience work: sign up to become a member of the UX Studies group (after you join, we'll notify you about future UX surveys and interviews)

• contact us in the Discourse forum or Zulip chat

• read our meeting notes

Thank you

Thanks to our contractors on this project: Simply Secure (specifically Georgia Bullen, Bernard Tyers, Nicole Harris, Ngọc Triệu, and Karissa McKelvey), Changeset Consulting (Sumana Harihareswara), Atos (Paul F. Moore), Tzu-ping Chung, Pradyun Gedam, and Ilan Schnell. Thanks also to Ernest W. Durbin III at the Python Software Foundation for liaising with the project.

 

This award continues our relationship with Mozilla, which supported Python packaging tools with a Mozilla Open Source Support Award in 2017 for Warehouse. Thank you, Mozilla! (MOSS has a number of types of awards, which are open to different sorts of open source/free software projects. If your project will seek financial support in 2021, do check the MOSS website to see if you qualify.)

This is new funding from the Chan Zuckerberg Initiative. This project is being made possible in part by a grant from the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation. Thank you, CZI! (If your free software/open source project is seeking funding and is used by researchers, check the Joint Roadmap for Open Science Tools Rapid Response Fund and consider applying.)

 

The funding for pip's overhaul will end at the end of 2020; if your organization wants to help continue improvements in Python packaging, please join the sponsorship program.

As with all pip releases, a significant amount of the work was contributed by pip's user community. Huge thanks to all who have contributed, whether through code, documentation, issue reports and/or discussion. Your help keeps pip improving, and is hugely appreciated. Thank you to the pip and PyPA maintainers, to the PSF and the Packaging WG, and to all the contributors and volunteers who work on or use Python packaging tools.

 

-Sumana Harihareswara, pip project manager

Answer these surveys to improve pip's usability

The pip team has been working on improving the usability of pip since the start of this year. We've been carrying this work out remotely - by interviewing pip users, by sending short surveys, and doing usability tests of new pip functions.

We want to thank everybody who is contributing input to this work and are taking part in this research, which is still ongoing. We've learned a lot about who uses pip and how you use it. This has helped the team make decisions to improve pip, such as error messages and documentation to help you fix dependency conflicts.

Our team has put together a User Experience (UX) section in pip's documentation to tell you about this UX work. It covers what has happened so far, how you can contribute, and what is coming in the future.

Contribute to current UX work

Right now, you can take part in a number of studies about:

1. What pip features do you use most, and what pip feature you'd like to see - give your input by completing this survey
2. How "pip force install" should behave - give your input by completing this survey
3. How "pip --force-reinstall" should behave - give your input by completing this survey
4. Help create a design brief for a pip logo - give your input by completing this survey
5. What is your experience of using pip search - give your input by completing this survey

If you have time, the team asks for you to answer all of these surveys. You can do them in your own time, all at once or over a few days.

At the end of these surveys you can give your email address to be contacted for a short interview. These interviews will be via web conference/videocall.

Contribute to future pip UX work

If you want to contribute to our UX work in the future, please sign up to become a member of the UX Studies group.

After you join, we'll notify you about future UX Studies (surveys and interviews).

Contacting the pip UX Team

You can contact the pip UX Team by email.

We look forward to talking with you!

-Bernard Tyers, user experience, pip team 

Sumana Harihareswara, project manager, pip team

Pip team midyear report

The grant-funded team working on improvements to pip in 2020 has now passed the halfway mark. Here's an update on where are so far and what's next.

Funding and Timeline Status

The plan that we proposed last year said that, by now, we would have finished Foundational work (Phase I) and Resolver work (Phase II), and the team would be doing Maintenance and Sustainability work (Phase III). Please see the timeline for user experience work and the timelines for development work.

We are behind where we had planned to be in the work roadmap. This is partially because the COVID-19 pandemic disrupted our work, but also because of policy and architecture decisions the project needed, and because foundational user experience research work has taken more time than we originally allotted. Thus, we have finished the Phase I and Phase II sections of the development work, and are approximately 75% of the way through the Phase I and Phase II user experience work. See below for accomplishments so far.

Funding: we predicted that we would be approximately 80% of the way through our one-year project budget (since the second half of the year has a slower work pace, primarily focusing on maintaining and deepening the work we started in the first half). We are now approximately 71% of the way through the budget, which gives us flexibility for the second half of the project.

Accomplishments

• Multiple pip releases (changelog) since we started in January, including 332 pull requests merged since we kicked off (out of 2637 merged since 2011), and including improvements to the resolver library resolvelib: 22 pull requests merged since we kicked off

• pip's new dependency resolver is about to go into beta. We released pip 20.1 in April which included an alpha version of the new resolver (hidden behind an optional "--unstable-feature=resolver" flag, but usable). This month we will release pip 20.2, which will include a robust beta of the new resolver (hidden behind an optional "--use-feature=2020-resolver" flag) that we will encourage users to test.

• User experience data-gathering included:

• Recruited over 200 Python community members to be part of the UX research "panel" of respondents (sign up if you are interested)

• Administered 5 surveys to gather feedback about issues with the pip resolver and dependency management

• Interviewed and/or did user tests with over 30 maintainers and users so far

• UX findings and resulting improvements included:

• Preliminary error message formatting guidelines and preliminary research findings to improve the ResolutionImpossible error message's format and content, which have now been implemented

• Improved documentation to help users fix conflicting dependencies

• Documentation improvements especially regarding pip internals and architecture

Next steps

Phase III development work commences next month. We will continue to improve the pip dependency resolver in response to testers' feedback. This will help us prepare to release pip 20.3, with the new resolver on by default, in October. We'll also review and respond to code contributions and new issues, to keep up with the pip code and issue review queue, help new contributors develop into continuing contributors, and help existing contributors grow into co-maintainers.

And our user experience work will also enter Phase III, deepening and expanding foundational research in Python packaging. We will recruit more users for interviews and surveys, develop user journey maps & workflows, work with maintainers to write documentation and help messages, develop templates for UI bugs, commands, error messages, output, documentation, and configuration files, and teach pip maintainers UX practices.

For more info or to contribute:

We run this project as transparently as possible, so you can:

•  read and participate in GitHub issues

• contact us in the Discourse forum or Zulip chat

• read our meeting notes

Thank you

Thanks to our contractors on this project: Nicole Harris, Bernard Tyers, and Georgia Bullen of Simply Secure; Pradyun Gedam; Ilan Schnell; Paul F. Moore of Atos; Tzu-ping Chung; Sumana Harihareswara of Changeset Consulting.

This award continues our relationship with Mozilla, which supported Python packaging tools with a Mozilla Open Source Support Award in 2017 for Warehouse. Thank you, Mozilla! (MOSS has a number of types of awards, which are open to different sorts of open source/free software projects. If your project is looking for financial support, do check the MOSS website to see if you qualify.)

This is new funding from the Chan Zuckerberg Initiative. This project is being made possible in part by a grant from the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation. Thank you, CZI! (If your free software/open source project is used by biology researchers, check the Essential Open Source Software for Science Request for Applications and consider applying for the next round).

Thank you to the pip and PyPA maintainers, to the PSF and the Packaging WG, and to all the contributors and volunteers who work on or use Python packaging tools.

Announcing a new Sponsorship Program for Python Packaging

The Packaging Working Group of the Python Software Foundation is launching an all-new sponsorship program to sustain and improve Python's packaging ecosystem. Funds raised through this program will go directly towards improving the tools that your company uses every day and sustaining the continued operation of the Python Package Index.

With this program we are asking companies that rely on Python, its ecosystem of packaging tools, and PyPI to help us build a dependable basis to continue our efforts. 

Improving the packaging ecosystem

Since 2017, the Packaging Working Group has secured multiple grants, completed one contract, and received a generous gift -- all with the goal of improving the Python packaging ecosystem for all users. Most of these projects were funded by not-for-profit organizations and all of them were one-time awards with specific objectives.

Results from these funded projects include:

• The successful relaunch of the Python Package Index, powered by the new 'Warehouse' codebase in 2018
• Adding security features to PyPI, including two-factor authentication in 2019
• Improving PyPI's web interface for users with disabilities and adding support for multiple locales in 2019
• Additional security-focused features for PyPI in 2019 and 2020
• Overhauling pip's user experience and dependency resolver in 2020

Companies have asked us how they can help fund the platform they depend on. With this new sponsorship program, the Working Group can sustainably fund packaging improvements not directed by a specific grant or contract and benefit millions of Python users around the world. Greater budget flexibility and a deeper reserve will help us invest in what the community needs.

Sustaining PyPI

As of April 2020, the Python Package Index responds to 800 million requests and delivers 200 million packages totalling 400 terabytes, during the typical day. Our users include hobbyists, scientists, companies, students, governments, nonprofits, and more.

Existing sponsors donate their services, which keeps PyPI free to users and to the PSF, aside from a subset of one staff member's time. Without these donations, the costs to operate PyPI each month would be staggering.

These critical service donations must not be taken for granted. Sponsoring the Packaging Working Group through this new program creates and maintains a stable reserve. We'll need that reserve in the event that we lose any of these in-kind service donations and must pay some or all of PyPI's operating costs.

Show your support!

As a company, your team can review the details of this new sponsorship program in our prospectus. Should you have any questions you can contact us at sponsorship@pypi.org. When you're ready, apply here. We are excited to hear from you!

If your company cannot donate: Even as an individual, your contributions count! No matter the size or frequency, please support us if you are able at donate.pypi.org.

New pip resolver to roll out this year

The developers of pip are in the process of developing a new resolver for pip (as we announced on the PSF blog last year). We aim to roll it out later this year. (Updated 29 September to add: see our migration guide.) As part of that work, there will be some major changes to how pip determines what to install, based on package requirements. In this post we share:

What will change
What you can do to help
When this will be happening

To understand what pip does and why we’re working on it, please read our earlier overview post.

What will change

The most significant changes to the resolver will be:

• It will reduce inconsistency: it will no longer install a combination of packages that is mutually inconsistent. At the moment, it is possible for pip to install a package which does not satisfy the declared requirements of another installed package. For example, right now, pip install "six<1.12" "virtualenv==20.0.2" does the wrong thing, “successfully” installing six==1.11, even though virtualenv==20.0.2 requires six>=1.12.0,<2 (defined here). The new resolver would, instead, outright reject installing anything if it got that input.
• It will be stricter - if you ask pip to install two packages with incompatible requirements, it will refuse (rather than installing a broken combination, like it does now).

Also, this is a major change to a key part of pip - it’s quite possible there will initially be bugs. We would like to make sure that those get caught before people start using the new version in production.

(For deep technical details, see this in-progress GitHub milestone.)

What you can do to help

We recognize that everyone’s work is being disrupted by the COVID-19 pandemic, and that many data scientists and medical researchers use Python and pip in their work. We want to make the upgrade process as smooth and bug-free as possible for our users; if you can help us, you’ll be helping each other.

1. First and most fundamentally, please help us understand how you use pip by talking with our user experience researchers. You can do this right now! You can take a survey, or have a researcher interview you over a video call. Please sign up and spread the word to anyone who uses pip (even a little bit).

2. Right now, even before we release the new resolver as a beta, you can help by running pip check on your current environment. This will report if you have any inconsistencies in your set of installed packages. Having a clean installation will make it much less likely that you will hit issues when the new resolver is released (and may address hidden problems in your current environment!). If you run pip check and run into stuff you can’t figure out, please ask for help in our issuetracker or chat.

3. Please make time to test the new version of pip, probably in May (see below). While we have tried to make sure that pip’s test suite covers as many cases as we can, we are very aware that there are people using pip with many different workflows and build processes, and we will not be able to cover all of those without your help.

   • If you use pip to install your software, try out the new resolver and let us know if it works for you.
   • If you have a build pipeline that depends on pip installing your dependencies for you, check that the new resolver does what you need.
   • Run your project’s CI (test suite, build process, etc.) using the new resolver, and let us know of any issues.
   • If you have encountered resolver issues with pip in the past, check whether the new resolver fixes them. Also, let us know if the new resolver has issues with any workarounds you put in to address the current resolver’s limitations. We’ll need to ensure that people can transition off such workarounds smoothly.

   As you and your colleagues plan for the next few months, please set aside time, ideally in May, to test the new resolver and tell us whether it breaks anything for your setup by filling out this survey. If you maintain an open source project or use pip at your job, you can make a ticket now and put it in your backlog.

4. Spread the word! If you know of other people who rely on pip, who might not have seen this message, let them know. You can do this right now.

5. And if you develop or support a tool that wraps pip or uses it to deliver part of your functionality, please make time to test your integration with our beta in May, and sure that the new resolver doesn’t cause you any issues. (It shouldn’t, as the resolver is an internal component of pip and shouldn’t be visible to people embedding pip, but we’d like to be sure of that.)

When this will be happening

We intend to release the new resolver early in the second half of 2020. We will provide alpha and beta releases before that point as the work progresses (probably starting in May), and we’ll provide a GitHub issue tracker and an email address where you can report bugs. We would appreciate as much feedback as we can get on the betas when they become available.

(We were already working as a distributed team. The COVID-19 pandemic and related disruptions are affecting us – for instance, we were aiming to meet, work together, and test and discuss our work at PyCon US. But we still anticipate releasing the new resolver in the second half of 2020.)

To find out when the new beta is available for testing, and how to report problems, subscribe to the pypi-announce mailing list. It’s very low-traffic.

(Why now? The Python Software Foundation’s Packaging Working Group obtained funding for this work because it’s sorely needed; many other features and tools are blocked waiting for this fix. We’ve spent years addressing technical debt in pip so we can properly untie this knot and refurbish the resolver. We started this chunk of donor-funded work on pip a few months ago and it’s now gotten far enough that we can make this pre-announcement.)

Thank you to the pip and PyPA maintainers, to the PSF and the Packaging WG, and to all the contributors and volunteers who work on or use Python packaging tools. And thank you to Mozilla (through its Mozilla Open Source Support Awards) and to the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation, for funding enabling this work!

Mozilla and Chan Zuckerberg Initiative to support pip

The Python Software Foundation is receiving $407,000 USD to support work on pip in 2020. Thank you to Mozilla (through its Mozilla Open Source Support Awards) and to the Chan Zuckerberg Initiative for this funding! This foundational, transformational work will release Python developers and users to concentrate on the tools they're making and using, instead of troubleshooting dependency conflicts. Specifically, CZI and Mozilla are funding user experience, communications/publicity, and testing work (including developing robust testing infrastructure) as well as core feature development and review.

 

What we're doing with the money

Computers need to know the right order to install pieces of software ("to install x, you need to install y first"). So, when programmers share software, we have to precisely describe those installation prerequisites, and our installation tools need to navigate tricky situations where they're getting conflicting instructions. This project will make pip, a package installer for Python, better at handling that tricky logic, and easier for people to use and troubleshoot.

Millions of people and organizations use tools written in Python, and nearly the entire ecosystem of Python software projects depends on pip. Our project will help everyone more easily install software, diagnose and fix problems, and maintain infrastructure.

The Chan Zuckerberg Initiative funding is, in particular, aimed at improving Essential Open Source Software for Science. Scientists use many tools written in Python -- many of CZI's awardees in this round are written in Python -- but, also, researchers often want to write tools and share them with each other to help science advance faster. Our work will include research and improvements so the installation process will be easier to use and understand, which will enable researchers to build better applications and compose complex toolchains more easily.

We've laid out a detailed three-phase work plan on our pip 2020 Donor-funded Roadmap wiki page. To summarize:

• Mozilla is awarding PSF a Mozilla Open Source Support Award (Track I: Foundational Technology) for $207,000, which is paying for 5 months of:
• Python development work: Reviewing and responding to open issues and pull requests, refactoring build logic, collaborating with downstream projects and users about config flags and transition schedules, working on the dependency resolver itself and fixing bugs.
• Initial user experience research and design work: Reading existing bug reports and posts about Python package management confusion, interviewing users and running user tests, developing user journey maps and workflows, and working with maintainers to write documentation and help messages and to design resolver user experience.
• CZI is giving PSF an Essential Open Source Software for Science grant for $200,000, which is paying for:
• 12 months of Python development, test infrastructure, and project maintenance: Triaging bugs and reviewing pull requests, writing test cases, testing lead developers' work, building test infrastructure, investigating and fixing bugs, and writing the raw material for documentation to help future maintainers onboard better.
• 4 months of Phase III user experience research and design work: Training maintainers in UX design, doing further user tests on the new pip, developing a checklist for developing new features, and making templates for commands, error messages, output, documentation, and config files.
• Travel for initial developer onboarding and for some contributors to attend PyCon North America.
• And both CZI and Mozilla are paying for project management (planning, testing, editing, coordinating, communicating with stakeholders, announcing, reporting to funders, and getting obstacles out of everyone's way) and PSF administrative work (recruiting and overseeing contractors, project oversight, and financial processing).

The Packaging Working Group is currently selecting contractors for the programming parts of this work. The other two contractors have already been selected:

• Simply Secure is "an educational nonprofit 501(c)3 that supports practitioners in designing ethically-informed, values-driven technology that protects human rights." As experts in open source user experience, Simply Secure will bring UX research and design skills to the command-line experience of Python package management.
• Changeset Consulting, LLC, providing project management, is a returning contractor, having previously worked on the PyPI rewrite launch and improvements to PyPI's accessibility, security, and internationalization. Changeset lead Sumana Harihareswara was also the lead grantwriter for these funding proposals.

 

Why this and why now?

We're partway through a next-generation rewrite of pip's dependency resolver. The project ran into massive technical debt, but the refactoring is nearly finished and prototype functionality is in alpha now.

We need to finish the resolver because many other packaging improvements are blocked on it, it'll fix many dependency issues for Python users, and it'll fix installation problems for conda, certbot, WebSocket, and many other projects. And we need to improve pip's UX by providing better error messages and prompts, logs, output, and reporting, consistently across features, to fit users' mental models better, make hairy problems easier to untangle, and reduce unintended data loss.

The Packaging Working Group looks for potential improvements in Python packaging and distribution that are well-scoped, have community consensus, and could be expedited through funding. In the past three years, the Packaging WG has received funding for several improvements to PyPI -- $170,000 from Mozilla, $80,000 from OTF, and $100,000 from Facebook -- and is seeking to help other packaging tools. In June, pip maintainers and Packaging Working Group members discussed the importance and difficulty of rolling out the new resolver. We worked together to write and submit proposals to Mozilla and the Chan Zuckerberg Initiative.

 

What's next?

This work will start by early January 2020. Day-to-day work will mostly happen in pip's GitHub repository and the Python developers' Zulip livechat. You can check for regular reports at the Python Insider blog and the Packaging category of Python's Discourse developer forum, archived at the Packaging WG's wiki page. And we'll publicize calls for volunteers, especially for user interviews and tests, on this blog, on community mailing lists, and on Twitter.

The Packaging WG will continue to seek funding for future improvements in pip, manylinux, setuptools, the Python Packaging User Guide, PyPI, etc.

 

Thanks

This award continues our relationship with Mozilla, which supported Python packaging tools with a Mozilla Open Source Support Award in 2017 for Warehouse. Thank you, Mozilla! (MOSS has a number of types of awards, which are open to different sorts of open source/free software projects. If your project is looking for financial support, do check the MOSS website to see if you qualify.)

This is new funding from the Chan Zuckerberg Initiative. This project is being made possible in part by a grant from the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation. Thank you, CZI! (If your free software/open source project is used by biology researchers, check the Essential Open Source Software for Science Request for Applications and consider applying for the next round).

Thank you to the pip and PyPA maintainers, to the PSF and the Packaging WG, and to all the contributors and volunteers who work on or use Python packaging tools.