Monday, September 30, 2019

Grants Awarded for Python in Education

The Python Software Foundation has been asked about Python in education quite a bit recently. People have asked, “Is there an official curriculum we can use?”, “Are there online resources?”, “Are there efforts happening to improve Python on mobile?”, and so on.

9 years ago we instituted the Education Summit at PyCon US where educators as well as students work together on initiatives and obstacles. Earlier this year we decided we needed to do more. In November of 2018, the PSF created the Python in Education Board Committee and it was tasked with finding initiatives to fund to help improve the presence of Python in education.

In January of this year, the Python in Education Board Committee launched a “request for ideas” phase taking suggestions from the community on what we should focus our funding on. After the RFI period, we came up with 3 areas of education we wanted to focus on and asked to receive grant proposals on the following: resources (curriculums, evaluations, studies, multidisciplinary projects), localization (primarily translations), and mobile (development on mobile devices).

We are happy to publish more details on the grants the PSF approved from this initiative!

Beeware

The BeeWare Project wants to make it possible for all Python developers to write native apps for desktop and mobile platforms. Most desktop operating systems and iOS are supported already, but Android needs attention. Since Android users outnumber other mobile OS users worldwide by over 3 to 1, we determined it is important to fund this project. Beeware was awarded a $50,000 grant to help improve Python on Android. Phase one will be starting soon with this set of goals:

  1. A port of the CPython runtime to Android, delivered as a binary library ready to install into an Android project.
  2. A JNI-based library for bridging between the Android runtime and the CPython runtime.
  3. A template for a Gradle project that can be used to deploy Python code on Android devices. 

Beeware announced that they are looking for contractors to help with the work. Check out their blog post for more information.

Python in Education Website

Educational resources are in demand.  The PSF awarded a grant of $12,000 USD to Meg Ray, to work on creating a Python in Education website where we can curate educational information from all over the world. Meg will begin by collecting resources and after auditing the shared information, she will work on organizing it on an official PSF webpage. This work will begin in October of 2019 so please keep an eye out for updates via tweets and blogs!

Friendly-tracebacks

Lastly is a project called friendly-tracebacks. This project is not in need of financial support but is asking the PSF to help publicize it.  Friendly-traceback aims to provide simplified tracebacks translated into as many languages as possible. The project maintainer is looking for volunteers to help with tasks such as documenting possible SyntaxError use cases and documenting exceptions that haven't already been covered. Read more on their blog for the full call to action from the maintainer.

We hope to continue this initiative yearly! Companies that are passionate about supporting Python in Education should get in touch; we can't continue our work without your support!  As a non-profit organization, the PSF depends on sponsorships and donations to support the Python community.

Donate to the PSF: https://www.python.org/psf/donations/
Sponsor the PSF: https://www.python.org/psf/sponsorship/

Thursday, September 26, 2019

Felipe de Morais: 2019 Q2 Community Service Award Winner

Pythonistas everywhere benefit when our community reflects the many backgrounds and experiences of Python’s users. However it can be challenging to participate in the community when there are no local user groups or harder yet if groups do exist but you do not feel represented in them. After learning that a friend was experiencing gender descrimination at work, Felipe de Morais of Porto Alegre, Brazil, decided to start Django Girls Porto Alegre. By starting this group, women like his friend who were facing similar challenges could have a community to call their own.

Since Django Girls Porto Alegre took off in 2015, it has become one of the most active Django Girls groups in the world. Inspired by Django Girls and PyLadies, Felipe also started AfroPython, an initiative to empower Black people through technology. Additionally, Felipe contributes to Operação Serenata de Amor, an open source project that monitors public spending by politicians. For this work the PSF is pleased to award Felipe de Morais with the Q2 2019 Community Service Award:

RESOLVED, that the Python Software Foundation award the Q2 2019 Community Service Award to Felipe de Morais for his work towards facilitating the growth of the Python Brazilian community by organizing workshops, contributing to open source code that benefits the Brazilian people and for setting an example for all community organizers.

Felipe grew up in Rio De Janeiro where he earned a graduate degree in Computer Science and later moved to Porto Alegre in southern Brazil. With a strong desire to be part of a community, Felipe traveled to IT-related Meetup groups but longed for more accessible means to network and teach. Python was his go-to language for its simplicity and ease, and he loved to teach the language to help other people along in their careers. It is no surprise that the groups he’s started have both a Python and inclusivity focus.

When asked about his motivation for starting Python groups, Felipe says that that he simply gets great joy out of helping people. “I've seen people starting their work life because the workshop unlocked this career path for them,'' he says. “The most important part of it is the relationships made along the way, which make a great support system for a lot of people making positive life changes.”

AfroPython 2017
Renata D'Avila, a Django Girls Porto Alegre organizer, met Felipe 2016 at a Hackerspace event and the two have kept in touch ever since. “He is one of the people behind one of the biggest Django Girls workshop ever organized.” she recalls, “Django Girls Porto Alegre 2017 had about 180 people involved, among participants, mentors, and organizers.” However the event, as most events do, encountered some bumps in the road. As lunchtime rolled around and the planned caterers never showed up, Felipe raced across campus and resolved the issue, returning to the event with enough food for everyone. “That was one of the greatest achievements of that huge workshop,” says Renata, “that everyone could eat and that the schedule for the workshop was kept without people even knowing about how much effort it took to make it work.”

AfroPython 2018
Amanda Vieira met Felipe at Django Girls Porto Alegre and later worked with him to create AfroPython. She says of his work in these groups, “He has always helped both in the organization of events and as a coach, encouraging more and more women to get started in the area [of tech].” He has even developed a nickname, she quips “In his workplace and with friends, he is known as Felipython for all his involvement with language and the community.”

In May of 2017 when Felipe was attending Python Sudeste, a regional Python event in southeastern Brazil, he met Jessica Temporal. Jessica had been working as a data scientist on a large political open-source project, Operação Serenata de Amor. Serenata analyzes open data from the Brazilian government and flags expenses made by elected politicians that might be unlawful. Jessica was organizing a sprint at the conference and Felipe jumped in to help. In addition to working on some important refactoring and code readability issues, “Felipe was significant in making the project more friendly to newcomers,'' she says. He continues to contribute to the project today.

Felipe (left) with Seranata founder Eduardo Cuducos (middle) and Seranata data scientist Jessica Temporal (right)
Though Django Girls Porto Alegre and AfroPython have taken off, Felipe has no plans to slow down. In addition to continuing to work on open source projects, he is on the organizing team for the first ever AfroPythonConf later this year, and will be a keynote speaker at Python Brazil 2019.

CSA Award Winner Felipe de Morais
In his free time, Felipe enjoys Brazilian Jiu Jitsu, playing video games that remind him of his childhood, watching anime, and helping friends with finances and investments.

Wednesday, September 25, 2019

PyPI Security Q4 2019 Request for Proposals period opens.

The Python Software Foundation Packaging Working Group has received a grant from Facebook Research to implement advanced security features for PyPI. These features include cryptographic signing of uploaded artifacts and the infrastructure necessary to implement automated detection of malicious files uploaded to the index.
The Python Package Index (PyPI) is a foundational component of the Python ecosystem and broader computer software and technology landscape. This project aims to improve the security and accessibility of PyPI for all users worldwide, whether they are direct users, like project maintainers and pip installers, or indirect users. The impact of this work will be highly visible and improve crucial features of the service.
We plan to begin the project in Quarter 4 of 2019. Because of the size of the project, funding has been allocated to secure one or more contractors to complete the development, testing, verification, and assist in the rollout of necessary features.

Timeline

DateMilestone
September 25Request for Proposal period opened.
October 21Request for Proposal period closes.
October 29Date proposals will have received a decision.
December 2Contract work commences.

What is the Request for Proposals period?

A Request for Proposal (RFP) is a process intended to allow us (The Python Software Foundation) to collect proposals from potential contractors and select contractor(s) best suited to fulfill the specified work.
After the RFP period closes we will evaluate the received proposals based on the evaluation criteria, seek clarification from proposers as necessary, and select one or more contractors to complete the work specified in the scope.
The Request for Proposals period opens today, September 25th, 2019, and is scheduled to close October 21, 2019 AoE.

How do I submit a proposal?

First, please read the full contents of the Request for Proposals here!
You'll find the instructions for submissionevaluation criteria, as well as scope of the project there.

Tuesday, September 24, 2019

The Python Software Foundation has updated its Code of Conduct

The Python community values members who are accepting, helpful, and respectful: for many years, the Python Software Foundation (PSF) has had an organization-wide Code of Conduct that defines these values, and behaviors that we want to have in our community. The Foundation has also insisted for years that every event that we sponsor have a Code of Conduct in place.

But spaces where our community meets – online, or in person – need a Code of Conduct that does more than just emphasize our values. The PSF’s flagship conference, PyCon US, has had its own Code of Conduct – separate from the PSF Code of Conduct – for many years. The PyCon US Code of Conduct not only highlights our community’s values, but it also identified behaviors that are not acceptable at the conference, explained how to report violations, and included enforcement procedures.

The PSF Board approved a new organization-wide Code of Conduct and enforcement guidelines at the August 2019 board meeting, and reporting guidelines at the September 2019 board meeting, taking effect immediately.

Our new Code of Conduct brings together the statement of values defined in the former PSF Code of Conduct, and enforcement guidelines – proven through our experience at PyCon US – that the PSF can now apply to every space that we oversee.

It saves the PSF from having to enforce two Codes of Conduct: one for PyCon US, and another for our other spaces. In crafting the Code of Conduct, we undertook an intentional effort to account for the unique needs of an international community that spans all seven continents on Earth.

Community members will now know that if they’re participating in an online space, a project, or an event facilitated by the PSF they will be subject to the same Code of Conduct, and will be able to report incidents in the same way.

The process of defining the new Code of Conduct was led by the PSF’s Conduct Working Group, which the PSF established in 2018. The PSF worked with Sage Sharp of Otter Tech to produce the draft of the new Code of Conduct. Sage has previously worked on the Codes of Conduct for Open Source communities including the Data Carpentries, Elastic Search, and GNOME, and previously worked with the PSF on modernizing PyCon US’ Code of Conduct and incident response procedures. 

In the future, the Conduct Working Group will help the Board oversee the reporting and enforcement of Code of Conduct reports, following the enforcement guidelines that accompany the new Code of Conduct.

The Board thanks the Conduct Working Group, and Sage Sharp for their invaluable service in getting our new Code of Conduct in place.

Wednesday, August 28, 2019

PyPI Security Q4 2019 Request for Information period opens.

The Python Software Foundation Packaging Working Group has received funding from Facebook research to develop and deploy of enhanced security features to PyPI.
PyPI is a foundational component of the Python ecosystem and broader computer software and technology landscape. This project aims to improve the security and accessibility of PyPI for all users worldwide, whether they are direct users like project maintainers and pip installers or indirect users. The impact of this work will be highly visible and improve crucial features of the service.

Specifically, this project aims to implement verifiable cryptographic signing of artifacts and infrastructure to support automated detection of malicious uploads to the index.
We plan to begin the project in December 2019. Because of the size of the project, funding has been allocated to secure one or more contractors to complete the development, testing, verification, and assist in the rollout of necessary features.
Register Interest
To receive notification when our Request for Information period closes and the Request for Proposals period opens, please register your interest here.

What is the Request for Information period?

A Request for Information (RFI) is a process intended to allow us (The Python Software Foundation) and potential contractors to openly share information to improve the scope and definition of the project at hand. Also, we encourage stakeholders in the community with expertise in the project areas to contribute their viewpoints on open questions for the scope of the work.
We hope that it will help potential contractors better understand the work to be completed and develop better specified proposals. Additionally we have designed the RFI with an open nature in order to expose the project to multiple perspectives and help shape the direction for some choices in the project.
The Request for Information period opens today, August 28, 2019, and is scheduled to close September 18, 2019.
After the RFI period closes, we will use the results of the process to prepare and open a Request for Proposals to solicit proposals from contractors to complete the work.

More Information

The full version of our Request for Information document can be found here.

Participate!

Our RFI will be conducted on the Python Community Discussion Forum. Participants will need to create an account in order to propose new topics of discussion or respond to existing topics.
All discussions will remain public and available for review by potential proposal authors who do not wish to or cannot create an account to participate directly.

Tuesday, August 27, 2019

Python Software Foundation Fellow Members for Q1 & Q2 2019


We are happy to announce our newest PSF Fellow Members! This group includes nominated Fellows from Q1 and Q2 of 2019.

Q1 2019

Christoph Gohlke

Q2 2019

Aaron Yankey

Chris Jerdonek

Florian Bruhin
Matt Lebrun
Micaela Reyes
Pradyun Gedam
Rami Chowdhury
Tania Allard
Congratulations! Thank you for your continued contributions. We have added you to our Fellow roster online.

The above members have contributed to the Python ecosystem by teaching Python, maintaining popular libraries/tools, maintaining pip, organizing Python events, starting Python communities in their home countries, and overall being great mentors in our community. Each of them continues to help make Python more accessible around the world. To learn more about the new Fellow members, check out their links above.

Let's continue to recognize Pythonistas all over the world for their impact on our community. Here's the criteria our Work Group uses to review nominations:

  • For those who have served the Python community by creating and/or maintaining various engineering/design contributions, the following statement should be true:
    • Nominated Person has served the Python community by making available code, tests, documentation, or design, either in a Python implementation or in a Python ecosystem project, that 1) shows technical excellence, 2) is an example of software engineering principles and best practices, and 3) has achieved widespread usage or acclaim.
  • For those who have served the Python community by coordinating, organizing, teaching, writing, and evangelizing, the following statement should be true:
    • Nominated Person has served the Python community through extraordinary efforts in organizing Python events, publicly promoting Python, and teaching and coordinating others. Nominated Person's efforts have shown leadership and resulted in long-lasting and substantial gains in the number and quality of Python users, and have been widely recognized as being above and beyond normal volunteering.
  • If someone is not accepted to be a fellow in the quarter they were nominated for, they will remain an active nominee for 1 year for future consideration.
  • It is suggested/recommended that the nominee have wide Python community involvement. Examples would be (not a complete list - just examples):
    • Someone who has received a Community Service Award or Distinguished Service Award
    • A developer that writes (more than one) documentation/books/tutorials for wider audience
    • Someone that helps translate (more than one) documentation/books/tutorials for better inclusivity
    • An instructor that teaches Python related tutorials in various regions
    • Someone that helps organize local meet ups and also helps organize a regional conference
  • Nominees should be aware of the Python community’s Code of Conduct and should have a record of fostering the community.
  • Sitting members of the PSF Board of Directors can be nominated if they meet the above criteria.
If you would like to nominate someone to be a PSF Fellow, please send a description of their Python accomplishments and their email address to psf-fellow at python.org. We are accepting nominations for quarter 4 through November 20, 2019. More information is available at: https://www.python.org/psf/fellows/.

Monday, August 19, 2019

Humble Bundle by No Starch supports the Python Software Foundation!

We are super excited to announce that the Python Software Foundation is featured as a charity in a Humble Bundle by No Starch Press this month.



This bundle features books such as Python Playground, Mission Python, Invent your own Computer Games with Python, and so much more.

Click here to see all the books being featured and GET THE BUNDLE before it closes! The bundle will run from August 19th to September 2nd (11am Pacific). Proceeds received help charities such as the Python Software Foundation (PSF). Once you click to get your bundle, you can also choose where your money goes if you'd like to customize the split of proceeds.



Humble Bundle sells games, ebooks, software, and other digital content. Their mission is to support charity while providing awesome content to customers at great prices. Thanks to past Humble Bundles that the PSF has been a part of, this program has helped the PSF raise more than $300,000 since 2017! The PSF and the Python community thank Humble Bundle and all of the featured products that have selected the PSF as one of their charities. This funding has had a positive impact for Pythonistas all around the world.

No Starch is a long time community contributor supporting the PSF in various Bundles and supporting Young Coder classes that happen at PyCon US.  “As one of the leading publishers of Python books worldwide, No StarchPress is very excited to support the organization at the core of the Python programming language” said No Starch Press Founder Bill Pollock. “Python is at the core of so much technical work today and very much at the core of our publishing program.”

The PSF staff and board of directors send a big "Thank You!" to everyone involved.


Wednesday, July 31, 2019

PyPI now supports uploading via API token

We're further increasing the security of the Python Package Index with another new beta feature: scoped API tokens for package upload. This is thanks to a grant from the Open Technology Fund, coordinated by the Packaging Working Group of the Python Software Foundation.

Over the last few months, we've added two-factor authentication (2FA) login security methods. We added Time-based One-Time Password (TOTP) support in late May and physical security device support in mid-June. Now, over 1600 users have started using physical security devices or TOTP applications to better secure their accounts. And over the past week, over 7.8% of logins to PyPI.org have been protected by 2FA, up from 3% in the month of June.

Add API token screen, with textarea for token name and dropdown menu to choose token scope
PyPI interface for adding an
API token for package upload
Now, we have another improvement: you can use API tokens to upload packages to PyPI and Test PyPI! And we've designed the token to be a drop-in replacement for the username and password you already use (warning: this is a beta feature that we need your help to test).

How it works: Go to your PyPI account settings and select "Add API token". When you create an API token, you choose its scope: you can create a token that can upload to all the projects you maintain or own, or you can limit its scope to just one project.


API token management interface displays each token's name, scope, date/time created, and date/time last used, and the user can view each token's unique ID or revoke it
PyPI API token management interface
The token management screen shows you when each of your tokens were created, and last used. And you can revoke one token without revoking others, and without having to change your password on PyPI and in configuration files.

Uploading with an API token is currently optional but encouraged; in the future, PyPI will set and enforce a policy requiring users with two-factor authentication enabled to use API tokens to upload (rather than just their password sans second factor). Watch our announcement mailing list for future details.

A successful API token creation: a long string that only appears once, for the user to copy
Immediately after creating the API token,
PyPI gives the user one chance to copy it

Why: These API tokens can only be used to upload packages to PyPI, and not to log in more generally. This makes it safer to automate package upload and store the credential in the cloud, since a thief who copies the token won't also gain the ability to delete the project, delete old releases, or add or remove collaborators. And, since the token is a long character string (with 32 bytes of entropy and a service identifier) that PyPI has securely generated on the server side, we vastly reduce the potential for credential reuse on other sites and for a bad actor to guess the token.


Help us test: Please try this out! This is a beta feature and we expect that users will find minor issues over the next few weeks; we ask for your bug reports. If you find any potential security vulnerabilities, please follow our published security policy. (Please don't report security issues in Warehouse via GitHub, IRC, or mailing lists. Instead, please directly email security@python.org.) If you find an issue that is not a security vulnerability, please report it via GitHub.

We'd particularly like testing from:
  • Organizations that automate uploads using continuous integration
  • People who save PyPI credentials in a .pypirc file
  • Windows users
  • People on mobile devices
  • People on very slow connections
  • Organizations where users share an auth token within a group
  • Projects with 4+ maintainers or owners
  • People who usually block cookies and JavaScript
  • People who maintain 20+ projects
  • People who created their PyPI account 6+ years ago
What's next for PyPI: Next, we'll move on to working on an advanced audit trail of sensitive user actions, plus improvements to accessibility and localization for PyPI (some of which have already started). More details are in our progress reports on Discourse.

Thanks to the Open Technology Fund for funding this work. And please sign up for the PyPI Announcement Mailing List for future updates.

Thursday, July 11, 2019

2019 PSF Fundraiser - Thank you & debrief



Thank you to all who donated to our 2019 Fundraiser, Building the PSF.  Our fundraiser ended June 30th and we successfully surpassed our goal!

The PSF received over $75,000 in donations!


To learn about how the Python Software Foundation uses its funding, check out our Annual Impact Report: https://www.python.org/psf/annual-report/2019/.

What did we learn from this experience?

Collaborating with a company that supports our mission was a tremendous help. This was our first fundraiser collaborating with another organization. JetBrains sold PyCharm licenses at a discounted rate and donated all of the proceeds to the PSF. This partnership contributed $29,609 to this fundraiser. We thank JetBrains and their entire team for all of the effort and time they put into our fundraiser. This opportunity helped us raise more money and hopefully introduced new people to our community.

Additionally, we learned that having a strong beginning is important to reach our fundraising goal in an appropriate timeframe. Initially, the fundraiser was going to end at the end of May. Because we weren't close to reaching our goal of $60,000, we extended the fundraiser twice. Going forward we will have a better plan in place for an impactful start.

We want to hear from our community. If anyone has any suggestions or comments about our fundraisers, please don't hesitate to reach out. We aim to have an "open door" policy since our fundraisers directly impact our community.

Want to collaborate with the PSF on our next fundraiser?

As you know, the Python Software Foundation is a non-profit organization and depends on sponsorships and donations for revenue, which in turn support sprints, meetups, community events, Python documentation, fiscal sponsorships, software development, and community projects.

The PSF would love to partner with organizations to support our mission and help sustain a vibrant community. By working together, we hope to raise more funding so we can provide more community support! Contact sponsors@python.org for more information!

Tuesday, July 09, 2019

The Python Software Foundation is looking for bloggers!

Interview prominent Pythonistas, connect with the community, expand your circle of friends and learn about events in the Python world!


The Python Software Foundation (PSF) is looking to add bloggers for the PSF blog located at http://pyfound.blogspot.com/. As a PSF blogger, you will work with the PSF Communication Officers to brainstorm blog content, communicate activities, and provide updates on content progression. Example of content includes PSF community service awardee profiles, details about global Python events and PSF grants, or recent goings-on within the PSF itself. One goal of the 2019 - 2020 PSF Board of Directors is to increase transparency around PSF activities by curating more frequent blog content.

The Python Software Foundation is a 501(c)(3) non-profit corporation that holds the intellectual property rights behind the Python programming language. We also run the North American PyCon conference annually, support other Python conferences/workshops around the world, and fund Python related development with our grants program. To see more info on our grants program, please read: https://www.python.org/psf/grants/.


Job Description

  • Capacity to contribute one to two blog posts every three months
  • Passionate about Python and the global Python community
  • Independently report progress and activities to Python Software Foundation Staff and Communication Officers on a monthly basis
  • Actively brainstorm content ideas for blog content individually as well as with Python Software Foundation Staff and Communication Officers

Needed Experience

  • Ability to work independently and on virtual teams
  • Familiarity with Python programming
  • Experience contributing to a technical blog or website in English
  • Professional proficiency in English

Bloggers for the Python Software Foundation receive a fixed fee per post they write.

To apply please email two to three examples of recent articles (e.g. personal blog, contribution to professional publication) written in English as well as a brief description of your writing experience to psf-blog-apply@python.org. Please include in the email subject "PSF Blogger Application - (Your Name)". If you have questions, direct them to psf-blog-apply@python.org as well. The Python Software Foundation will be accepting applications until 11:59:59pm Pacific Standard Time Sunday, September 1, 2019.

Tuesday, June 18, 2019

PyPI Now Supports Two-Factor Login via WebAuthn

To further increase the security of Python package downloads, we're adding a new beta feature to the Python Package Index: WebAuthn support for U2F compatible hardware security keys as a two-factor authentication (2FA) login security method. This is thanks to a grant from the Open Technology Fund, coordinated by the Packaging Working Group of the Python Software Foundation.

Last month, we added the first 2FA method for users logging into the canonical Python Package Index at PyPI.org and the test site at test.pypi.org. Hundreds of project owners and maintainers have now started using that method (generating a code through a Time-based One-time Password (TOTP) application) to better secure their accounts.

Starting today, PyPI also supports (in beta) WebAuthn (U2F compatible) security keys for a second login factor. A security key (also known as a universal second factor, or U2F compatible key) is hardware device that communicates via USB, NFC, or Bluetooth. Popular keys include Yubikey, Google Titan and Thetis. PyPI supports any FIDO U2F compatible key and follows the WebAuthn standard. Users who have set up this second factor will be prompted to use their key (usually by inserting it into a USB port and pressing a button) when logging in. (This feature requires JavaScript.)

This is a beta feature and we expect that users will find minor issues over the next few weeks; we ask for your bug reports. If you find any potential security vulnerabilities, please follow our published security policy. (Please don't report security issues in Warehouse via GitHub, IRC, or mailing lists. Instead, please directly email one or more of our maintainers.) If you find an issue that is not a security vulnerability, please report it via GitHub.

We encourage project maintainers and owners to log in and go to your Account Settings to add a second factor. This will help improve the security of your PyPI user accounts, and thus reduce the risk of vandals, spammers, and thieves gaining account access. If you're not yet comfortable using a beta feature, you can provision a TOTP application for your second factor.

You'll need to verify your primary email address on your Test PyPI and/or PyPI accounts before setting up 2FA. You can also do that in your Account Settings.

2FA only affects login via the website, which safeguards against malicious changes to project ownership, deletion of old releases, and account takeovers. Package uploads will continue to work without users providing 2FA codes.

But that's just for now. We are working on implementing per-user API keys as an alternative form of multifactor authentication in the setuptools/twine/PyPI auth flows. These will be application-specific tokens scoped to individual users/projects, so that users will be able to use token-based logins to better secure uploads. And we'll move on to working on an advanced audit trail of sensitive user actions, plus improvements to accessibility and localization for PyPI. More details are in our progress reports.

Thanks to the Open Technology Fund for funding this work. And please sign up for the PyPI Announcement Mailing List for future updates.

Wednesday, June 12, 2019

2019 Board of Directors Election - Voting is Open

Voting is currently open for the 2019 Python Software Foundation Board of Directors Election. We have a great list of candidates this year so if you received a ballot, please vote.

Who should have received a ballot?

If you became a PSF Supporting Member*, Contributing Member, Managing Member and/or Fellow by May 31, 2019 you are eligible to vote. You should have received a ballot from Helios with details on how to cast your vote. If you cannot find the email, please search your inbox and also check your spam for the word "helios".

Once you login to Helios, be sure to follow the process until you see "Congratulations, your vote has been successfully cast!".

* Must be a current membership and not expired as of May 31, 2019

When do I need to vote by?

Voting opened June 7th and will close by the end of June 16 AoE.

How do I become a voting member?

If you're currently not a voting member but wish be to a voting member for future elections (2020 and on), here are some options for you to consider:

  • Contribute to the PSF $99 yearly by becoming a Supporting Member. You can sign up via http://psfmember.org.
  • If you dedicate at least five hours per month working to support the Python ecosystem you can become a Managing Member. If you dedicate at least five hours per month working on Python-related projects that advance the mission of the PSF you can become a Contributing Member. You can self certify via https://forms.gle/vbJvweHW8rimAjYd6. You must be a basic member before you apply to be a Contributing/Managing member.
  • If you know of someone that has gone above and beyond in their contributions to the Python community, consider nominating them for the PSF Fellow membership level. Details are available here: https://www.python.org/psf/fellows/.

If you have any questions about the PSF Election, please contact the PSF staff: psf-staff at python dot org.

--------------------------------------------------

The PSF is currently holding its 2019 Fundraiser. As a non-profit organization, the PSF depends on sponsorships and donations to support the Python community. Check out our Annual Impact Report for more details: https://www.python.org/psf/annual-report/2019/.

Please consider contributing to the PSF's 2019 fundraiser; we can't continue our work without your support! https://www.python.org/psf/donations/2019-q2-drive/.

Tuesday, June 04, 2019

Python Language Summit Lightning Talks, Part 2

The Summit concluded with a second round of lightning talks, which speakers had signed up for that day. These talks were therefore more off-the-cuff than the morning's talks, and several of them were direct responses to earlier presentations.

Read more 2019 Python Language Summit coverage.

Christian Heimes

SSL Module Updates




Python’s ssl module depends on OpenSSL. On Linux, Python uses the system OpenSSL, but on Mac and Windows it ships its own. Christian Heimes explained to the Language Summit that Python 3.7 must upgrade its included version of OpenSSL to 1.1.1 to receive long-term support, but he warned that this change might cause unexpected issues for Python programmers on Mac and Windows.

Heimes wants to deprecate support for TLS 1.1 as soon as possible. Recent Linux distributions and browsers already prohibit the old protocol for security reasons. In Python 3.8 he plans to document that TLS 1.1 “may work”, depending on the underlying OpenSSL version, and in Python 3.9 it will be explicitly banned. Larry Hastings asked whether this change could be spread over two Python releases, in the way that most features are first deprecated and then removed. Heimes replied that OpenSSL itself is moving this quickly.

Python has poor support for the root certificate authority files included in the operating system. On Linux and BSD, ssl.create_default_context() uses the root CAs correctly. On Windows, according to Heimes, root CAs are partly broken despite “a hack I added a couple years ago that does not work for technical reasons.” And on macOS there is no support without installing the certifi package. Heimes proposed to rely more on the operating system: on Mac and Windows in particular, the interpreter should ask the OS to verify certificates against its known CAs, instead of asking OpenSSL.

It has been several years since Heimes and Cory Benfield began drafting PEP 543. This PEP would decouple Python’s API from the specifics of the OpenSSL library, so it could use OS-provided TLS libraries on macOS, Windows, and elsewhere. Heimes told the Language Summit that he and Paul Kehrer would work on PEP 543 during the post-PyCon sprints.

Larry Hastings

Let’s Argue About Clinic


Argument Clinic is a tool used in the implementation of CPython to generate argument-parsing code for C functions that are used from Python; i.e., “builtin” functions. (It is named for a Monty Python sketch.) Since its original intent was to create C functions that handle their arguments like Python functions do, it only handles Python-like function signatures.

Larry Hastings addressed the Language Summit to ask whether Argument Clinic ought to be extended to handle argument parsing more generally, including function signatures that would not be possible in pure Python. For example, some builtin functions have parameters with a default value of NULL, which is representable in C but not in Python. Hastings said he had observed developers wanting to use Argument Clinic for all builtin functions because it is convenient to use and generates fast code.

Eric Snow

The C API



One of the reasons for CPython’s success is its powerful C API, which permits C extensions to interact with the interpreter at a low level for the sake of performance or flexibility. But, according to Eric Snow, the C API has become a hindrance to progress because it is so tightly coupled to CPython’s implementation details. He identified several problems with the current CPython implementation, such as the GIL, but said, “we can't go very far fixing those problems without breaking the C API.”

One solution is to split the C API into four categories. The CPython header files would be split into four directories to make it more obvious to core developers and extension authors which category each type or function belongs to:

  • “internal” — “Do not touch!”
  • “private” — “Use at your own risk!”
  • “unstable” — “Go for it (but rebuild your extension each Python release)!”
  • “stable” — “Worry-free!”
There are a number of other solutions proposed or in progress:
Snow finished by inviting interested people to join him on the C API special interest group mailing list.

Steve Dower

Python in the Windows Store



When a Windows user types python in the command shell on a clean system, the shell typically responds, “python is not recognized as an internal or external command”. After the May Windows update this will change: Typing python in the shell will now open the Microsoft Store and offer to install Python. When Steve Dower showed the install screen to the Language Summit the audience broke into applause.
screenshot

The package is owned by the Python core developers; Microsoft’s contribution was to add the python command stub that opens the install page. Compared to the installer that users can download from python.org, said Dower, “the Microsoft Store is a more controlled environment.” Its distribution of Python is easier to install and upgrade, at the cost of some inconsistencies with the full Python install. “It's not going to work for everyone.” Advanced developers and those who want multiple versions of Python will prefer to install it themselves, but the Microsoft Store will satisfy programmers who simply need Python available. “Everyone with a Windows machine in any of the tutorial rooms right now should probably be using this,” he said.

It had not been possible to install Python so conveniently on Windows, until recent changes to the Microsoft Store. For example, Store apps were originally prohibited from accessing their current working directory, but apps are now permitted virtually the same permissions as regular programs.

Carol Willing asked whether the Store version of Python could be used for reproducing data science results. “There are a number of situations where I would say don't use this package,” responded Dower. Since the Microsoft Store will automatically update its version of Python whenever there is a new release, data scientists who care about reproducibility should install Python themselves.

Nathaniel Smith

Bors: How Rust Handles Buildbots and Merge Workflow

(Or: One way to solve Pablo’s problems)


In response to Pablo Galindo Salgado’s earlier talk about the pain caused by test failures, Nathaniel Smith whipped up a talk about the Rust language’s test system. The Rust community observes what they call the Not Rocket Science Rule: “Automatically maintain a repository of code that always passes all the tests.” Although it is obvious that all projects ought to adhere to this rule, most fail to, including Python. How does Rust achieve it?

When a Rust developer approves a pull request, the “bors” bot tests it and, if the tests pass, merges the pull request to master.


This seems quite elementary, as Smith acknowledged. But there are two unusual details of the bors system that enforce the Not Rocket Science Rule. The first is that bors tests pull requests in strict sequence. It finds the next approved pull request, merges it together with master, tests that version of the code, and if the tests pass bors makes that version the new master, otherwise it rejects the pull request. Then, bors moves to the next pull request in the queue. Compared to the typical system of testing pull requests before merging, bors’s algorithm tests the version of the code that will actually be published.

The second way the Rust community enforces the Not Rocket Science Rule is by requiring the bors process for all pull requests. “They do this for everything,” said Smith. “This is how you merge. There's no green button.” Taken together, bors’s algorithm and the workflow requirement ensure that Rust always passes its tests on master.

Smith described some conveniences that improve the Rust developer experience. First, bors can be triggered on a pull request before approving it, as a spot check to see whether the code passes the test suite as-is. Second, since bors must test pull requests one at a time, it has an optimization to prevent it from falling behind. It can jump ahead in the queue, merging a large batch of pull requests together and testing the result. If they pass, they can all be merged; otherwise, bors bisects the batch to find the faulty change, alerts its author, and merges all the pull requests before it.

The Rust project currently uses a successor to bors called Homu, written in Python. There are several other implementations, including bors-ng, which is available as a public service for any GitHub repository.

Victor Stinner

Status of stable API and stable ABI in Python 3.8



Python 3.8 will improve the long-term stability of the C API and ABI for extension authors. Some of the details are settled, but Victor Stinner’s presentation to the Language Summit showed there are still many unanswered questions.

As Eric Snow had mentioned, C header files in Python 3.8 will be split into directories for the public stable APIs, the unstable CPython-specific API, and the internal API. Afterwards there will be less risk of exposing an internal detail in the public API by mistake, since it will be obvious whenever a pull request changes a public header.

CPython’s internal API headers were not installed by “make install” in the past, but it could be useful for a debugger or other low-level tool to inspect the interpreter’s internal data structures. Thus, in Python 3.8 the internal headers will be installed in a special subdirectory.

In the course of Stinner’s regular work at RedHat he often debugs problems with customers’ third-party C extension modules. A debug build of the extension module might not be available, but Stinner could gather some useful information by loading the extension module with a debug build of Python. Today, this is impossible: debug builds of Python only work with debug builds of extension modules and vice versa. The debug build of Python 3.8, however, will be ABI compatible with the release build, so the same extension modules will work with both.

Another motivation for updating the C API is isolation of subinterperters. Stinner referred to Petr Viktorin’s talk about removing process-wide global state for the sake of proper isolation, and possibly giving each subinterpreter its own GIL.

Attaining a clean, stable API and ABI may require breaking the current one; the core developers’ discussion focused on how much to break backwards compatibility and what techniques might minimize the impact on extension authors. The shape of Python 3.8’s C API and ABI are not yet settled. When Steve Dower asked whether Stinner was proposing a new stable ABI, Stinner answered, “I’m not sure what I’m proposing.”

Yarko Tymciurak

Cognitive Encapsulation

The Anchor of Working Together



Tymciurak began his lightning talk by complimenting the Language Summit participants. “In terms of communication skills, you do such a great job that I'm shocked sometimes.”

The factors that contribute to collaboration, he said, begin with a diversity of skills. As Victor Stinner had mentioned in his mentorship talk, a team with mixed skill sets and skill levels has advantages over a team of homogenous experts. Members of high-performing teams are also enthusiastic about common goals, they are personally committed to their teammates, and they show strong interpersonal skills.

Tymciurak credited Guido van Rossum for establishing the importance of teamwork from the beginning. Nevertheless, he said, “sometimes things may go off track.” The cause of irreconcilable disagreements or emotional blowups are not always obvious, but Tymciurak claimed that to him, “it's immediately obvious and really simple to fix.”

Cognitive encapsulation is the awareness that one’s experience of reality is not reality itself. “It’s your own mental model,” said Tymciurak. When we communicate, if explicitly share with others what we think, see, or hear, then we are respecting cognitive encapsulation. As Tymciurak describes it, “That’s being aware that my thoughts are my own.” On the other hand, if we assume that others already agree with us, or we represent our personal experience as if it is the only possible experience for the whole group, then encapsulation is violated and we are likely to cause conflict.

As an example of cognitive encapsulation at work, Tymciurak contrasted two types of communication. One is transactional. Someone asks, “Where’s the meeting?” You answer by saying which room it is in. Another type is control communication. If an instructor commands students to “turn to page 47,” then control communication is appropriate and the students will accept it. But when a team member uses control communication without the team’s acceptance, conflict arises. Tymciurak said, “When you tell someone else what to do, you're breaking the encapsulation. Be careful. There's times when it's appropriate. But be aware of when it's not.”

Another key practice that preserves cognitive encapsulation is to truly listen. Especially when the speaker is a junior teammate, it is crucial to be able to listen without agreeing, disagreeing, or correcting. Tymciurak described the outcome of a team that works together this way. Individuals know that they understand each others’ views, and they can advocate for their own views, speaking from their own experience. “Then you can speak with authority and power. And that's part of the magic of encapsulation.”

Monday, June 03, 2019

Python Language Summit Lightning Talks, Part 1

The Summit began with six pre-selected lightning talks, with little time for discussion of each. Five of them are summarized here. An upcoming article will cover Pablo Galindo Salgado's lightning talk on improvements in Python's test infrastructure.

Read more 2019 Python Language Summit coverage.

Jukka Lehtosalo

Writing Standard Library C Modules In Python



Jukka Lehtosalo described his work with Michael Sullivan on an experimental compiler called mypyc.

The Python standard library, Lehtosalo said, contains the modules that most programmers use by default, so it should be fast. The main optimization technique has historically been to write C extensions. So far, 90 standard library modules are partly or entirely written in C, often for the sake of speed, totaling 200,000 lines of C code in the standard library. But C is hard to write and error prone, and requires specialized skills. “C is kind of becoming a dinosaur,” he said, provoking laughter from the core developers.

As an alternative, Lehtosalo proposes “writing C extensions in Python.” The mypyc compiler reads PEP 484 annotated type-checked Python, and transforms it into C extension modules that run between 2 and 20 times faster than pure Python. Some of Python’s more dynamic features such as monkeypatching are prohibited, and other features are not yet supported, but the project is improving rapidly.

The project has a similar goal to Cython’s: to transform Python into C, which is then compiled into extension modules. Compared to Cython, however, mypyc supports a wider range of PEP 484 types such as unions and generics. In Lehtosalo and Sullivan’s experiments it offers a greater performance improvement. They propose further experimentation, testing how well mypyc translates certain performance-sensitive standard library modules, such as algorithms, random, or asyncio. The translated modules could be published on PyPI first, rather than replacing the standard library modules right away. If the test goes well, mypyc would offer “C-like performance with the convenience of Python.”

Core developer Brett Cannon suggested an experiment using some module, such as datetime, that is already implemented in both Python and C. The Python version could be translated with mypyc and then pitted against the handwritten C version.

Matthias Bussonnier

Async REPL And async-exec



Python’s interactive shell makes it easy for beginners to learn Python, and for all Python programmers to experiment as they develop. However, async Python code is practically unusable with the shell. The await keyword must be used within a coroutine, so a programmer who wants the result of an a waitable object must define a coroutine and run an event loop method to execute it.

Matthias Bussonnier presented his work, which integrates async and await into the alternative IPython shell. IPython permits the await keyword at the top level, so a user can get the results of coroutines or other awaitables in the shell without defining a coroutine:
In [1]: from asyncio import sleep

In [2]: await sleep(1)

In [3]: from aiohttp import ClientSession

In [4]: s = ClientSession()

In [5]: response = await s.get('https://api.github.com')
IPython supports asyncio and other async frameworks such as trio. In the future, a plugin system will allow any async/await-based framework to be usable in the shell.
Bussonnier argued that some of his ideas should be adopted by core Python. If asynchronous coding were convenient in the shell, it would be useful for educators, and it would remove what he considers the misconception that async is hard. Best of all, Python would get ahead of Javascript.

However, to support async and await in the shell currently requires some unsatisfying hacks. There are subtle issues with local versus global variables, background tasks, and docstrings. Bussonnier has filed issue 34616, implement "Async exec", to make full async support in the shell possible.

Update: After the Language Summit, Bussonnier and Yury Selivanov updated the Python compiler to permit await, async for, and async with as top-level syntax in the shell when executed like python -m asyncio:


Jason Fried

Asyncio And The Case For Recursion



A typical asyncio application has a single call to run_until_complete() near the top level of the application, which runs the asyncio event loop for the entire application. All code beneath this level must assume that the loop is running.

Facebook engineer Jason Fried presented to the Language Summit a scenario in which this application structure fails. Consider an async application that contains a mix of async code and blocking calls that are tolerably fast. Deep within the call stack of one of these blocking calls, a developer sees an opportunity for concurrency, so she adds some async code and executes it with run_until_complete(). This call raises “RuntimeError: This event loop is already running.” As Fried explained, any call to run_until_complete() in a call chain under async def has this result, but due to modularization and unittest mocking in Facebook’s Python architecture, this error can first arise late in the development cycle.

How should this problem be avoided? The asyncio philosophy is to avoid mixture by converting all blocking code to asynchronous coroutines, but converting a huge codebase all at once is intractable. “It's a harder problem than moving from Python 2 to 3,” he said, “because at least I can go gradually from Python 2 to 3.”

Fried suggested a solution for incrementally converting a large application, and to allow developers to add asyncio calls anywhere “without fear.” He proposed that the asyncio event loop allow recursive calls to run_until_complete(). If the loop is already running, this call will continue running existing tasks along with the new task passed in. Library authors could freely use asyncio without caring whether their consumers also use asyncio or not. “Yeah sure it's ugly,” he conceded, “but it does allow you to slowly asyncio-ify a distinct code base.”

Thomas Wouters objected that this proposal would violate many correctness properties guaranteed by the current loop logic. Amber Brown concurred. She explained that Twisted’s loop prohibits reentrance to ensure that timeouts work correctly. One of the core tenets of asynchronous programming is that all tasks must cooperate. There is no good solution, she said, for mixing blocking and async code.

Mark Shannon

Optimising CPython, Or Not



“Every few years someone comes along with some exciting new potential for speeding up CPython,” began Mark Shannon, “and a year later everyone's forgotten about it.” Some of these optimizations are worth pursuing, however. We can identify promising optimizations with a heuristic.

First, Shannon advised the audience to think in terms of time, not speed. Do not measure the number of operations Python can execute in a period; instead, measure the amount of time it requires to finish a whole task and divide the total time into chunks. As an example, Shannon described a recent proposal on the python-dev mailing list for implementing a register-based virtual machine, which would store local variables in fixed slots, rather than on a stack as the Python VM does today. How much time could such a change save? Shannon walked the audience through his thought process, first estimating the cost of the Python interpreter’s stack manipulation and guessing how much cheaper a register-based VM would be. Shannon estimates that up to 50 percent of Python’s runtime is “interpretive overhead,” and a register-based VM might reduce that significantly, so it is worth trying. However, only an experiment can measure the actual benefit.

Shannon compared the register-based VM to another optimization, “superinstructions.” The technique is to find a common sequence of bytecodes, such as the two bytecodes to load None onto the stack and then return it, and combine them together into a new bytecode that executes the whole sequence. Superinstructions reduce interpretive overhead by spending less time in the main loop moving from one bytecode to the next. Shannon suspects this technique would beat the register-based optimization.

In conclusion, Shannon advised the audience that the next time another Unladen Swallow or similar project appears, to determine first which part of the interpreter it optimizes. If the optimization targets a part of the interpreter that represents less than 90% of the total runtime, said Shannon, “it’s pretty much doomed to fail.”

Łukasz Langa

Black under github.com/python



The past year has been marked by controversy in the Python community, but consensus is forming on the most unexpected topic: code formatting. Łukasz Langa’s Black code formatter is only a year old, but it has been adopted by pytest, attrs, tox, Django, Twisted, and numerous other major Python projects. The core developers are enthusiastic about Black, too: When Langa introduced himself as its author, the room broke into applause.

Langa proposed moving black from his personal repository to the Python organization on GitHub. He said, “My goal for this is to provide a suitable default for users who don't have any preexisting strong opinions on the matter.”

Some core developers dissented, arguing that since Black is already so successful, there is no need to move it. Gregory Smith said it is not the core team’s role to bless one code formatter over others; he regrets that opinionated tools like mypy are in the official organization and he opposes adding more. Guido van Rossum suggested moving it to the Python Code Quality Authority organization; Langa responded that beginners haven’t heard of that organization and moving Black there would have no effect.

Update: Despite some objections at the Language Summit, Black is now in the official Python organization on GitHub.

Pablo Galindo Salgado: The Night's Watch is Fixing the CIs in the Darkness for You


Python is tested on a menagerie of “buildbot” machines with different OSes and architectures, to ensure all Python users have the same experience on all platforms. As Pablo Galindo Salgado told the Language Summit, the bugs revealed by multi-platform tests are “Lovecraftian horrors”: race conditions, bugs specific to particular architectures or compiler versions, and so on. The core team had to confront these horrors with few good weapons, until now.

Read more 2019 Python Language Summit coverage.

The Solemn Duty of Bug Triage


When a test fails, the core developer who triages the failure follows an arduous process. “It's not glamorous by any means,” said Galindo, “but someone needs to do it.” Galindo, Victor Stinner, and Zachary Ware are the main bug triagers, and they all follow a similar sequence: read the failure email, search for duplicate failures, read the voluminous logs to characterize the problem, and file a bug with a detailed description. Then, optionally, they try to reproduce the problem. Since failures are often specific to one buildbot, the triagers must contact the buildbot’s owner and get permission to ssh into it and debug.

According to Galindo, typical test failures are “really, really complicated,” so the triage team takes a firm stance about reverting changes. If they suspect that a change has broken a test, its author has one day to follow up with a fix or the change will be reverted. “Nobody likes their commits to be reverted,” he told the Language Summit. But test failures can cause cascading failures later on, so the team must be ruthless.

New Tools for Squashing Bugs


A pull request is not tested by the buildbots until after it is merged, so the author does not immediately know if they have broken any tests. Galindo and his colleagues have written a bot which reacts to a test failure by commenting on the merged pull request that caused it, with reassuring instructions to help the panicked author respond. “We have some arcane magic,” he said, to distinguish compiler errors from tracebacks and neatly format them into the message, so the author get begin diagnosing immediately.


Since the bot was deployed in September, the mean time to fix a test failure has fallen dramatically. When Galindo showed this chart, the core developers broke into applause.


Nevertheless, there are still severe problems with Python’s tests. Flaky tests break about 40% of the builds; the system is programmed to retry a failure and consider it successful if the second run passes, but this is clearly just a stopgap. Galindo urged the core team to reduce flaky tests by eliminating race conditions and sleeps. He also asked for help writing a tool that would analyze which flaky tests fail most often, and a tool to detect and merge duplicate test failures.

Finally, Galindo proposed allowing contributors to test their pull requests on the buildbots before merging. This feature should be implemented cautiously. “The buildbots are very delicate,” he said. They cannot safely run arbitrary code like on Travis or other commercial test infrastructures. Still, it would be worth the effort, if contributors could catch mistakes before they are merged.