Why I want people to donate to the PSF

The Python Software Foundation is having its annual donation drive (on track to reach its goals), and I want to talk a little about why I really want people to donate this year, more so than before. I posted this as a long thread on twitter originally, but several people asked me to paste it in a blog for easier reading. I've tweaked a bit of the text to fit the different medium, but the message is the same.

TLDR: I'll double-match your donations to the PSF.

The PSF was created nearly twenty years ago because Python grew to a point where it needed a legal custodian for its IP/trademarks. I was a (new) core developer at the time, so I became a founding PSF member, and then got elected to its first Board of Directors (and second/third).

The PSF was originally modelled after the Apache Software Foundation, which Greg Stein (another core dev & founding board member) was active in. We did a lot of things because they did it (like the original membership model), and I do believe it gave us a massive head start. I don't think many of us, at the time, really knew what we were doing. We had ideas of what we wanted the PSF to be, but not sure how we would get there. We were also all just volunteers, most with little experience in non-profit work. We learned a lot, though, or at least I did.

The early PSF had no staff, not much money, and not much to spend money on. I think we had an idea of growing the PSF, pulling in corporate sponsorship to pay for Python development, but we didn't know how to get there. We didn't really have the infrastructure to spend money. It took years and a lot of trial and error to build that infrastructure. I stepped down from the board after 3 years, but kept involved with the PSF. The PSF turned itself into a community support vehicle, spending money on outreach and local communities, as well as PyCon US.

Then the PSF hired Ewa Jodlowska as Secretary, then as Director of Operations, then as Executive Director, and suddenly we could manage staff! Hire people! Organise volunteers! Ewa has been immensely valuable in pushing the PSF forward. Others, too! Lots of good volunteers. The PSF also changed its membership model, going from purely voted in members (what are now Fellows) to a more open model, giving voting rights to anyone who put in effort, organising power, or money. We dropped the idea of corporate members, making them non-voting sponsors.

The PSF now has 7(!!) employees (mostly part-time), to take care of finances, tax rules, fundraising, organising PyCon US, all the technical infrastructure of python.org and of core development . But it's all supporting the volunteer community, not doing their work. Because of that staff expertise, we can now offer fiscal sponsorship, allowing Python projects to offer tax-deductible donations (in the US) without having to set up their own non-profit or all the financial scaffolding that requires.

Also because of the staff expertise, as well as a lot of volunteer effort, we've been able to get specific projects funded, and hired contractors to do the work (a lot of it on PyPI, which has made tremendous leaps forward because of it). PyPI's last ~5 years have been amazing.  One of the things the Python Core Developers have been wanting to do is move from bugs.python.org to GitHub Issues (see PEP 581 and PEP 588). Thanks to Ewa and the PSF, as well as GitHub itself, that's now actively being worked on.

Similarly, the Python Steering Council and the PSF have facilitated Kyle Stanley's Core Developer internship (https://twitter.com/aeros_py/status/1337223467754807296). We wouldn't have been able to offer the support an intern needs without the growth of the PSF and the investment in people like Ee Durbin, the PSF's Director of Infrastructure.

One of the next steps we want to take here is to hire people to work on Python itself. Actual developers to assist the volunteers, to do some of the less attractive jobs or the more long-term projects. We had planned to fundraise for this starting at last PyCon US, but... well...

Hiring people is a serious investment. It's not just that you have to pay them, you have to offer them support, room to grow, worthwhile goals, empower them. You also have to give them job security. A fixed contract is fine for project work, but that's not what we're after here. We want to hire core developers, which we now have the infrastructure and support for, but not just for a year. We want to make long-term investments in long-term employment for long-term benefits to Python. We need more than a year's or two of funding for this.

We need the PSF to be in a good financial state to support this. We don't want to be forced to fire anyone, and we don't want to bleed the PSF dry. COVID-19's effects on PyCon US were a big blow to our plans. The PSF survived it pretty well, but it definitely changed the prospects. The PSF is looking to improve its financial situation, to rely less on PyCon or donations. We have a number of things we're considering, in fact, but we have to do this all with great care to avoid alienating the Python community, not to mention tax laws. It's work in progress.

So, please! If you can, donate to the PSF. If you go to https://matcher.pyfound.org afterwards, I'll triple your donation. (If your company uses something like Benevity and you want me to match you, let me know and we'll figure something out.)

Also, if you're a corporate user of Python, consider getting your company to support the PSF. Lots of companies have done so already, many for years. Thank you very much! But please, consider giving more ;-) The PSF is putting it to really good use.

My view of the Python Steering Council

Python 3.9 has been released, which means it's time to elect a new Python Steering Council. I want to give potential candidates an idea of what to expect -- similar to what Brett Cannon posted last year, but with my own point of view. I also want to highlight what we did and what I would like the next SC to do.

I joined the Steering Council in January 2020. It's only the second year we have a Steering Council, so we are still in the process of figuring out expectations, standards and practices. The first SC provided a good base for this, and the fact that three members of the first SC were re-elected helped with continuity. For those who don't know, I'm not a stranger to this kind of thing: I've been a Python core developer since somewhere in 2000, a founding member of the Python Software Foundation, and a member of the PSF Board of Directors for a number of years (2001-2004, 2017-2019, 2020-onward). That said, the Steering Council is distinctly different from those roles.

Besides me, the 2020 SC consists of Barry Warsaw, Brett Cannon, Carol Willing and Victor Stinner. Like the first SC, this SC decided to meet (via video chat) for an hour every week, and have Ewa Jodlowska (PSF's Executive Director) attend to take notes and facilitate the co-operation with the PSF. Because of the frequency of the meetings, most discussion took place there rather than in email. We also used collaboration tools (primarily Google Docs) to draft emails, announcements and responses, so that we could provide and process feedback asynchronously.

This year's SC neglected to keep the community as up-to-date as was done previously, which I regret. There were a number of causes for that: we had some long, difficult discussions without a reportable result (like the long-term vision we've been working on). COVID-19 ruined or complicated our ideas for presenting our plans and discussing them with the community. We also discussed two serious Code of Conduct issues, which are hard to report on, emotionally and practically. Even so, I wish we had done a better job of keeping the community appraised. We've done so retroactively now, but we should have done this every quarter, like the last SC, or preferably even every month. That is something I would like the next SC to take more seriously.

Despite that regret, I am proud of what the SC has achieved in the last year, and stand behind all of our decisions.

• We've had in-depth discussions about various PEPs and accepted a number of them. We kept an eye on open PEPs and their progress, asking PEP authors and sponsors for updates periodically. We've continued the approach of designating PEP delegates for PEPs requiring domain-specific knowledge, and keeping PEPs for our own pronouncement when they're trivial and obvious, or when they're contentious and difficult decisions. The biggest and most controversial change on our plate is the Structural Pattern Matching proposal (originally PEP 622, now PEPs 634, 635 and 636, and relatedly 640 and 642), and we've discussed this at length, both between ourselves and with the PEP authors. We haven't made our decision on that proposal yet. We did decide that considering its nature, this is not something we can hand off to a PEP delegate. We also decided that at this point, so close to the next SC elections, we can't in good conscience make the final decision on this. After all, the next SC will be in charge of the next Python release, and there are still five months before the deadline for 3.10. We will make our decision, but leave it as a recommendation for the next SC.
• We also continued plans to actually spend money on core development (and solicit donations or sponsorship for it) by hiring core developers to focus on the less-desirable work. We surveyed the core developers about this plan back in January (https://mail.python.org/archives/list/python-committers@python.org/thread/XJ373N2H5O2OXMEQEEGYIIZ3U7RNHVHJ). The plans to find funding for this were stymied by COVID-19, but we still want to continue that idea, and we're hoping to draw in more sponsorship in the coming years. (The PSF plays a big role in that as well, but I keep my SC role and my PSF Director role separated. It hasn't been necessary much, but I very much try to avoid conflict of interest situations.)
• We've dealt with two major Code of Conduct incidents, where the PSF's Code of Conduct WG recommended certain actions. One of these incidents I was personally involved with and so the WG recommended (and I concurred) that I recuse myself from discussions of its recommendations. Even so, I'm glad the SC (unanimous but for my recusal) took the actions it took, and stand behind them. The other incident was more difficult and more complicated, but the SC was unanimous in the handling of that incident as well. In the handling of both of these cases the SC made choices not just for the specific incident, but also to set standards, procedures and expectations for any future cases.
• With Ewa's (and the PSF's) substantial help, we've hired a project manager to help shepherd the GitHub Issues migration (PEP 581), funded by a GitHub donation and managed by the PSF. We managed to hire an existing core developer with intimate knowledge of bugs.python.org (Ezio Melotti), which fills me with tremendous hope for the project. Of course, the ability to move back out of GitHub in the future, should the need arise, is a part of that project.

Going forward, I very much want the Steering Council to continue along these lines. Managing PEPs, drawing up long-term plans, Code of Conduct enforcement, and spending money on (and consequently acquiring funds for) core development are all crucial to the success of CPython, the SC model, and the core developer community. Even so, there are two things I wish we had managed to do better:

• Communication with the rest of the core developer community and the larger Python community, both in keeping everyone appraised of our discussions and hearing their concerns. As five core developers, the current SC naturally kept up to date on discussions in the core developer community, but representation in other parts of the Python community wasn't as good as it could have been. I think it would be very healthy for the SC to have one or more non-core-developer members (which, in case you're wondering, is specifically allowed in PEP 13).
• Similarly, I wish we had kept closer contact with specific projects that are closely related to CPython: other Python implementations, projects like Beeware, Jupyter, the various type checkers, IDEs, etc. It's not like we weren't in contacts with them at all, but the cancellation of PyCon 2020 as an in-person conference had a significant impact on this. We missed out on a lot of in-person contact and I wish we had managed to compensate for that with virtual meetings or email discussions. Since in-person meetings are unlikely to happen in 2021 as well, at least the in first half, I hope the next SC can improve on this.

With regards to the biggest hot potato, the Structural Pattern Matching proposal, I haven't made my own decision yet. I am excited for pattern matching, especially the way the proposal delegates matching behaviour to the types being matched. I see how this will allow libraries to provide simpler, less error-prone APIs to complex code, similar to the benefit provided by the with statement. Even so, I have serious reservations because of the incongruity of the proposal with the rest of Python. The latest iteration of the proposal alleviates some of these problems, and I believe I can now live with the unfortunately vague distinction between what is or isn't assignment in the pattern matching cases. My main concern at this point is the wildcard pattern, which is why I proposed PEP 640 -- however, even with PEP 640 I've not yet decided whether the utility of pattern matching is actually worth the not-insubstantial cost of adding the new feature. As I said, the SC will leave a recommendation for the next SC, but I believe it will be a tough decision for them, as well.

[EDIT: I've since posted a longer email with my current views on Structural Pattern Matching to the python-dev mailing list.]

I say "them", but it might be "us", as I will in fact be running again in the next SC elections. Nominations for the SC election are currently open. Nominations have to be made by core developers (they can self-nominate) -- if you're not a core developer but looking to be nominated, feel free to talk to me. I encourage everyone who wants to make an impact to consider running. It might be hard work with difficult problems, but it's well worth it.

Super-wrong

Every year for the last couple of years, at or around PyCon US, I run into code that uses super(type(self), self)(or the equivalent super(self.__class__, self)). I don't go looking for it, it's just coincidence. Every year, at PyCon US, I intend to give a little lightning talk about why this is wrong, and how super-wrong it is, and perhaps shame a bunch of big-name projects that show up in a codesearch. (Originally Google Code Search, but as +Aaron Gallagher, who reviewed this article, pointed out to me, Github's search makes a good enough replacement.) Every year I look at the long, long signup sheet for the lightning talks and I think "oh, why bother, everyone knows this is wrong anyway" and leave my hypothetical spot for something much more awesome.

And every year, shortly after PyCon US, I regret my decision. This year it was at dinner during the sprint days, when I mentioned the super-wrongness of super(type(self), self) and someone who shall remain anonymous said "wait, what? I was always told to do it that way." So, clearly, I need to say this explicitly.

If you see super(type(self), self), the code is doing it wrong. It is a bug. It is super-wrong.

I understand why people do it super-wrong. super(ThisClass, self) is annoyingly verbose and repeats the class name and it doesn't work when you do things like rebind the ThisClass name. But super(type(self), self) has a bigger problem: it does the wrong thing for all subclasses of your class. In fact, if I see code that uses super(type(self), self), it tells me two things: there's a bug in the code and a glaring gap in the tests.

How is it broken, you ask? If you use super() super-wrong, your class will appear to work:

>>> class C(object):
...     def spam(self):
...         pass
...
>>> class D(C):
...     def spam(self):
...         print "In D.spam"
...         super(type(self), self).spam()
...
>>> D().spam()
In D.spam
>>>

But as soon as you involve a subclass, even one that doesn't define the spam method, it will fail:

>>> class E(D):
...     pass
...
>>> E().spam()
In D.spam
In D.spam
[...]
In D.spam
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in spam
  File "<stdin>", line 4, in spam
[...]
  File "<stdin>", line 4, in spam
RuntimeError: maximum recursion depth exceeded while calling a Python object

So why does it break? super() is a way of telling Python "call the method that would have been called had this class not defined it". It does that by giving you a proxy object (an instance of the super type) that continues attribute lookup, starting from just after the class you give as first argument. (If you want the full story on how attribute lookup works, +Guido van Rossum gave a nice explanation a while ago.)

So super(ThisClass, self).spam() says "skipping just past ThisClass, call the spam attribute of self." And super(type(self), self).spam() says "skipping just past the actual class of self, call the spam attribute of self." If type(self) is ThisClass, that is the same thing. If type(self) happens to be a subclass of ThisClass, this will end up calling the same method again, and again, and again, and your code will recurse until Python stops you.

Of course, this is all assuming you're using super() to call the same method in one of the baseclasses. There are other ways of using super() incorrectly. For example, if you're using super() to call another method altogether, you should really ask yourself why that would need to skip just the first class in the method resolution order, regardless of what it is -- and if you come up with a good answer, please tell me; I'd be interested to find cases where this isn't just a plain bug. If you're using super() in situations where you know for a fact that the class will never be subclassed (something I've never managed to know in any of the code I wrote) you should ask yourself why you're using super() at all, and what hoops you want to jump through when you change your mind. I'll bet it's not going to be worth the minor short-term convenience of not repeating a class name.

Bottom line, if you want to use super() -- and you should -- then you can't pass type(self) or self.__class__ as the first argument. If you aren't using Python 3 (in which you can just use super() with no arguments), that means repeating the name of the class. There are things you could do that aren't obviously wrong, like the self.__super trick Guido suggested here, but they're complicated and still fragile and frankly just not worth it. Or you could decide not to use super() at all, but then you're back to repeating the baseclass name(s) instead and nobody wants that. Using super() the most straightforward way, despite the repetition of the class name, is by far the best solution.

Now let's fix all the code, so I won't have a reason to give a lightning talk next year.