Paul Ganssle @pganssle@qoto.org

@freemo I am not sure which alternate interpreters don't have a GIL, but I would be shocked if they had the level of compatibility with third party (and private/proprietary) libraries and applications that would be required for upstreaming into CPython.

Another issue is that you may be able to get perfect compatibility at the cost of performance degradation for everything else. Having no GIL but being 30% slower is not a good trade-off for most people, particularly in a mature software ecosystem that evolved with the presence of the GIL.

@freemo Yes and yes.

PyPy, for example, does have a GIL, and has an extensive compatibility layer to support the C API. My understanding is that for a long time PyPy didn't work *at all*, or worked very poorly, when used with anything that uses the C API.

Even now, it's touch-and-go, and many third party libraries aren't tested against PyPy.

@freemo Probably the most promising thing on the horizon for the GIL removal (and many other problems caused by the way the C API works) is HPy: https://github.com/hpyproject/hpy

That still basically involves rewriting all C extensions to use handles instead of manually managed reference counts, and there is very little appetite for another "break the universe" change when people generally have a number of good solutions for this problem already.

@freemo The hard part isn't fixing it for random non-standard interpreters. The hard part is fixing it in the *core* interpreter without breaking all the stuff built on top of it.

Like, it's easy to swap out the tires on your car when the thing is on a jack and not even fully assembled. It's a lot harder to swap them out while driving down the highway at 65 miles per hour while rushing someone to the hospital 😛

@freemo Well, regardless, I'd like to think I'd feel similarly even if I weren't a CPython core dev (and I have no personal stake in the GIL, I didn't design it and I didn't even become a core dev until well after the 2→3 migration).

In my time in open source, I've found that a lot of the stuff that seems obviously idiotic actually frequently has some real, hard problems at its core. I gave a keynote at PyConf Hyderabad about this recently: https://www.youtube.com/watch?v=aRqulQUgiIA

My experiences have really emphasized the value of humility in evaluating technical decisions.

@nate_river @freemo It's fairly common with Python and I think other interpreted languages.

Cython in particular is barely a different language.

Consider this fairly-optimized Cython compared to Python: https://gist.github.com/pganssle/d0dab1ce513b092d009bf3d819e1ddef

The Cython code compiles to C++ and the resulting Python function is something like 10x faster. Hand-crafted Rust or C can be much faster, but isn't always quite as straightforward to embed.

@freemo @nate_river By the way I did not necessarily recommend C. I've given several talks about writing Python extensions in Rust that are relevant here: https://ganssle.io/talks/#python-backend-rust

For your cases, I would guess Cython would be sufficient, and Cython is very nearly as easy to read and write as Python is.

One of Python's great strengths is that it is super easy to wrap bindings to other languages, so your API can be written in Python (easy to read and write) and your backend can be written in C or Rust or some other more complicated language.

I find there's often some 80/20 rule where 80% of the work is done by 20% of the code. Speeding up that 20% by writing it in Cython is trivial compared to writing the other 80% of the code in C or, to a lesser extent, Rust.

@freemo Not saying there's nothing worth complaining about, TBH, but I've been involved in a lot of language design discussions in Python lately, and I've actually killed at least one "white whale", see, e.g. this thread, which resolved an issue that another `datetime` maintainer said he tried to fix *10 years earlier* and was stymied: https://mail.python.org/archives/list/python-dev@python.org/thread/74WMZ4I3KFTOODAFPUBNSFBXKHPYCLSM/#ENTRVAFELVB4OQGXWUAOTUQALWLIQGAF

The fact of the matter is that a lot of the stuff that superficially looks sloppy is actually part of a complex and consistent ecosystem. It's third-order consequences of something, or it's something that was designed in another context but cannot be changed for backwards compat reasons.

And frankly, I wouldn't be surprised to find that a lot of the stuff that makes Python really good is tied up in the stuff that makes it need wrapper libraries and such.

@freemo Sorry, I think I may not have fully understood the context here. It sounded like you were being forced to use Python for some reason and were complaining about this need.

I think you may have a better experience if instead of complaining about why it doesn't fit your preconceptions you asked, "How do people do X?" Usually if other people complain, "Oh yeah that's basically impossible", then yeah it's a sore spot 😛

@freemo It's incredibly popular and widely used by an *increasing* fraction of the ecosystem for a good reason.

Your criticisms are not uncommon, and they tend to be leveled at Python by people who come from other languages and then don't try and learn the "Python way" to do things.

In any case, I am not a Python maximalist. I suspect that you are not really giving Python a fair shot, or are judging based on some subjective measure like aesthetic, but you have every right to your opinion.

If you think you are going to need to use Python a lot in the future for whatever reason, it would probably be a good idea to start coming up with objective measures of what you dislike about it and see how experienced Python programmers would solve the problems. You may be surprised with how competent and versatile Python actually is.

@freemo In a lot of cases, that lower level language is actually just Cython, which is a superset of Python that compiles to C and basically reads and writes like Python.

And in most cases, that lower level language code is already written *for you*, since you can usually find a language that does most of the hard work for you in C. Numpy, scipy, pandas, scikit-learn, etc, are all examples of this from the scientific/data stack, but there are equivalents in other stacks as well.

If you want to do a lot in parallel, check out Dask.

@freemo Ripping out the GIL has been the subject of multiple multi-year projects that have failed for various reasons.

Honestly, I would love to see the GIL gone, but I'm not sure how much better that would make things in a lot of cases. Python tends to be slow for a number of reasons, so the most common idiom is that when you need real performance on something CPU bound, you drop into a lower level language like Rust or C (at which point it's very easy to drop the GIL, incidentally *also* making it easier to parallelize).

@freemo FYI this kind of thing is kinda disrespectful. Python is a remarkably successful language and a lot of incredibly smart people work very hard on it.

When you start working on a project as widely used and complicated as Python, you realize that backwards compatibility concerns very quickly make seemingly simple things incredibly difficult.

@alex

@freemo I mean, there's sortedcontainers, which is very popular and seems relatively well-maintained. It's implemented in pure python and I don't think it's a red-black tree, so if it's a performance bottleneck then you may need something else, but if you just want to simplify your code, then that should work.

@freemo I don't know the specifics of your situation so I may be way off base, but I often find that when people start using a new language or ecosystem they reach for the familiar idioms from their more comfortable language and don't realize that in this new language you are meant to structure things differently. I have been guilty of this myself many, many times.

@freemo I am a very active coder, what I mean to say is that I have basically never been in a situation where my code was slow enough that it was a problem (and I do a lot of low-level library programming so that's often "slow at all") and benchmarks showed that dictionary hash map lookup is the bottleneck and switching implementations would fix it.

Python has a huge amount of overhead for most operations, so often the way to get high performance when you need it is to basically call an API that is a wrapper around an optimized implementation in a low-level language (e.g. numpy).

Admittedly, it may also be that the kind of programming I do is not likely to hit this problem, but I imagine the fact that there's *not* an obvious choice here means that it's either not a very common problem or this is an XY problem and you are overlooking more idiomatic solutions to your overall problem.

@freemo I don't think I've ever needed such a thing, I suspect most people haven't, which is why there's nothing built-in.

You may find what you want by looking instead at how people solve similar problems in Python. Might be something out there that has a red-black tree or similar at its core but isn't advertised as such.

@eric That article doesn't mention the geographical extent of the outbreak. Everything I'm finding seems localized to the Pacific Northwest (Washington, Portland, BC). Any idea if the northeast is affected as well?