🇳🇱
Have I mentioned yet today how much I hate python multiprocessing, or writing cpu-limited algorithms on python in general... my god why could they not fix the GIL lock problem in python3, are the devs really this lazy!
#programming #Python #algorithms #multithreading #threads #threading #Python3
@freemo FYI this kind of thing is kinda disrespectful. Python is a remarkably successful language and a lot of incredibly smart people work very hard on it.
When you start working on a project as widely used and complicated as Python, you realize that backwards compatibility concerns very quickly make seemingly simple things incredibly difficult.
@freemo Ripping out the GIL has been the subject of multiple multi-year projects that have failed for various reasons.
Honestly, I would love to see the GIL gone, but I'm not sure how much better that would make things in a lot of cases. Python tends to be slow for a number of reasons, so the most common idiom is that when you need real performance on something CPU bound, you drop into a lower level language like Rust or C (at which point it's very easy to drop the GIL, incidentally *also* making it easier to parallelize).
@freemo In a lot of cases, that lower level language is actually just Cython, which is a superset of Python that compiles to C and basically reads and writes like Python.
And in most cases, that lower level language code is already written *for you*, since you can usually find a language that does most of the hard work for you in C. Numpy, scipy, pandas, scikit-learn, etc, are all examples of this from the scientific/data stack, but there are equivalents in other stacks as well.
If you want to do a lot in parallel, check out Dask.
@pganssle In this case I am already using numpy and scipy, and pandas which i think also has some low level bindings.
As far as I can tell none of them do parallel processing out of the box (if they did I probably wouldnt need to do it myself). There were some projects that add it on, parallellal or whatever it was called, modin, and I think one other. Tried all three none of them really helped and had marginal improvements.
With that said I **have** been able to multiprocess this and the performance has went from 2 hours to process a year simulation down to 10 minutes. Not as good as i could get with better parallelization tools, but good enough to work with it for now. My main issue is how convoluted multiprocessing is rather than if it can do it at all.
I tried Dask through modin and it didnt do much speedup, I may need to just bite the bullet and try pure Dask at this point, though im not sure if it will solve my problem. Even if i could get my numpy and pandas operations parallel the core of the issue is realtime processing of streaming data effectively and neither numpy nor pandas handles that well, which suggests to me Dask either wont fix the issue either or if it does it will be even more convoluted than just using multiprocessing.
Either way, for now i am able to get things working with multiprocessing but I really dont like what the code looks like to get it there, or the compromises I had to make that limit me from seeing better performance.
@pganssle Oh and as for dropping into a low level language, if i were to do that, particularly on this project where the multiprocessing component cant be easily isolated fromt eh rest, then there would be very little if any of what im doing actually in python.
At that point I would just pick a language that didnt have these issues but still high level and avoid python all together. If i wanted to code in C I would have.