@freemo @trinsec Based on my limited knowledge of python, the multithreading part is pretty heavy, if I recall correctly, you need a new python process to start a new thread (sounds familiar with JVM ). And go is pretty good at multithreading (I mean user-mode threads). If not limited by the IO, I would assume a go implementation will speed up some of the process. Maybe also ease the load on developers, considering go offers some great built-in multithreading structures.
As a JVM lover, I am jealous of the ability to call C code directly from CPython.
And yes, coroutine is powerful. I'm using kotlin coroutines and it's a huge (free) improvement of Java's native thread.
@freemo co-routines can run on multiple threads, where the "tasks" can yield and the thread from a pool can switch to another "task" without suspension or something. At least kotlin coroutine can, and according to sdgathman, Python can do it too. That's the ultimate free boost you can get by just switching to another tech.
But if Python can do that, then my earlier hypothesis about switching to go will give your free boost is wrong
You can call C from any high level language. The difference is the varrier in python is very low, in java it is very high. Moreover in java it is a very expensive operation to cross the barrier from native to jvm unlike in python.
Compared to Java functions, yes, but overall no. According to StackOverflow, each JNI call is several nano seconds more compared to Java function calls. The main overhead is copying data from the Java heap to native memory. But if you use something like native buffer, there should be no such overhead.
If you load a big dataset in Java using a byte array, then you want to pass it to, let's say some BLAS implementation, then good luck copying all those data. But with native buffers, you can just pass the pointer of that buffer to the BLAS implementation and you're good to go.
https://stackoverflow.com/questions/13973035/what-is-the-quantitative-overhead-of-making-a-jni-call
@freemo @sdgathman
JNI is still Java oriented, which requires you to write some glue code to translate the difference between C and JVM world. (but with JNI, you can call Java code in C)
Now I'm using JNA, which is a black magic that automatically generates the proxy to call native shared libraries without any additional C code, with the cost of performance.