Well I think I finally got to the bottom of why the matrix server has been slow in the past. Now that i have good stats on resources I can better address it... Should have matrix running much smoother soon.

@trinsec

@freemo @trinsec I do hear the matrix developers say there will be a go implementation to replace the slow and resource demanding python version. Is that still in alpha stage?

(As a jealous JVM lover, I don't trust Python 😂 )

@skyblond

That I have no idea.. But if they think go automatically means faster then it will probably be an abysmal failure.

@trinsec

@freemo @trinsec Based on my limited knowledge of python, the multithreading part is pretty heavy, if I recall correctly, you need a new python process to start a new thread (sounds familiar with JVM :ablobthinking: ). And go is pretty good at multithreading (I mean user-mode threads). If not limited by the IO, I would assume a go implementation will speed up some of the process. Maybe also ease the load on developers, considering go offers some great built-in multithreading structures.

C-Python can link with and interact with multi-threaded C code no problem. See https://github.com/sdgathman/pymilter/blob/master/miltermodule.c for an example of the C code end (key object is PyThreadState).

C-Python code itself does not multi-thread, except through multi-processing. (Jython is a Java python implementation that does multi-thread.)

HOWEVER, what is ultimately more efficient than multi-threading for pure python or most other interpreted code is co-routines. Remember those from Knuth's "Fundamental Algorithms"? Python has built-in support for very clean and efficient co-routines - key syntax is the "yield" statement. They work very well ad-hoc, and there are frameworks like the "twisted" library that provide consistent interaction between many parts.

The key limitation of co-routines is that other "threads" do not have a chance to run until the current code hits a "yield". This also means you don't need to bother with locks and stuff like with true multi-tasking, hence the efficiency for an interpreted language.

Another name for this is "cooperative multi-threading". It is harder to do in a language like C with linear stacks. Python stack frames are linked lists - and the resulting conceptual tangle of stack frames when hundreds of complex co-routine tasks cooperate gave rise to the name of the "twisted" library.

@sdgathman

As a JVM lover, I am jealous of the ability to call C code directly from CPython.

And yes, coroutine is powerful. I'm using kotlin coroutines and it's a huge (free) improvement of Java's native thread.

@freemo co-routines can run on multiple threads, where the "tasks" can yield and the thread from a pool can switch to another "task" without suspension or something. At least kotlin coroutine can, and according to sdgathman, Python can do it too. That's the ultimate free boost you can get by just switching to another tech.

But if Python can do that, then my earlier hypothesis about switching to go will give your free boost is wrong

@skyblond

Agreed, while python's biggest shame-to-fame is its inability to leverage multiple cpus through multithreading (or coroutines).. the flip side is that it is so trivial to interact with C-code that it makes up for that in a unique way that has its own value.

IF you need a language where you need to do a lot in C but what the convience of high-level language where you can get away with it, python is great... if you need a high level language that is cpu-intensive and effecient and easy to write without wanting to touch C, then python is a horrible choice.

Me personally, I use python a lot (just finished a 2-year project in python)... but over the years have found it just isnt a suitable language for most things, because of these very reasons.

@sdgathman

Follow

@freemo

It will be great if Python can be ported to other platforms. The flexibility of Python is great for exploring things like new network structures, etc. But once you have decided most of the specifications, it's better to use something like C or Java to build a more strong code base (so a typo won't screw you up, LOL)

And I have to say, I'm really jealous of Python on machine learning stuff, where JVM is (almost) completely being ignored until people need a more robust way to develop and only find out the Java is a complete nightmare to do operator overloading and have to write something like "a = a.mul(b)"

@sdgathman

@skyblond

Python is a nice little language for somethings... The more i use it the more I personally dislike it.. the lack of multi-cpu is a deal breaker for sure.. but i also just fine the language a bit ugly and hackish in its presentation.. which i could have gotten past if it wasnt for the cpu issue.

For high level stuff I much prefer java if i need something rigerous and formal, or ruby if i need something fast and loose... of course thats for traditional OO high level stuff.. For other categories stuff like Haskell is a lot of fun too.

@sdgathman

Sign in to participate in the conversation
Qoto Mastodon

QOTO: Question Others to Teach Ourselves
An inclusive, Academic Freedom, instance
All cultures welcome.
Hate speech and harassment strictly forbidden.