Well I think I finally got to the bottom of why the matrix server has been slow in the past. Now that i have good stats on resources I can better address it... Should have matrix running much smoother soon.

@trinsec

@freemo @trinsec I do hear the matrix developers say there will be a go implementation to replace the slow and resource demanding python version. Is that still in alpha stage?

(As a jealous JVM lover, I don't trust Python 😂 )

@skyblond

That I have no idea.. But if they think go automatically means faster then it will probably be an abysmal failure.

@trinsec

@freemo @trinsec Based on my limited knowledge of python, the multithreading part is pretty heavy, if I recall correctly, you need a new python process to start a new thread (sounds familiar with JVM :ablobthinking: ). And go is pretty good at multithreading (I mean user-mode threads). If not limited by the IO, I would assume a go implementation will speed up some of the process. Maybe also ease the load on developers, considering go offers some great built-in multithreading structures.

C-Python can link with and interact with multi-threaded C code no problem. See https://github.com/sdgathman/pymilter/blob/master/miltermodule.c for an example of the C code end (key object is PyThreadState).

C-Python code itself does not multi-thread, except through multi-processing. (Jython is a Java python implementation that does multi-thread.)

HOWEVER, what is ultimately more efficient than multi-threading for pure python or most other interpreted code is co-routines. Remember those from Knuth's "Fundamental Algorithms"? Python has built-in support for very clean and efficient co-routines - key syntax is the "yield" statement. They work very well ad-hoc, and there are frameworks like the "twisted" library that provide consistent interaction between many parts.

The key limitation of co-routines is that other "threads" do not have a chance to run until the current code hits a "yield". This also means you don't need to bother with locks and stuff like with true multi-tasking, hence the efficiency for an interpreted language.

Another name for this is "cooperative multi-threading". It is harder to do in a language like C with linear stacks. Python stack frames are linked lists - and the resulting conceptual tangle of stack frames when hundreds of complex co-routine tasks cooperate gave rise to the name of the "twisted" library.
Follow

@sdgathman

> The key limitation of co-routines is that other "threads" do not have a chance to run until the current code hits a "yield". This also means you don't need to bother with locks and stuff like with true multi-tasking, hence the efficiency for an interpreted language.

The whole point of multithreading in this discussion is its ability to leverage mutlicpus like it does in other languages... This sounds like they are sharing one thread since only one is running at a time.

@skyblond @trinsec

Yes, for CPU bound tasks, it does not help. But synapse is largely IO bound between network API and database calls. The apache/nginx HTTPS processing and the PostgreSQL database all run in different hardware threads. This normally occupies 2 cpus. With more cpus, synapse lets you split the co-routines into "workers" in 2 or more processes.

The Go implementation (Dendrite) involves rethinking a lot of the design - it is not "GO IS FASTER!!!" Among other things, it aims to support (optional) fully decentralized (single user) operation that does not rely on DNS (DNS has been effectively centralized through ICANN since 1996 or so). The monolithic binaries produced by GoLang are easier for end-users to deploy in that situation.

OT: TLS has also been centralized via the shadowy "TLS cabal" that decides which CAs are "trusted" in popular browsers. The way to fight this is browsers that limit trust in a CA. E.g. "trust this CA for domains ending in .roger.org only". Currently, trust is all or nothing.

The safest way for power users to play with this is a browser extension that can "veto" trust in a CA after examining the Cert. That way, worst case fail in case of bugs is rejecting a CA that should have been trusted, or trusting one the browser would have trusted by default.
Sign in to participate in the conversation
Qoto Mastodon

QOTO: Question Others to Teach Ourselves
An inclusive, Academic Freedom, instance
All cultures welcome.
Hate speech and harassment strictly forbidden.