@helgek @darnell @volkris @nickapos @jan that is purely technical.
For now we are stuck with Mastodon which is a Rails app, which comes with all sorts of performance characteristics. The larger alternatives (elixir+pg, php+pg) rather similar.
Performance and scaling are stuck there now. Regardless of the amount of people or amount of followers, the limits are imposed by the tech stack , today.
So that's where work is needed.
@berkes @helgek @darnell @volkris @nickapos Rails is not the main reason for the scaling issues. Mainly background jobs where Ruby is just a wrapper for transcoding libs etc. You can scale Puma and Sidekiq horizontal by adding more compute/workers. Same for storage. Meaning its not the language or the framework, its the protocol and overall architecture which leads to more resource consumption.
@jan @berkes @helgek @darnell @volkris @nickapos I also don’t think it can be so much an issue with raw compute power. That only affects how the server performs within it’s own workloads, not between computers at all.
Network IO takes magnitudes longer than compute. But while doing it, the thread often waits most of the time, doing very little compute.
Maybe a layer of distributed caches? Delivery is the key.
@gimulnautti @jan @berkes @helgek @darnell @nickapos
Remember, this is not theoretical. This is actual experience hearing from real people running instances and finding themselves having to unexpectedly shell out more money for higher hosting prices that they weren't expecting.
The protocol requires poorly scaling processing and bandwidth.
And that's not even getting into expensive design decisions that Mastodon in particular put on top of everything else. For example the intentional decision not to redistribute image previews but instead require each instance to go out and pull its own image preview, duplicating that effort throughout the whole platform.
It seriously sounds like [almost] nobody involved in this from protocol design up through platform implementation gives a second thought to what's going to happen at scale.
And I may have said it in this thread, but when I was in school for computer science we were hammered with big-O analysis of algorithmic scaling but someone recently told me that's not emphasized in school these days. It sure looks like that's the case.
@volkris @gimulnautti @jan @helgek @darnell @nickapos i am one of those that (help) run a large insatnce.
And no. Bandwidth isn't an issue, nor is the amount of requests.
I've built and provisioned backends that handle way more requests an ingress/egress that this large mastodon instance, on way smaller and cheaper infra.
In Mastodon the only and most costly piece is the async workers. Which is this costly. 1/2
@volkris @gimulnautti @jan @helgek @darnell @nickapos because rails is too slow to handle most stuff inline.
Ruby is slow, but most of all, the architecture (AR, MVC) isn't one optimized for scale. Typically with rails, "scaling" means just throwing more resources at it. Costly. With the typical SaaS usecase for Rails, not a big problem. But a real issue on the Fediverse.
The protocol isn't the problem. Yet.
@berkes @volkris @gimulnautti @jan @helgek @darnell @nickapos I am shocked how big an instance I have had to provision. Disk space management and aggressive caching doesn’t help. Far too much seems to be preemptively cached.
Back over 20 years ago e-mail delivery for 1000s of clients with loads of network I/O over shitty links was more efficient. On hardware less powerful than your phone.
@snookerarmchair 's comment is exactly kind of thing I'm referring to when I say I see people surprised by the amount of resources it takes to run a #Mastodon / #ActivityPub instance.
This thread highlights comparing against different levels of expectation.
Someone starting from a place where they expect it to take those resources doesn't change that someone else is surprised that it does.
Personally, I suspect we set the bar too low and waste resources because we tolerate systems that don't operate efficiently.
And thus we get ActivityPub, which seems to live down to those expectations.
And again, you're basing your statements against a particular standard that others don't necessarily hold.
In your case ActivityPub isn't causing any problems *given the resources you're already devoting to it* while other people who aren't devoting those resources are having problems.
Other people who haven't expected the platform to be so bandwidth intensive are finding themselves surprised by the amount of bandwidth it takes. Your expectations are different and you're not surprised. That just speaks to expectations, not to the platform.
@volkris who is surprised by the BW used? Honestly, never encountered that limit in all the servers I helped run.
All cloud providers, or even rack space providers give some "free" bandwidth with their servers. And I'd be very surprised to find anyone reaching that free limit before running into the CPU, mem and storage limits. And when one ups that, the amt of "free" data increases.
When or in what case is it a bottleneck?
I've heard probably half a dozen admins complaining about it in the last month or so, sharing graphs of their bandwidth, and wondering if something is going wrong.
Storage too.
@volkris have you ran a gotosocial or even a pleroma instance?
Again, and I'm repeating myself, the amount of requests nor the bandwidth imposed by Activitypub are any kind og problem yet. It really only is Rails, underlying mastodon.
I urge you to provision a gotosocial and experience how much a tiny VPS can handle. How much toots it can move on under €5/month infra.