Woot #QOTO is now 50% through catching up on the sidekiq backlog!
So the Sidekiq backlog on #QOTO is still progressing. As of a few hours ago at the peak of the problem our backlog was 1.2 million jobs. As of right now its down to 0.7 million jobs. It is steadily decreasing there is just a lot to get through from the downtime. We are expecting everything to be back in working order when its done which should be around end of day. In the meantime things are still usable but you may experience very long lag on some actions.
Images still cant be uploaded, we hope this is the same problem.
As spam comes in we will block the servers on the #Fediverse. Please be patient this is happening across the whole fedi and we are working on better ways to address it.
#QOTO update.
So I went to bed last night and work up to find the sidekiq workers were backlogging and we have 800K backlogged jobs. It was due to a misconfiguration that I have now fixed and it appears the backlog is quickly resolving itself.
If you noticed any weirdness this should be resolved int he next few hours as the backlog clears.
#QOTO is back up.
Please keep in mind #QOTO will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.
Give it 24 hours and hopefully things will be back to normal at that time.
QOTO's Funkwhale instance has now been upgraded to the latest version (1.1.2). Enjoy!
One small issue existing users will need to fix. The avatar's feature is new and existing users will have a broken avatar image shown in the upper left when logged in as a broken image link. To fix go into your settings and delete your avatar (do not upload a new one just yet). Then log out and back in. You should now see that the image is no longer broken but replaced with a generic avatar.
If anyone has any other issues please let @freemo know
#music #qoto #funkwhale #fediverse #decentralized @QOTO@groups.qoto.org
QOTO's instance of Peertube has just been upgraded, we are now on the latest version of v3.2.x. Everything appears to be working but if you have any problems please reach out to @freemo.
Our Peertube instance is open to everyone to use according to our rules, registration is open. However you must contact a qoto administrator to be given upload rights (which we generally grant). To register or use the service the link is as follows.
#peertube #mastoadmin #qoto #federation #decentralized #youtube #videos #video @QOTO@groups.qoto.org
I spent some time last night improving the spaces and our room directory.
I've added several rooms from different fediverse associated instances to the #Fediverse_space:qoto.org space.
I have also added about half a dozen room servers related to other instances and servers to our room search. So should be easy for you to find most rooms anywhere in the federated universe.
Again: you can join this space (or the others in the linked post) from any matrix user on any server. If you dont have a user you are free to register one on QOTO at: https://element.qoto.org
So I setup a list of maybe 30 highly active matrix rooms organized into Matrix Spaces on the QOTO Matrix server (though you can access them from any spaces enabled matrix server).
Just login to your existing matrix account on any server, or register for a free one at https://element.qoto.org (registration open to everyone). Then join any of the following spaces (make sure you turn on spaces in your config under labs first).
Please let me know if you have any rooms you'd like to suggest to be added to any of the below spaces. Also, feel free to create your own space and share it.
#QOTO_space:qoto.org - Rooms related to QOTO itself
#Fediverse_space:qoto.org - Various rooms from matrix servers that have a fediverse hosted as part of the same community.
#STEM_space:qoto.org - A collection of general topic STEM rooms like Electronics, Physics, Math, etc.
#Regional_space:qoto.org - Rooms specific to Countries, towns or regions around the world.
#Projects_space:qoto.org - Channels for various open-source projects
#linux-distros_space:qoto.org - Rooms for various linux distribtuion flavors.
Matrix servers that have it enabled (including QOTO's matrix server) allow for ephimerial rooms. That is, you can set your messages in a room to automatically delete after a certain point in time. Anyone who is the admin/mod of a room can set it. That, combined with the end-to-end encryption and identity verification really makes this a win on privacy.
Please let me know if anyone needs help.
If you want to sign up or log in you can go to: https://element.qoto.org
Everyone is welcome to signup, you dont need to have a QOTO account. You can also login as a guest if you wish.
To anyone out there hosting a #Matrix server of any scale, interested in connecting with the QOTO matrix server?
What I am proposing is I will add any Matrix server on the fediverse into our configuration under room directories on the condition the other server adds QOTO as well. This wont change anything functionally except to allow the server to show up int he drop down for matrix servers when searching for servers to join. It just makes sense to have the various fediverse servers show up in that list by default.
We have now added a new service to supplement the QOTO Matrix chat server; a dedicated Element web client with a ton of features baked in the standard Element web client doesnt have.
Just go to https://element.qoto.org and you can register for, or sign into, a QOTO matrix account.
Please reach out to @freemo If you have any questions or problems.
#QOTO @QOTO@groups.qoto.org #mastoadmin #matrix #chat #federation #decentralized.
We are happy to announce a new QOTO service, our brand new matrix server!
Registration is open to everyone.
To either login or register a new account use any compatible Matrix/Riot client (like https://app.element.io/ ). When doing so you will have to set the home server to "matrix.qoto.org" (it is usually set to matrix.org). However keep in mind once logged in your username will be @user:qoto.org, as the matrix.* prefix is dropped. Similarly any rooms you create will be #room:qoto.org.
The best part is this service is decentralized and federated so you can join any matrix room on any other server. Even cooler you can join rooms on servers that arent federated at all like IRC or gitter.im. So you really can connect to almost any chat server out there all from one identity and server.
**NOTE**: We will also be bringing up a client, likely the same client run at app.element.io, for the added security. It wont change anything and you can switch clients once we have that up if you want. You can also download the client and use it as a desktop app.
#matrix #QOTO @QOTO@groups.qoto.org #Chat #ChatRooms #Mastoadmin
WOOT! Our GitLab instance just got approval for another year of free ultimate license (normally costs 100$ per person, per month).
As for costs... the last year the total cost to run QOTO was **11,384.43$**. However I recently paid to reserve the instances instead of spot instances. So I expect the next year to be about 25% cheaper.
Attached a picture of the QOTO swarm's resources for those who are curious.
Some interesting facts about the #QOTO cluster for those who are curious:
* Runs as a docker swarm, so easily scales as needed and services can move between computers in the cluster.
* Runs on a 7 computer cluster.
* Utilizes only about 5% at most of its CPU potential
* CPU has burst capability, so can go well beyond 100% for momentary high loads.
*Utilizes 33% of the memory (this includes caching). Almost all of the memory is from Gitlab.
* Requires half a terrabyte of on-disk size, with an additional 2 TB for used currently as shared disks.
* Supports 12 different services right now: fediverse-bots, discourse, wikijs, funkwhale, gitlab, 5-node load balancer, nextcloud, peertube, piwig, qoto-mastodon groups server, and two different monitoring tools.
* These 12 services run as 60 different interconnected docker containers spread across the cluster.
We are currently upgrading all of the #QOTO services. Our configuration allows for no-downtime upgrades, so most users shouldnt notice anything (except for their services suddenly upgrading). However be aware downtime is possible if things dont go well.
So far all database and Redis back-end services have been upgraded to the latest. Next we will be working on the front-end services. This may be over the course of several days. We will announce when upgrades are complete but in the meantime if anything goes down, don't panic, we will be back.
I have recently uncovered a user who seems to be opening temporary accounts at QOTO, blocking all the moderators so we dont see their posts and then posting high offensive material (nazi imagery, transphobic and hateful derogatory content, etc) and then proceeding to announce to block list moderators and possibly on the #Fediblock list in general in an attempt to get us banned.
The issue of users being able to block moderators and do **exactly** this is something I have brought up to @Gargron before probably over a year ago but he has never acted on fixing this issue and does not see it as needing fixing from what I could gather (he can interject here if he would like).
It is something we as moderators are powerless to detect or act against unless someone actually reports the content first, that is the only way we will even see it. Thankfully none of the block lists have actually acted on this and no damage to QOTO has been done. But seeing as this may continue I want to make others aware of the problem, press Gargron to fix the issue once more, and also encourage **anyone** who sees such content on QOTO to please report it so we can act on it.
I can not personally confirm this but according to the block-list operator they seem to have reason to think it is @snow behind the attacks. This would be in line with the behaviors from snow I have witnessed in the past.
For the attempt by snow to spam and falsely accuse instances see here:
https://likeable.space/notice/A21ux0IbhoDJIkNxzc
For more information see these attempted block requests on QOTO that were made in this fashion:
https://schlomp.space/FediBlock/data/pulls/255/files
https://schlomp.space/FediBlock/data/pulls/248
(Note: originally posted from @freemo account. Repasted here to show up in QOTO announcements as well)
We only posts announcements here and occasionally use polls to help us make decisions that effect our user base.
We don't usually respond quickly to direct messages to this account. If you need help with anything related to the QOTO servers, including moderation, then please contact one of our administrators. They are listed out about page: