QOTO Announcements & Polls @QOTO@qoto.org

We are going to try to add some healthchecks to our containers to help protect them in case they become unresponsive. No one on #QOTO should notice any real downtime unless i break something. If I do downtime should be mere seconds as I bring it back up.

Just letting people know in case anything goes wrong, shouldnt be noticed if things go right..

after another short down time everything is fixed! #QOTO had to recreate an index that got lost in the migration which was slowing down the DB and everything else. No amount of resources was going to help.

But it is fixed now all ques are empty or very close to it. we will now downgrade the DB to a more sane level now that it is fixed (we upgrades shortly to maintain the system). But it will sill be a pretty hefty system for us, and we can always scale back up when needed.

TL;DR everything is wording fine now.

PS we are now going to work on a staging environment to test upgrades so we can safely start moving the main server through the upgrade cycle. stay tuned.

So we found the real problem haunting us. Turns out we didnt even need the bigger database. There was just an index that got dropped during the migration. We are working now to put it back in place. At which point things should be back up to their normal speed.

I just upgraded the #QOTO DB server to x2 the CPU. This seems to fix the underlying issue. The pull queue (not related to most things) is now recovering as well.

So reviewing everything the next day at #QOTO it seems almost everything is back and working with one exception that wont effect things too visibly.

One of our low priority sidekiq queues, the pull queue is backlogging now. All other queues are staying ahead of the curve.

This largely deals with pulling in remote media so you may see the occasional dead image. It is partly working though. We think the problem is with ulimits and working on it, shouldn't effect main operations and hopefully will be fixed soon. I will keep everyone updated.

It appears now that the #QOTO backlog is resolved pages load quickly and images can be uploaded with little delay.

There seems to be one minor issue behind the scenes I need to tweak but overall it does appear to be working.

If you find any lingering issues please report them to one of the admins.
QT: https://qoto.org/@freemo/111981397864429953

One last update before I go to bed and disapear for 12 hours.

The backlog has went from 1.2 mil at its peak earlier today to 0,4 mil now after we reconfigured things. It is steadily going down and should have everything up to working order before I get up.

One or two people were able to get images loaded after a VERY long wait. So while images still arent working it seems very likely related to the backlog. In a few hours when the backlog clears I expect image uploads should work again. If not I will check what the problem is in the morning.

Other than that most things appear to be working and everything should be functional soon.

The back log is about 2/3rds complete. This afternoon it peaked at 1.2 million and now it is ~0.4 million. I jut moved to pg_bouncer to sped that up a bit. Looks like I more than doubled the process time. Almost there.

Woot #QOTO is now 50% through catching up on the sidekiq backlog!

So the Sidekiq backlog on #QOTO is still progressing. As of a few hours ago at the peak of the problem our backlog was 1.2 million jobs. As of right now its down to 0.7 million jobs. It is steadily decreasing there is just a lot to get through from the downtime. We are expecting everything to be back in working order when its done which should be around end of day. In the meantime things are still usable but you may experience very long lag on some actions.

Images still cant be uploaded, we hope this is the same problem.

As spam comes in we will block the servers on the #Fediverse. Please be patient this is happening across the whole fedi and we are working on better ways to address it.

#QOTO

#QOTO update.

So I went to bed last night and work up to find the sidekiq workers were backlogging and we have 800K backlogged jobs. It was due to a misconfiguration that I have now fixed and it appears the backlog is quickly resolving itself.

If you noticed any weirdness this should be resolved int he next few hours as the backlog clears.

#QOTO is back up.

Please keep in mind #QOTO will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.

Give it 24 hours and hopefully things will be back to normal at that time.

Please note today's server migration was moved to Friday. Expect some downtime on Friday, EST morning. We are migrating servers and begining the upgrade process for several updates we have in the pipeline. Expect brand new features in the near future.

QOTO's Funkwhale instance has now been upgraded to the latest version (1.1.2). Enjoy!

One small issue existing users will need to fix. The avatar's feature is new and existing users will have a broken avatar image shown in the upper left when logged in as a broken image link. To fix go into your settings and delete your avatar (do not upload a new one just yet). Then log out and back in. You should now see that the image is no longer broken but replaced with a generic avatar.

If anyone has any other issues please let @freemo know

#music #qoto #funkwhale #fediverse #decentralized @QOTO

QOTO's instance of Peertube has just been upgraded, we are now on the latest version of v3.2.x. Everything appears to be working but if you have any problems please reach out to @freemo.

Our Peertube instance is open to everyone to use according to our rules, registration is open. However you must contact a qoto administrator to be given upload rights (which we generally grant). To register or use the service the link is as follows.

https://video.qoto.org

#peertube #mastoadmin #qoto #federation #decentralized #youtube #videos #video @QOTO