@antares No our sidekiq queues are still growing for some reason. working on it.

@sergeant Not quite. Give it 24 hours. Ther eis a big backlog from the downtime so will take some time for everything to catch up. I just scaled everything up to speed it up as much as possible. Should be somewhat usable until the rebuild is done.

@GeorgeMari Yea, we had intended to update to the beta server but accidentally updated to here.

Normally in the future updates shouldnt have any noticable downtime.

@GeorgeMari Yes. I was going to setup a backup account for that. We have one on this server but as you notice we will need an official one on another server. I'll likely set this up through my fedipage code.

@lupyuen Happy to... just be patient with me while i optimize the new environment a bit.

Keep in mind #QOTO will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.

Keep in mind #QOTO will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.

@nibble4bits Didnt even know this was a thing...

@ClaraListensprechen4

Should be good for a hot minute. It only went down because we were trying to test updates in a beta env and accidentally wrote the code to this env. .. All part of the move. The code itself was/is stable as far as we can tell.

Its a big move though so we will keep an eye on it. We totally move infrastructure and we made some big changes that will help us go fully elastic and scale up

@katykaty Philly represent!

oooookkkkkaaaaaayyyyyyy......
QT: https://glammr.us/@katykaty/111972052629957886

Sorry everyone on #QOTO for the extended downtime. It was a massive migration to entirely new infrastructure. It will allow us to apply the 6 updates we have ready that we wrote with minimal future down time. Will also allow us to scale dynamically.

After we check that everything is working ok post-migration we will start applying updates in a beta environment everyone can test before we move updates to mainline. This should prevent all future downtime.

For the record this is the first time we had downtime of more than 5 minutes since the founding of QOTO over 5 years ago. So this is not normal for the server.

@hesgen It is not normal for the fediverse or QOTO. In fact it is the first time we have had downtime of more than 5 minutes since our founding something like 5+ years ago.

I did make multiple posts (over 4) in the weeks leading up to it that the migration was happening when it was and that the downtime would be significant. I also made one post from the QOTO account itself.

This was a very major move to entirely new architecture (from bare metal to a scalable docker swarm like the rest of our services). This move was done to give us easier control over beta testing and updating. This is in preparation of about 6 updates we have to do in short order, all of which should be able to occur with minimal downtime due to the migration to the new infrastructure. It will also allow us to scale.

Please keep in mind while we are backup redis is reindexing so it should be slow for a short while.

@LouisIngenthron At the time yes, web interface is back up and recreating indexes.