@antares No our sidekiq queues are still growing for some reason. working on it.
@sergeant Not quite. Give it 24 hours. Ther eis a big backlog from the downtime so will take some time for everything to catch up. I just scaled everything up to speed it up as much as possible. Should be somewhat usable until the rebuild is done.
@freemo thank you for all your work on this. When the dust settles, would it be possible to have a dedicated account to follow, just for maintenance and downtime updates? (Preferably on a different server, in case qoto.org is down)
@GeorgeMari Yea, we had intended to update to the beta server but accidentally updated to here.
Normally in the future updates shouldnt have any noticable downtime.
@GeorgeMari Yes. I was going to setup a backup account for that. We have one on this server but as you notice we will need an official one on another server. I'll likely set this up through my fedipage code.
#QOTO is back up.
Please keep in mind #QOTO will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.
Give it 24 hours and hopefully things will be back to normal at that time.
@lupyuen Happy to... just be patient with me while i optimize the new environment a bit.
Keep in mind #QOTO will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.
Keep in mind #QOTO will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.
@nibble4bits Didnt even know this was a thing...
Should be good for a hot minute. It only went down because we were trying to test updates in a beta env and accidentally wrote the code to this env. .. All part of the move. The code itself was/is stable as far as we can tell.
Its a big move though so we will keep an eye on it. We totally move infrastructure and we made some big changes that will help us go fully elastic and scale up
@katykaty Philly represent!
oooookkkkkaaaaaayyyyyyy......
QT: https://glammr.us/@katykaty/111972052629957886
@bwbeach Should be mostly fixed now.
Sorry everyone on #QOTO for the extended downtime. It was a massive migration to entirely new infrastructure. It will allow us to apply the 6 updates we have ready that we wrote with minimal future down time. Will also allow us to scale dynamically.
After we check that everything is working ok post-migration we will start applying updates in a beta environment everyone can test before we move updates to mainline. This should prevent all future downtime.
For the record this is the first time we had downtime of more than 5 minutes since the founding of QOTO over 5 years ago. So this is not normal for the server.
We will fix lists now that we see the problem, no worries. The updates should be coming soon, this was a migration to infrastructure that should allow us to update with less downtime now moving forward.
@hesgen It is not normal for the fediverse or QOTO. In fact it is the first time we have had downtime of more than 5 minutes since our founding something like 5+ years ago.
I did make multiple posts (over 4) in the weeks leading up to it that the migration was happening when it was and that the downtime would be significant. I also made one post from the QOTO account itself.
This was a very major move to entirely new architecture (from bare metal to a scalable docker swarm like the rest of our services). This move was done to give us easier control over beta testing and updating. This is in preparation of about 6 updates we have to do in short order, all of which should be able to occur with minimal downtime due to the migration to the new infrastructure. It will also allow us to scale.
Please keep in mind while we are backup redis is reindexing so it should be slow for a short while.
@LouisIngenthron At the time yes, web interface is back up and recreating indexes.
#QOTO is back after a long migration in the first stage of our upgrade. Now that we are on a new home we will give it 24 horus to see if the move was bug free (As it seems to be) and then we will start upgrading versions one at a time day by day.. The upgrades are prepared and ready and should be **much** quicker.
The move took a while as we keep a backlog of every message to the start of the fediverse so our media and DB stores are quite large and it took a long time. But we are back!!!
Jeffrey Phillips Freeman
Innovator & Entrepreneur in Machine Learning, Evolutionary Computing & Big Data. Avid SCUBA diver, Open-source developer, HAM radio operator, astrophotographer, and anything nerdy.
Born and raised in Philadelphia, PA, USA, currently living in Utrecht, Netherlands, USA, and Thailand. Was also living in Israel, but left.
Pronouns: Sir / Mister
(Above pronouns are not intended to mock, i will respect any persons pronouns and only wish pronouns to show respect be used with me as well. These are called neopronouns, see an example of the word "frog" used as a neopronoun here: http://tinyurl.com/44hhej89 )
A proud member of the Penobscot Native American tribe, as well as a Mayflower passenger descendant. I sometimes post about my genealogical history.
My stance on various issues:
Education: Free to PhD, tax paid
Abortion: Protected, tax paid, limited time-frame
Welfare: Yes, no one should starve
UBI: No, use welfare
Racism: is real
Guns: Shall not be infringed
LGBT+/minorities: Support
Pronouns: Will respect
Trump: Moron, evil
Biden: Senile, racist
Police: ACAB
Drugs: Fully legal, no prescriptions needed
GPG/PGP Fingerprint: 8B23 64CD 2403 6DCB 7531 01D0 052D DA8E 0506 CBCE