Show newer

WOOT! the sidekiq backlog is now at 0.6 million... so its 50% of the way through from its peak! finally I can get some sleep tonight.

So the Sidekiq backlog on is still progressing. As of a few hours ago at the peak of the problem our backlog was 1.2 million jobs. As of right now its down to 0.7 million jobs. It is steadily decreasing there is just a lot to get through from the downtime. We are expecting everything to be back in working order when its done which should be around end of day. In the meantime things are still usable but you may experience very long lag on some actions.

Images still cant be uploaded, we hope this is the same problem.

It boggles my mind how people will hate more vigorously for someones opinions on how governments should be run or obscure ethical opinions, and yet a persons actual actions, and how they treat others, means almost nothing.

Like you can be the worst person in the world as long as your doing it to someone who has some obscure opinion you disagree with... like how are virtual ethics more important than actual ethics a person acts on.

Since that last tweak our Queue has went from 1.2 million backlog down to 0.9 million and continues to decrease.

We are also setting up a mechanism that should allow me to speed up the process further soon, but either way for now we are still on track to be back to caught up soon. At that point hopefully all functionality will be restored (I found a few issues and fixed them). If not we can start addressing any remaining problems at that time.

So the wqueue is going down.. its just a lot of old errors retrying. Gonna try pg_bouncer to up the connection.

@freemo I never doubted you could do it. 🙃

Now about the "24 horus". I know Horus the Falcon, Horus the Elder, Horus the Younger, Horus the Child and Horus of Behdebt.
That's only 5. The question is: Who are the other 19? 😉

Insidious question: Did you try to make an upgrade while moving to a new server? :ablobblastoff:

Had a lot of things wanting to say, but I ultimately forgot them when I couldn't spill them out during the maintenance of qoto.

I still don't think I'm a mastodon user several days ago, until I noticed qoto is down and I have nowhere to post my words. Rethink a bit, and I realized that qoto has become my major platform to spill out words. And I'm feeling comfortable with that. What a luxury.

(I just remembered I still have no update for my blog for this month. And the clock is ticking)

As spam comes in we will block the servers on the . Please be patient this is happening across the whole fedi and we are working on better ways to address it.

update.

So I went to bed last night and work up to find the sidekiq workers were backlogging and we have 800K backlogged jobs. It was due to a misconfiguration that I have now fixed and it appears the backlog is quickly resolving itself.

If you noticed any weirdness this should be resolved int he next few hours as the backlog clears.

@freemo thank you for all your work on this. When the dust settles, would it be possible to have a dedicated account to follow, just for maintenance and downtime updates? (Preferably on a different server, in case qoto.org is down)

is back up.

Please keep in mind will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.

Give it 24 hours and hopefully things will be back to normal at that time.

Keep in mind will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.

Keep in mind will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.

oooookkkkkaaaaaayyyyyyy......

Katy Rawdon / Katy James  
My kid, a computer science major, said that the way programming is taught is "too capitalist" because it emphasizes efficiency and time saving over...

Sorry everyone on for the extended downtime. It was a massive migration to entirely new infrastructure. It will allow us to apply the 6 updates we have ready that we wrote with minimal future down time. Will also allow us to scale dynamically.

After we check that everything is working ok post-migration we will start applying updates in a beta environment everyone can test before we move updates to mainline. This should prevent all future downtime.

For the record this is the first time we had downtime of more than 5 minutes since the founding of QOTO over 5 years ago. So this is not normal for the server.

is back after a long migration in the first stage of our upgrade. Now that we are on a new home we will give it 24 horus to see if the move was bug free (As it seems to be) and then we will start upgrading versions one at a time day by day.. The upgrades are prepared and ready and should be **much** quicker.

The move took a while as we keep a backlog of every message to the start of the fediverse so our media and DB stores are quite large and it took a long time. But we are back!!!

is back after a long migration in the first stage of our upgrade. Now that we are on a new home we will give it 24 horus to see if the move was bug free (As it seems to be) and then we will start upgrading versions one at a time day by day.. The upgrades are prepared and ready and should be **much** quicker.

The move took a while as we keep a backlog of every message to the start of the fediverse so our media and DB stores are quite large and it took a long time. But we are back!!!

@adrianhon

I think if a problem can be solved by burdening the individual, rather than burdening eveyone collectively, that is a good thing not a bad thing. This way an individual can choose their level of desired safety and the cost/effort associated. One size fits all solutions are rarely good choices.

That said, I think this depends largely on what we mean here. Certainly some solutions make sense on society and some on individuals and there is no blanket statement that works there. So th real question is, what solution are you purposing at a societal level?

I think some reasonable things, of course, are many of the laws we already have in place. Where there may be lacking is actual enforcement and investigation of such matters, and I'd agree that should be done well to help protect citizens against fraud that is blatant and provable (the person lied, knew they were lying and intentionally lied)

Show older
Qoto Mastodon

QOTO: Question Others to Teach Ourselves
An inclusive, Academic Freedom, instance
All cultures welcome.
Hate speech and harassment strictly forbidden.