🇳🇱
#QOTO is back after a long migration in the first stage of our upgrade. Now that we are on a new home we will give it 24 horus to see if the move was bug free (As it seems to be) and then we will start upgrading versions one at a time day by day.. The upgrades are prepared and ready and should be **much** quicker.
The move took a while as we keep a backlog of every message to the start of the fediverse so our media and DB stores are quite large and it took a long time. But we are back!!!
@freemo Thanks for all the work over the long weekend. Much appreciated.
@freemo My Home feed is not working. It's showing this toot and nothing else. Local and Federated look fine.
@bwbeach Should be mostly fixed now.
@freemo Thanks for all your hard work on this!
@freemo If 24 Horus doesn't work, give it five or six Amun-Ra
@freemo Congratulations on leaping to the other side of the chasm! I look forward to exploring new features.
@freemo FWIW, my lists are now devoid of posts.
@peterdrake @freemo Likewise. I'm missing Home and List content.
@LouisIngenthron @freemo Home seems to be working for me, just not lists.
We will fix lists now that we see the problem, no worries. The updates should be coming soon, this was a migration to infrastructure that should allow us to update with less downtime now moving forward.
@freemo @LouisIngenthron Another issue: I can't upload images. Using the web interface, the "uploading" bar fills, but then hangs.
Yea I know that issue. It is either due to the backlog in in sidekiq or a slightly unrelated issue I may have just fixed. But im actively working on it. Should have most of the bugs fixed soon. Me and one other guy are on it 100% right now.
@LouisIngenthron @peterdrake @freemo add me to the group with no home or list content. Everything else I've checked seems ok: local, federated timelines, profile, bookmarks
@seedyh @LouisIngenthron @peterdrake @freemo
Same thing is happening for me. My home timeline has exactly two posts, both from ~1day ago.
@seedyh @LouisIngenthron @peterdrake @freemo
Also, my own post does not appear in my home timeline (or rather, appears but then disappears upon refresh). That suggests something is still broken, and that we're not dealing just with an aftereffect of something.
@seedyh @LouisIngenthron @peterdrake @freemo
Ah, this seems to be a terribly large backlog. I'm starting to see more posts, delayed by a day or so.
Sidekiq is backlogged, things being slow is normal as a result of that. We are trying to address the back log which itself is caused by a slight misconfiguration (it is related to inability to upload images)
We are actively working on it. Two problems we have seen that we are fixing.
@freemo Welcome back.... Missed you.
@freemo Thank you so much! Really appreciate your hard work :-)
@lupyuen Happy to... just be patient with me while i optimize the new environment a bit.
@freemo Thanks for your job you are working for us.
@freemo 👍 Great job, but... a black screen instead of the main interface. I checked in three browsers, even Tor, and it's the same. The settings page works, though.
@johan @freemo I had blank screen too about a day ago... but now showing as before... and testing it out...
❓ Tried to upload a small audio just now and it got stuck at 100% (or near) and didn't allow toot as still waiting for upload... so testing that out today and trying this post also as a test :)
@freeschool @freemo
> I had blank screen too about a day ago... but now showing as before
Yes, the server has been rolled back to version 3.2.1.
Yup, things are fixed more or less just restoring. We will be doing theupdates in a beta environment first moving forward. The update got triggered accidentally in production which set us back a bit.
@freemo looks like web interface is dead. Errors in Dev console about blocking loading of script-src. CSP may need adjustment?
@freemo also JavaScript exception in federation_dropdown.js
@sergeant That was earlier during migration. The web browser is up now.
Sidekiq queue is slowly catching up, should be back to normal in a few hours.
@freemo So much data. So much data. SO MUCH DATA.
@freemo I never doubted you could do it. 🙃
Now about the "24 horus". I know Horus the Falcon, Horus the Elder, Horus the Younger, Horus the Child and Horus of Behdebt.
That's only 5. The question is: Who are the other 19? 😉
Insidious question: Did you try to make an upgrade while moving to a new server? 
@mc Oh and to answer your question.. we accidentally did a version bump and had to revert as it was supposed to happen in our new beta environment.
@freemo Thanks for all your work!