One last update before I go to bed and disapear for 12 hours.
The backlog has went from 1.2 mil at its peak earlier today to 0,4 mil now after we reconfigured things. It is steadily going down and should have everything up to working order before I get up.
One or two people were able to get images loaded after a VERY long wait. So while images still arent working it seems very likely related to the backlog. In a few hours when the backlog clears I expect image uploads should work again. If not I will check what the problem is in the morning.
Other than that most things appear to be working and everything should be functional soon.
QOTO just did a very major migration and our systems were backlogged. You likely just seed a delayed action.
@TheOldGuy I checked and trump apparently couldnt even crack a million in individual contributions meanwhile biden is at 34 million. Interesting.
https://ballotpedia.org/Presidential_election_campaign_finance,_2024
@bibliolater Yup this isnt a bug so much as just backlog from the downtime. At our peak backlog a few hours ago we were at 1.2 million. Now we re down to 500K. After a few more hours everything should be back to normal assuming there are no unknown bugs.
@volkris I totally agree, though of course id say the dems are following suit, so hardly unique to republicans.. but yea.
@drrjv If this didnt come during a time where Biden is committing Arab genocide I would be all for it.. but this is just so tone deaf considering the number of small children that man is murdering right now.
So the Sidekiq backlog on #QOTO is still progressing. As of a few hours ago at the peak of the problem our backlog was 1.2 million jobs. As of right now its down to 0.7 million jobs. It is steadily decreasing there is just a lot to get through from the downtime. We are expecting everything to be back in working order when its done which should be around end of day. In the meantime things are still usable but you may experience very long lag on some actions.
Images still cant be uploaded, we hope this is the same problem.
File upload isnt working. I suspect its just because the sidekiq queue needs to catch up (as I see media being processed). After the sidkiq queue catches up we will reassess if that or anything else is truely broken.
@olives Long threads, or toots? There is, by default long toots are hidden by a read more.
The exception is when its CW'ed and you have it auto un-CWed
Yup, things are fixed more or less just restoring. We will be doing theupdates in a beta environment first moving forward. The update got triggered accidentally in production which set us back a bit.
@trinsec Exactly, quote toots respect privacy and settings, while other means like screen shots and copy and paste do not.
@sergeant That was earlier during migration. The web browser is up now.
Sidekiq queue is slowly catching up, should be back to normal in a few hours.
It boggles my mind how people will hate more vigorously for someones opinions on how governments should be run or obscure ethical opinions, and yet a persons actual actions, and how they treat others, means almost nothing.
Like you can be the worst person in the world as long as your doing it to someone who has some obscure opinion you disagree with... like how are virtual ethics more important than actual ethics a person acts on.
Since that last tweak our Queue has went from 1.2 million backlog down to 0.9 million and continues to decrease.
We are also setting up a mechanism that should allow me to speed up the process further soon, but either way for now we are still on track to be back to caught up soon. At that point hopefully all functionality will be restored (I found a few issues and fixed them). If not we can start addressing any remaining problems at that time.
Jeffrey Phillips Freeman
Innovator & Entrepreneur in Machine Learning, Evolutionary Computing & Big Data. Avid SCUBA diver, Open-source developer, HAM radio operator, astrophotographer, and anything nerdy.
Born and raised in Philadelphia, PA, USA, currently living in Utrecht, Netherlands, USA, and Thailand. Was also living in Israel, but left.
Pronouns: Sir / Mister
(Above pronouns are not intended to mock, i will respect any persons pronouns and only wish pronouns to show respect be used with me as well. These are called neopronouns, see an example of the word "frog" used as a neopronoun here: http://tinyurl.com/44hhej89 )
A proud member of the Penobscot Native American tribe, as well as a Mayflower passenger descendant. I sometimes post about my genealogical history.
My stance on various issues:
Education: Free to PhD, tax paid
Abortion: Protected, tax paid, limited time-frame
Welfare: Yes, no one should starve
UBI: No, use welfare
Racism: is real
Guns: Shall not be infringed
LGBT+/minorities: Support
Pronouns: Will respect
Trump: Moron, evil
Biden: Senile, racist
Police: ACAB
Drugs: Fully legal, no prescriptions needed
GPG/PGP Fingerprint: 8B23 64CD 2403 6DCB 7531 01D0 052D DA8E 0506 CBCE