Lup Yuen Lee 李立源 @lupyuen@qoto.org

… however, in some unfortunate moments, it brings our not-super-reliable storage setup down.

We are sorry that we underestimated the amount of small reads that Git operations can create. We are currently working hard to change our setup to cope with these load spikes.

What doesn't kill us makes us stronger: We have learned a lot, the hard way, but we are excited to build a better platform for tomorrow.

Want to be part of the journey? Consider joining our non-profit via https://join.codeberg.org/

Traffic at Codeberg is to a huge extent automated, and not uniform after all. There are times when nothing is happening. Sometimes, however, large operations occur.

And when a bot starts crawling the largest repositories, our disk traffic is going up like a rocket, and we have a short burst of insane traffic. Much more than we anticipated with the setup we have.

We implemented queues and scheduling to make most operations on Codeberg go smooth during this period, and few users should notice …

Why is Codeberg down? In short, we are running on Ceph, but on a very small setup (which derives from best practices). We chose this option, because we have been too small for a larger setup in the past, and while we have more servers and components ready, the time of our core contributors is very limited.
(If you happen to be in Berlin during September, you might help in getting the next server running - please let us know via the projects repository mentioned above).

Why does it fail, though?

Because we are tired of our current stability issues just like you, we are scheduling some operations during today's Sunday, 30 July 2023.
There will be scheduled downtimes of about 30 minutes each.

We try to limit the length to not interfere with your workflow too much.

We used the last outage to install some outstanding upgrades and fine-tune some database settings which have been identified in a recent contributor meeting.

Today, we will continue working on our storage … read on

@lupyuen Ah, so "$@" is handled specially, I always wondered why the quotes there didn't cause basically the same result as "$*".

@morenonatural @lupyuen fish my beloved