As I’m sure y’all have noticed, the server has been having stability issues the past week. Every once in a while, the server seems to “crash”; it will time out on all network requests for about 5 minutes, happening several times within an hour. Additionally, there have been several times over the past week when the server queues got backed up, resulting in incoming and outgoing federation traffic being delayed.
At first, I assumed the server hardware had simply become insufficient to handle all of the network and federation traffic we have been receiving, potentially due to a DDOS attack. I was honestly starting to think I would need to upgrade the server hardware, which is ridiculous, as the VPS is already quite powerful and is certainly expensive enough.
Luckily, @green_copper helped with debugging and noticed the server timeouts only occur when the microblog section is viewed (and sometimes when the combined section is viewed). So rather than a lack of hardware being the issue, I believe it actually has to do with the recent Mbin upgrade. My guess is that an SQL query was changed in one of the commits between Mbin v1.9.1 and v1.10.0, which resulted in a widely inefficient query, causing the server to freeze up and time out on requests.
I have and will be investigating the issue this week. I will start by analyzing which database queries are causing the timeouts, but if needed, roll back to Mbin v1.9.1 and try to figure out which specific Mbin commit introduced the bug. Additionally, I may have access restricted only to signed-in users while I’m working on this.
In the meantime, if you guys can refrain from viewing microblogs (or combined view), that would keep the server accessible for everyone and help me work on it faster, as I can’t do anything while the server is frozen. If you want any realtime updates on the server issue (or anything else related to kbin.earth), feel free to join the Matrix room.
Thanks for sticking around and for your patience!
Thanks for sticking around and for your patience!
And thank you for dedicating your time and money providing us with a nice corner of the internet ^^
Absolutely. We do appreciate you.
You rock @jwr1@kbin.earth
Good luck! If it’s truly a SQL issue, it seems you’ve got your work cut out for you. There’s some QB query builders, conditional string concatenation, big preformatted blocks of nested subqueries, the works. Will likely be hard to test.
I do wonder if it has to do with the move to cursors for pagination, there’s quite a number of changes related to that.
https://github.com/MbinOrg/mbin/pull/1985 https://github.com/MbinOrg/mbin/pull/2041
Oh, perhaps the included monitoring system will help?
While attempting to find where to change my default home page view off of the combined view, I mis-clicked and pressed the home button and then the server crashed. So I apologise for being a moron. Thank you for all your work with the server
Went in and switched my default view off combined. Good luck! This sounds like a complicated issue, but I hope it actually turns out to be simple
The prime24/nordpublications jackasses have switched tactics and are creating accounts still without posting their vacuous rubbish. Watch new accounts, in not saying it’s them but they’re highly suspect, imo.
@BentiGorlich@gehirneimer.de, maybe related to the RSS and php-fpm process situations at The/Brain/Bin?
that was actually mostly a pg-bouncer. I have no idea why it is happening though…
I just have a script that checks for this line and restarts php-fpm if necessary:
#!/bin/bash lineCount=`tail -n 1 /var/log/postgresql/pgbouncer.log | grep query_wait_timeout | wc -l` if [ $lineCount -gt 0 ]; then systemctl restart php8.3-fpm.service logger "Restarting php because pgbouncer has query_wait_timeout messages" else echo "No" fi
Just wondering if you’ve considered Anubis rather than cloudflare?
I have considered it in the past, but there are a number of reasons I am still using Cloudflare.
https://nerds.xyz/2026/07/cloudflare-precursor-bot-detection/
That may sound preferable to clicking every square containing a traffic light, but it also means Cloudflare is gathering a much broader picture of how visitors behave on a website.
Alright.
Noticed the banner isn’t up anymore. Are microblogs usable again or should we still refrain until you post an update?
Yup, go for it! The issue should be resolved for now.
Awesome, thanks!
