Nice post, I enjoyed the storytelling. Glad it’s all sorted now 😁
Btw, regarding this point:
All in all, this has been a fairly frustrating experience and I can’t imagine anyone who’s not doing IT Infrastructure as their day job being able to solve this. As helpful as the other lemmy admins were, they were relying a lot on me knowing my shit around Linux, networking, docker and postgresql at the same time. I had to do extended DB analysis, fork repositories, compile docker containers from scratch and deploy them ad-hoc etc. Someone who just wants to host a lemmy server would give up way earlier than this.
I think you’re totally right, but at the same time, I think the collaborative troubleshooting that happened on Matrix (and has happened many times in the past for other issues) is pretty healthy, and not something that is always possible for other open source software.
people interested in hosting their own instance is probably already interested in linux, or already using it, i don’t think it’s that bad
This reinforced in my mind that as much as I like the idea of lemmy (or any of the other threadiverse SW), this is only something experts should try hosting. Sadly, this will lead to more centralization of the lemmy community to few big servers instead of many small ones, but given the nature of problems one can encounter and the lack of support to fix them if they’re not experts, I don’t see an option.
This also gave me an insight about how the federation of lemmy will eventually break when a single server (say, lemmy.world) grows big enough to start overwhelming even servers who are not badly setup like mine was.
Lemmy has many scalability problems to solve, and not all of these problems are slow database queries. I believe your experience is going to become increasingly common as the community grows because that increased centralization will compound the scalability problems and continue to drive up the technical know-how required to host a successful instance. The software eventually needs to do more to detect and present operational problems to administrators in a friendly way. I2P is an example of a distributed network that’s quite good at reporting issues with the node.
With that said, not everything is doom and gloom. The community has proven itself highly resilient and smart people like yourself are finding solutions. It’s going to be tough road ahead.
Thank you for your hard work.
Thanks for the write up!
Very interesting read, thank you!
I (self)host a lot of stuff as well as developing and deploying some of my software via docker containers and dabbled in Full-Stack territory quite a few times.
Exposing stuff to the internet still scares the shit out of me. Debugging sucks. There’s so much that can go wrong, every layer multiplicates the possibilities of stuff that can wrong or behave in a way not expected. Your journey describes the pain of debugging perfectly. Yeah, in hindsight, it’s often something that probably should have been checked first. But that’s hindsight for you.
And that’s not even accounting for staying ahead of the game while securing your 24/7 publicly accessible service, running on ever-changing software, with infrastructural requirements you basically have no control over. In your spare time.
Hosting something for yourself can be a lot of fun, hosting something for other, potentially many thousand, people makes you kind of responsible. That can be rewarding and fun at times as well, but is also a prime source for headaches.
Deploying stuff is the easy part, knowing what to do when stuff inevitably breaks is where it is at. Therefore, IMHO, it’s probably a good thing that most Lemmy admins at least know where to ask/start when shit hits the fan. This unfortunately leads to more centralization, but for good reasons: teams of volunteers taking care of fewer instances will almost always lead to a better experience than a lot of lone wolfs curating a lot of small instances. Improving scalability, monitoring and documentation is always nice, but will never replace a capable admin such as yourself.
Well that was an entertaining read! Thanks for all your efforts to keep our instance running smoothly. I have noticed it seems a bit snappier since you fixed the problem.
Yep I can confirm is massively faster now to comment and post. Even faster than other instances and other corporation products.
Great writeup, thank you so much for sharing!
Nothing more frustrating than googling an issue and (only) finding forum threads ending in “nvm it works now” 😬
Very fascinating and informative. Thank you for sharing.
