Hey yall, just wanted to chime in with a quick update on why we were down and have gone down a few times these past few days.

So… for those who don’t know a bit on the backend way of how lemmy works right now, it’s a lil more than a mess. Lemmy itself uses very little computing resources, you could probably run a decent instance on a toaster (I wouldn’t suggest it though) but the problem that arises rapidly is the way photos are managed. The photo database downloads images both that are uploaded and federated. There is an option to disable caching for federated NSFW posts by unchecking enable NSFW within the instance admin panel. That’s the only option related disabling federated images as of right now.

You see where I’m going with this, with this instance being as large as it is? Our photo database has been filling up pretty rapidly. We are attempting to troubleshoot solutions to prevent caching all federated content. Unfortunately as our backend team is attempting to figure that out, our pictures database just keeps ballooning more and more.

We have deleted some stuff in hopes to give more time, but we unfortunately have no choice but to migrate to object storage ASAP. Tomorrow there will likely be downtime as we migrate things over, and unfortunately there is no telling how long that will take.

Thank you for being patient and understanding while we navigate this. If you want to see updates to our status when we go down, check out our mastodon page here or come to our matrix public operations channel for updates (link in the sidebar.)

Things should work today though. Emphasis on should.

We are working on open collective still, but hosting this instance does cost money. If you are able to, donations are greatly appreciated. Especially as we will have to factor in object storage into our costs now.

  • monkofapathy@lemmynsfw.com
    link
    fedilink
    English
    arrow-up
    61
    ·
    1 year ago

    Yeahhhhh you don’t want to store images in a database. That will kill the database in a hurry. Store references to object storage.

      • ramble81@lemmynsfw.com
        link
        fedilink
        English
        arrow-up
        19
        arrow-down
        2
        ·
        1 year ago

        MSSQL has had blob storage for a while and I don’t know anyone that uses it. Their Filestream stuff was interesting by letting you store it in the FS but be indexes via the DB, but come-on, use a damn pointer people.

        • harder@pornlemmy.com
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          My company stores some 80+ TB in a series of MSSQL databases. Yes, you read that right. In. Each DB is allowed to grow to 250GB, then a new one is created and written to while the old one becomes read only.

          • wankbank@lemmynsfw.com
            link
            fedilink
            English
            arrow-up
            2
            ·
            1 year ago

            That is horrific. In my day job, I stored images in a database in my first job, learnt my lesson and now have an allergic reaction just thinking about it.

      • monkofapathy@lemmynsfw.com
        link
        fedilink
        English
        arrow-up
        6
        ·
        1 year ago

        That is interesting. Oh well, I’ve met developers who didn’t always make the best design decisions for the backend until their ass has had chunks removed.