I’ve had to turn on Cloudflare’s full proxy with nearly 10’s of thousands of requests coming out of CN IPs, in the last hour making ~7000 requests:

there are ~250,000 requests from china in the last ~20 hours

    • Dave@lemmy.nz
      1·
      13 days ago

      Installing Anubis was certainly a big help for us, Iocaine should do something similar for you, fingers crossed!

      I have been toggling the Cloudflare proxying trying to handle bots without it. old.lemmy.nz has had over 100k hits in the last 12 hours, this is just the ones that got past anubis. Something tells me that’s not legit.

      • ᛒᛚᚢᛖᛇᚦᛖᚱ (BlueÆther)@no.lastname.nzOPM
        2·
        13 days ago

        Yeah, I had about 600k + the ones I blackholed in Caddy in the 20h yesterday (I blackholed the whole 36.0.0.0/8 block in the end on the server)

        3 requests (/api/v3/site /api/v3/post? /api/v3/comment/list?) in 50ms, from a IP then that IP would go silent for ages, then 500ms latter it was a fresh IP

        • Dave@lemmy.nz
          1·
          13 days ago

          I remember investigating and finding all the problematic IPs had requests evenly spread. Dozens of IPs from China, Singapore, and various South American countries. All of them had approx 80k requests in the period I was looking at, as if they send a few requests with one, then the next, then the next, rotating through then back to the start. Avoids rate limits I guess.

          I am considering adding Iocaine as a first line of defence before Anubis, and seeing if that helps

          • ᛒᛚᚢᛖᛇᚦᛖᚱ (BlueÆther)@no.lastname.nzOPM
            2·
            13 days ago
            CF ASN and bot block
│
▼
Caddy/Iocaine
│
├── response other than 421 ──► return Iocaine response
│
└── 421 ──────────────────────► continue to normal Caddy routing
                                  │
                                  ▼
                               Caddy - CN block
                                  │
                                  ▼
                                Lemmy

            Caddyfile:

            {$LEMMY_HOSTNAME} {
    import caddy-common

    ######################################
    ########## log files        ##########
    log cn_blocked {
            output file /data/cn-blocked.log
            format json
            no_hostname
    }

    ########## iocaine  code ##########
    @read method GET HEAD
            reverse_proxy @read host.docker.internal:42069 {
            @fallback status 421
            handle_response @fallback
    }

    ########## cn blocking code ##########
    @blocked {
            remote_ip 1.0.1.0/24 ...
    }

    route @blocked {
            log_name cn_blocked
            respond "<h1>Access Denied</h1>" 403
    }
    ##########   cn block ends  ##########
    ######################################

            Iocaine metrics with CF still on:

            $ curl -s http://127.0.0.1:42042/metrics | grep '^qmk_'
qmk_garbage_generated{host="127.0.0.1:42069"} 3888
qmk_garbage_generated{host="no.lastname.nz"} 251440
qmk_requests{host="127.0.0.1:42069"} 2
qmk_requests{host="172.18.0.1:42069"} 2
qmk_requests{host="no.lastname.nz"} 404
qmk_ruleset_hits{outcome="default",ruleset="default"} 223
qmk_ruleset_hits{outcome="default",ruleset="trusted-path"} 1
qmk_ruleset_hits{outcome="garbage",ruleset="ai.robots.txt"} 11
qmk_ruleset_hits{outcome="garbage",ruleset="major-browsers"} 172
qmk_ruleset_hits{outcome="garbage",ruleset="unwanted-visitors"} 1
            • Dave@lemmy.nz
              1·
              12 days ago

              How’s it holding up, making a difference? I reckon the bot hits are trending down from where they were a few days ago

                • Dave@lemmy.nz
                  2·
                  12 days ago

                  Damn, you’re getting similar levels to us. Got about 520k requests to the UIs in the past 24 hours (this excludes federation or app usage that hit APIs directly, I have Anubis placed after those are split off).

                  Of those 520k, Anubis flat denies about half. and virtually all the rest are challenged, with somewhere around 7% actually being completed. So we are talking about 18k of the 520k getting through now. Most of those are probably still bots (logged in users get a straight pass through, and as mentioned federation or app users don’t follow this path), but the server is much happier now than it was a few days ago.

                  I have Cloudflare disabled now, trying that out. I’ve had to get Grafana up to scratch as I don’t get to use Cloudflare stats to track anymore.