I’ve had to turn on Cloudflare’s full proxy with nearly 10’s of thousands of requests coming out of CN IPs, in the last hour making ~7000 requests:

there are ~250,000 requests from china in the last ~20 hours

  • Dave@lemmy.nz
    1·
    23 days ago

    I remember investigating and finding all the problematic IPs had requests evenly spread. Dozens of IPs from China, Singapore, and various South American countries. All of them had approx 80k requests in the period I was looking at, as if they send a few requests with one, then the next, then the next, rotating through then back to the start. Avoids rate limits I guess.

    I am considering adding Iocaine as a first line of defence before Anubis, and seeing if that helps

    • BlueÆther@no.lastname.nzOPM
      2·
      23 days ago
      CF ASN and bot block
│
▼
Caddy/Iocaine
│
├── response other than 421 ──► return Iocaine response
│
└── 421 ──────────────────────► continue to normal Caddy routing
                                  │
                                  ▼
                               Caddy - CN block
                                  │
                                  ▼
                                Lemmy

      Caddyfile:

      {$LEMMY_HOSTNAME} {
    import caddy-common

    ######################################
    ########## log files        ##########
    log cn_blocked {
            output file /data/cn-blocked.log
            format json
            no_hostname
    }

    ########## iocaine  code ##########
    @read method GET HEAD
            reverse_proxy @read host.docker.internal:42069 {
            @fallback status 421
            handle_response @fallback
    }

    ########## cn blocking code ##########
    @blocked {
            remote_ip 1.0.1.0/24 ...
    }

    route @blocked {
            log_name cn_blocked
            respond "<h1>Access Denied</h1>" 403
    }
    ##########   cn block ends  ##########
    ######################################

      Iocaine metrics with CF still on:

      $ curl -s http://127.0.0.1:42042/metrics | grep '^qmk_'
qmk_garbage_generated{host="127.0.0.1:42069"} 3888
qmk_garbage_generated{host="no.lastname.nz"} 251440
qmk_requests{host="127.0.0.1:42069"} 2
qmk_requests{host="172.18.0.1:42069"} 2
qmk_requests{host="no.lastname.nz"} 404
qmk_ruleset_hits{outcome="default",ruleset="default"} 223
qmk_ruleset_hits{outcome="default",ruleset="trusted-path"} 1
qmk_ruleset_hits{outcome="garbage",ruleset="ai.robots.txt"} 11
qmk_ruleset_hits{outcome="garbage",ruleset="major-browsers"} 172
qmk_ruleset_hits{outcome="garbage",ruleset="unwanted-visitors"} 1
      • Dave@lemmy.nz
        1·
        23 days ago

        How’s it holding up, making a difference? I reckon the bot hits are trending down from where they were a few days ago

        • BlueÆther@no.lastname.nzOPM
          2·
          22 days ago

          470k blocked at CF in the past 24h, another 80+k tarpited in iocaine since 12h and still some getting through

          Edit: Moved from mostly from CN to a mix of SE Asia and South America now

          • Dave@lemmy.nz
            2·
            22 days ago

            Damn, you’re getting similar levels to us. Got about 520k requests to the UIs in the past 24 hours (this excludes federation or app usage that hit APIs directly, I have Anubis placed after those are split off).

            Of those 520k, Anubis flat denies about half. and virtually all the rest are challenged, with somewhere around 7% actually being completed. So we are talking about 18k of the 520k getting through now. Most of those are probably still bots (logged in users get a straight pass through, and as mentioned federation or app users don’t follow this path), but the server is much happier now than it was a few days ago.

            I have Cloudflare disabled now, trying that out. I’ve had to get Grafana up to scratch as I don’t get to use Cloudflare stats to track anymore.

              • Dave@lemmy.nz
                1·
                19 days ago

                Damn. I’ve got Cloudflare off so should go install a geoip database so I can check what country things are coming from.

                Not to out do you but we actually hit 50 requests per second a few hours back! That’s through the reverse proxy so includes federation and other traffic not hitting anubis.

                • BlueÆther@no.lastname.nzOPMEnglish
                  2·
                  19 days ago

                  Baseline is about 3.5-4.5k per 15min (4/s), this China traffic is 3x base traffic across 24h

                  For the last 3 days (since turning on the block) there has been a dead spot from China from 4:00 to 14:00 NZST.

                  @Dave@lemmy.nz you seeing the same drop?

                  • Dave@lemmy.nz
                    1·
                    19 days ago

                    It’s just crazy, these indescriminate scrapers are the worst. They think they are getting more data but they are just scraping the same data over and over via different instances 🙄

                  • Dave@lemmy.nz
                    1·
                    19 days ago

                    For the last 3 days (since turning on the block) there has been a dead spot from China from 4:00 to 14:00 NZST.

                    @Dave@lemmy.nz you seeing the same drop?

                    Ah i missed your edit. I don’t have per country data like that, I’ve only very recently played with removing Cloudflare from the equation and haven’t set up any geo-location stuff yet. My requests are coming pretty evenly over the day though, up and down a bit bit depending on time but nothing like what you’re describing. I’m looking at all traffic though, not just China.