![](https://lemmy.ml/pictrs/image/8cdef1ba-bc60-4109-87fa-0e2b29a7901e.png)
![](https://lemmy.ml/pictrs/image/st9xYggWap.png)
It’s not. It just doesn’t get enough hits for that 86k to matter. Fun fact: most AI crawlers hit /robots.txt
first, they get served a bee movie script, fail to interpret it, and leave, without crawling further. If I’d let them crawl the entire site, that’d result in about two megabytes of traffic. By serving a 86kb file that doesn’t pass as robots.txt and has no links, I actually save bandwidth. Not on a single request, but by preventing a hundred others.
Sadly, that’s not code Linus wrote. Nor one he merged. (It’s from git, copied from rsync, committed by Junio)