actually awesome and fast search engine (depending on which instance you use) with no trashy AI and ADs results also great for privacy, if you don’t know which instance to use go to https://searx.space/ and choose an instance closest to you
actually awesome and fast search engine (depending on which instance you use) with no trashy AI and ADs results also great for privacy, if you don’t know which instance to use go to https://searx.space/ and choose an instance closest to you
Do you have topics that are censored? I searched for my reddit post “what I’ve learnt from the mantis aliens”, and it does not show up in your results. Neither at google’s. But it does on other search engines. The ufo/alien stuff are censored in most search engines, while there isn’t a reason to be. That is how I judge search engines. And Mojeek doesn’t give me the results I asked for.
Reddit doesn’t allow us to crawl: https://www.reddit.com/robots.txt
Is that legally binding? What happens of they catch you, ban your IPs then you’re in the same situation as now. Literally no reason to not do it IMO.
IP already hits a wall, also better to not get a reputation as a bad bot, it’s taken a while to get known for being friendly and respecting rules, to us you should follow robots
I seem to recall creative ways to index things without robots, e.g. browser addon that users opt into to send pages and such, essentially crowdsourcing the indexing. Anyways good to see you’re taking the high road!